Information processing device and its control method

Abstract

To enable more precise subject tracking. [Solution] The information processing device includes an image acquisition means for acquiring frame images contained in a video; a depth acquisition means for acquiring depth information relating to the depth in each region within the frame image; a tracking target setting means for setting a tracking target; a candidate detection means for detecting one or more tracking candidates similar to the tracking target from the frame image; and a selection means for identifying a tracking target from one or more tracking candidates. The selection means identifies the tracking target based on the similarity between the tracking target in a preceding past frame image and one or more tracking candidates in the frame image. The similarity is based at least on a first similarity between the depth of the tracking target in the past frame image and the depth of one or more tracking candidates in the frame image.

Description

Technical Field

[0001] The present invention relates to a technique for tracking a subject in a moving image.

Background Art

[0002] As techniques for tracking a specific subject in a moving image, there are those that utilize luminance or color information, those that utilize template matching, and the like. In recent years, techniques using DNN (Deep Neural Network) have attracted attention.

[0003] Non-Patent Document 1 discloses a tracking method using CNN (Convolutional Neural Network). Specifically, an image of a tracking target and an image in a search range are respectively input into a CNN with the same weights, and the position of the tracking target in the image in the search range is specified by calculating the cross-correlation between the respective feature amounts obtained from the CNN. Patent Document 1 discloses a technique for improving the tracking accuracy by acquiring three-dimensional (vertical, horizontal, depth) position information of an object and predicting the positions of current multiple objects based on the position information of multiple past objects.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, since the above method relies on detection from image features, it has the drawback of being prone to false tracking when multiple objects similar to the target are located close together in the image.

[0007] This invention has been made in view of these problems and aims to provide a technology that enables more accurate subject tracking. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the information processing apparatus according to the present invention has the following configuration. That is, the information processing apparatus is Image acquisition means for acquiring frame images contained in a video, Depth acquisition means for acquiring depth information relating to the depth in each region within the frame image, Tracking target setting means for setting the tracking target, Candidate detection means for detecting one or more tracking candidates similar to the tracking target from the frame image, A means for identifying the target to be tracked from the one or more tracking candidates, Equipped with, The identification means identifies the tracking target based on the similarity between the tracking target in a past frame image preceding the frame image and each of the one or more tracking candidates in the frame image. The aforementioned similarity is based at least on a first similarity between the depth of the tracking target in the past frame image and the depth of each of the one or more tracking candidates. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that enables more accurate subject tracking. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram shows the hardware configuration of an information processing device. [Figure 2] It is a diagram showing the functional configuration of the information processing apparatus (during inference processing). [Figure 3] It is an overall flowchart of the inference processing. [Figure 4] It is a diagram exemplarily showing a plurality of objects detected from an image. [Figure 5] It is a detailed flowchart of S303. [Figure 6] It is a detailed flowchart of S305. [Figure 7] It is a diagram showing a situation where success or failure in detecting the position of the tracking target has occurred. [Figure 8] It is a flowchart showing the processing in the CNN. [Figure 9] It is a detailed flowchart of S306. [Figure 10] It is a diagram showing the relationship between the defocus amount and the phase difference between two focus detection signals. [Figure 11] It is a diagram showing the functional configuration of the information processing apparatus (during learning processing). [Figure 12] It is a flowchart of the learning processing. [Figure 13] It is a diagram showing examples of a template image and a search range image. [Figure 14] It is a diagram showing examples of an inference result and a true value. [Figure 15] It is a diagram explaining object tracking using depth information. [Figure 16] It is a detailed flowchart of S306 when using the reliability of depth information. [Figure 17] It is a diagram showing the time change of the focus lens position of the imaging apparatus. [Figure 18] It is a detailed flowchart of S306 (Second Embodiment). [Figure 19] It is a diagram explaining the update of the feature amount acquisition position using depth information. [Figure 20] It is a detailed flowchart of S306 (Third Embodiment).

Mode for Carrying Out the Invention

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) As a first embodiment of the information processing device according to the present invention, a device that performs object tracking processing using a neural network (NN) will be described below as an example. In particular, a configuration that suppresses false tracking, where similar objects are mistakenly targeted, by using depth information to track the target will be described.

[0013] <Device configuration> Figure 1 shows the hardware configuration of the information processing device (computer) in the first embodiment. The CPU 101 controls the entire device by executing the control program stored in the ROM 102. The RAM 103 temporarily stores various data from each component and also operates as the work memory for the CPU 101.

[0014] The storage unit 104 stores the data to be processed in this embodiment and stores the data to be tracked. The storage unit 104 can use a hard disk drive (HDD), flash memory, various optical media, etc. The input unit 105 consists of a keyboard, touch panel, dial, etc., and accepts user input, and is used when setting the tracking target, as described later. The display unit 106 consists of a liquid crystal display, etc., and provides the user with subject images and tracking results through image display. The communication unit 107 is a functional unit for the information processing device to communicate with other devices such as a camera.

[0015] <Inference Processing> Figure 2 shows the functional configuration of the information processing device during inference processing. Figure 3 is an overall flowchart of the inference processing. However, the information processing device 1 does not necessarily have to perform all the steps described in this flowchart. The processes executed by the CPU 101 are shown as functional blocks.

[0016] In S301, the input image acquisition unit 201 acquires an image of a person (a frame image included in the video). The depth information acquisition unit 202 acquires information regarding the depth of the image acquired by the input image acquisition unit 201 (depth acquisition). The input image acquisition unit 201 may acquire an image captured by an imaging device connected to the information processing device 1, or an image stored in the storage unit 104. The depth information acquisition unit 202 may acquire depth information from an imaging device connected to the information processing device 1, or depth information stored in the storage unit 104.

[0017] In S302, the tracking target setting unit 203 determines the tracking target in the image according to the instructions specified via the input unit 105. One specific method for determining the tracking target is for the user to touch the subject displayed on the display unit 106. In addition to being specified by the input unit 105, the tracking target may also be determined by automatically detecting the main subject in the image. For example, Reference A is cited as a method for automatically detecting the main subject in the image. Alternatively, the tracking target may be determined based on both the specification by the input unit 105 and the results of object detection in the image. For techniques for detecting objects in an image, Reference B is cited. (Document A) Patent No. 6556033 (Reference B) Liu et al., "SSD:Single Shot Multibox Detector", arXiv:1512.02325, ECCV, 2016

[0018] Figure 4 is an illustrative diagram showing multiple objects detected from an image. Objects 403, 405, and 407 are detected candidates for tracking. Rectangles 402, 404, and 406 are bounding boxes (BBs) indicating candidates. The user can determine the tracking target by touching one of the BBs shown on the display unit 106 or by selecting it using a dial or the like. There are various methods for determining the tracking target, and it is not limited to the method described above.

[0019] In S303, the tracking target template generation unit 204 generates a template (tracking target template) that represents the characteristics of the tracking target determined by the tracking target setting unit 203.

[0020] In S304, the input image acquisition unit 201 acquires a search range image (for example, a frame image following the image in S301).

[0021] In S305, the candidate detection unit 205 detects one or more candidates for the tracking target specified by the tracking target setting unit 203 (one or more tracking candidates) from the search range image. The feature extraction unit 206 extracts the features of the tracking target candidates obtained by the candidate detection unit 205 from the image.

[0022] In S306, the tracking target identification unit 207 identifies the candidate that is most likely to be a tracking target from among the one or more candidates obtained by the candidate detection unit 205 as the tracking target.

[0023] Figure 5 is a detailed flowchart of the tracking target template generation (S303). The tracking target generation unit 203 generates a template for representing the tracking target based on the image obtained by the input image acquisition unit 201 and the tracking target BB obtained by the tracking target setting unit 203.

[0024] In S501, the tracking target template generation unit 204 identifies the region where the tracking target exists based on the tracking target BB obtained by the tracking target setting unit 203.

[0025] In S502, the tracking target template generation unit 204 extracts the area around the region obtained in S501 from the image and resizes it to a predetermined size.

[0026] In S503, the tracking target template generation unit 204 inputs the resized image into the CNN. The CNN is pre-trained to obtain features that make it easy to distinguish between tracking targets and non-tracking targets. The training method will be described later.

[0027] In S504, the tracking target template generation unit 204 stores the feature quantities obtained in S503 as the tracking target template.

[0028] Figure 8 is a flowchart showing the processing in the CNN in S503. The CNN is composed of layers such as convolutional layers, rectified linear unit (ReLU) layers, and max pooling layers.

[0029] It should be noted that the ReLU and Max Pooling described here are merely examples. Instead of ReLU, you may use Leaky ReLU, PRelu, GELU, or the Sigmoid function, or instead of Max Pooling, you may use Average Pooling. Furthermore, you may use a network architecture other than a CNN, such as a Vision Transformer or MLP Mixer. Detailed disclosures for the Vision Transformer are found in reference C, and detailed disclosures for the MLP Mixer are found in reference D. However, the network architecture used is not limited to these. (Reference C) Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", arXiv:2010.11929, 2020 (Reference D) Tolstikhin et al., "MLP-Mixer: An all-MLP Architecture for Vision", arXiv:2105.01601, 2021

[0030] Figure 6 is a detailed flowchart of candidate detection (S305). The objective of S305 is to detect the target to be tracked from the image that constitutes the search range obtained in S304.

[0031] In S601, the candidate detection unit 205 determines the area to search for candidates. The search area may be the entire search image, or it may be the area around the position of the previously tracked target.

[0032] In S602, the candidate detection unit 205 extracts a region based on the determined search region and resizes it to be equivalent to the resizing ratio in S502.

[0033] In S603, the candidate detection unit 205 inputs the image of the extracted region into the CNN. The CNN in S603 has some or all of the same weights as the CNN in S503.

[0034] In S606, the candidate detection unit 205 acquires the tracking target template obtained in S504. In S604, the candidate detection unit 205 calculates the cross-correlation between the tracking target template and the result obtained in S603. At this time, the cross-correlation value will be large for locations where the probability of a tracking target existing within the search range is high due to some or all of the weights of the CNN in S603 and the CNN in S503 being the same. Therefore, it becomes possible to detect locations where the cross-correlation value is above a threshold as locations of tracking target candidates. Here, a tracking target candidate refers to one or more candidates that could be tracking targets. A tracking target candidate may include either tracking targets or non-tracking targets, or both.

[0035] In S605, the candidate detection unit 205 detects the tracking target candidate obtained in S604 as a candidate BB. In order to obtain a BB, not only the position but also the width and height of the BB are required. The position of the BB is determined based on the position that showed a high response in the cross-correlation.

[0036] Figure 7 shows the situation when the tracking target's position detection was successful (Figure 7(a)) and the situation when it failed (Figure 7(b)).

[0037] Map 701 shows a map obtained based on cross-correlation. The object being tracked is object 702, and the cross-correlation value of cell 704 near the center of object 702 is high. If this correlation value is above a threshold, it can be estimated that object 702 is located in cell 704.

[0038] Furthermore, as shown in Figure 7(b), if an object 712 with similar image features is located nearby, the cross-correlation value of cell 714 may increase for the tracking target object 702. If the system estimates an incorrect location for the tracking target object, it may acquire the features of object 712 as features of the tracking target object 702, resulting in false tracking where the tracking target switches from object 702 to object 712. Therefore, in S306, it is necessary to accurately identify the tracking target by matching the tracking target with similar objects using depth information.

[0039] The width and height of the broadband can be pre-trained so that the CNN can estimate them (see below). Alternatively, the width and height of the tracking target broadband obtained in S302 can be used directly.

[0040] Figure 9 is a detailed flowchart of the tracking target identification (S306). In the tracking target identification unit, the tracking target is identified from among the tracking target candidates obtained in the candidate detection unit 205.

[0041] In S905, the tracking target identification unit 207 acquires object BB, feature quantities, and depth information of the input image acquired by the depth information acquisition unit 202.

[0042] In S901, the tracking target identification unit 207 calculates the similarity between candidates in past timeframe images (past frame images) stored in the memory unit 211 and candidates at the current time obtained by the candidate detection unit 205. The past frame images are frame images preceding the frame image of interest for which the tracking target is currently being detected. Past candidates are assigned tracking target / non-tracking target labels. Object BB, feature quantities, and depth information are used to calculate the similarity. Here, the defocus amount is used as the depth information for explanation. However, depth information such as depth information measured using a ToF (Time of Flight) reflective laser sensor may also be used.

[0043] Defocus amount (defocus) is the deviation at the imaging plane obtained by multiplying the image displacement amount calculated from a pair of images with parallax (image A and image B described later) by a predetermined conversion coefficient. The information of the defocus amount distribution to which the defocus amount is assigned to a predetermined pixel area of ​​the imaging plane is called a defocus map. In this invention, the unit of defocus amount is the product of the aperture F value and the allowable circle of confusion diameter δ in the imaging device optical system during image acquisition [Fδ].

[0044] Figure 10 shows the relationship between the amount of defocus in the imaging optical system and the phase difference between the two focus detection signals. Here, the phase difference (image shift) between the first focus detection signal and the second focus detection signal acquired from the image sensor is shown.

[0045] An image sensor (not shown) is placed on the imaging surface 1000, and the exit pupil of the imaging optical system is divided into two parts: a first pupil region 1011 and a second pupil region 1012.

[0046] The amount of defocus d is defined such that |d| is the distance (magnitude) from the imaging position C of the light beams from subjects 1021 and 1022 to the imaging plane 1000, and a negative sign (d<0) represents a "front-focused state" where the imaging position C is on the subject side of the imaging plane 1000. Conversely, a positive sign (d>0) represents a "back-focused state" where the imaging position C is on the opposite side of the subject from the imaging plane 1000. In the in-focus state where the imaging position C is on the imaging plane 1000, d=0. The imaging optical system is in focus (d=0) with respect to subject 1021 and in a front-focused state (d<0) with respect to subject 1022. The front-focused state (d<0) and the back-focused state (d>0) together are called a defocused state (|d|>0).

[0047] In the front-focused state (d<0), the light beam from the subject 1022 that passes through the first pupil region 1011 is focused and then spreads out with a width Γ1 centered on the centroid position G1 of the light beam, forming a blurred image on the image sensor 1000. This blurred image is received by each first focus detection pixel on the image sensor, and a first focus detection signal is generated. In other words, the first focus detection signal is a signal representing a subject image in which the subject 1022 is blurred by a blur width Γ1 at the centroid position G1 of the light beam on the image sensor 1000. Similarly, the second focus detection signal is a signal representing a subject image in which the subject 1022 is blurred by a blur width Γ2 at the centroid position G2 of the light beam on the image sensor 1000.

[0048] The blur widths Γ1 and Γ2 of the subject image increase roughly in proportion to the increase in the magnitude of the defocus amount d |d|. Similarly, the magnitude of the image shift amount p (=difference in the centroid position of the light beam G1-G2) between the first focus detection signal and the second focus detection signal |p| also increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. The same applies even in a back-focus state (d>0), although the direction of the image shift between the first focus detection signal and the second focus detection signal is opposite to that in a front-focus state.

[0049] Thus, as the amount of defocus increases, the amount of image shift between the first and second focus detection signals also increases. In this embodiment, a phase-difference detection method is used to calculate the amount of defocus from the amount of image shift between the first and second focus detection signals obtained using the image sensor. In the following description, the first and second focus detection signals will be referred to as image A and image B, respectively.

[0050] As an example of similarity calculation in S901 described above, the similarity L between past candidate c1 and current candidate c2 is calculated as shown in equation (1).

[0051]

number

[0052] Here, BB is a vector summarizing the four variables (center coordinate x, center coordinate y, width, height) of each candidate BB, and f represents the features of each candidate. The features are extracted from the feature map obtained from the aforementioned CNN, indicating the location of each candidate. Also, e1 and e2 represent the defocus amounts of past candidate c1 and current candidate c2. W1, W2, and W3 are empirically obtained coefficients, where W1>0, W2>0, and W3>0.

[0053] In S902, the tracking target identification unit 207 performs matching based on the similarity between past candidates and current candidates calculated in S901. A high similarity between past and current candidates indicates a high probability that the past and current candidates are the same object. By performing appropriate matching, the past and current tracking targets can be recognized as the same object. Other matching methods can also be used, such as a method that prioritizes matching candidates with high similarity, or a method that uses the Hungarian algorithm.

[0054] In S903, the tracking target identification unit 207 identifies the tracking target. Based on the matching results obtained in S902, the current candidate that matches with past tracking targets can be identified as the tracking target.

[0055] In S904, the tracking target identification unit 207 updates the tracking target and its candidate BBs, characteristics, and defocus amount stored in the DB. In this embodiment, past candidates that did not match the current candidate are retained when the DB is updated, but this is not limited to this. For example, past candidate information for which no match occurred even after being retained for a predetermined time may be deleted to save system capacity.

[0056] In this embodiment, the defocus amount for each candidate is obtained from the aforementioned defocus map superimposed on the image. The defocus amount for each candidate is obtained from the vicinity of the candidate range for each candidate (for example, the estimated position of the cross-correlation result and the average value of its adjacent region). The method for obtaining the defocus amount for each candidate is not limited to this. For example, the defocus amount of the subject may be inferred and obtained by eliminating the effects of near and far views, etc., using a subject defocus inferencer that has been trained with the image, defocus map, and the defocus amount of the subject as the true value (GT: Ground Truth). Furthermore, the defocus amount may not be a single point value, but rather a range value in which the subject exists, having a maximum and minimum value. In addition, if depth information is used, depth information of the region in which the subject exists may be obtained by depth estimation, etc. A detailed disclosure of depth estimation methods is provided in reference E. (Reference E) Huynh et al., "Boosting Monocular Depth Estimation with Lightweight 3D Point Fusion", arXiv:2012.10296, 2020

[0057] Furthermore, although this embodiment shows a configuration in which both the target to be tracked and similar objects are stored in the DB and matching is performed between past candidates and current candidates, the information stored in the DB may, for example, only be the target object BB, feature quantities, and depth information. In that case, the matching in S902 can be skipped, and in S901, based on the similarity calculation results between the target to be tracked and each current candidate, the candidate showing the value most similar to the target to be tracked in S903 can be identified as the target to be tracked.

[0058] <Learning Process> This section describes the training process of the CNN used for searching for the target to track. Figure 11 shows the functional configuration of the information processing device during the training process. The information processing device 2 includes a template image acquisition unit 1101, a search range image acquisition unit 1102, a GT acquisition unit 1103, a target to track unit 1104, a loss calculation unit 1105, a parameter update unit 1106, a parameter storage unit 1107, and a storage unit 211.

[0059] The template image acquisition unit 1101 acquires an image in which the target to be tracked exists. The search range image acquisition unit 1102 acquires an image that will be used to search for the target to be tracked. For example, the template image acquisition unit 1101 selects an arbitrary frame from the sequence video, and the search range image acquisition unit 1102 selects another frame from the sequence video that was not selected by the template image acquisition unit 1101.

[0060] The GT acquisition unit 1103 acquires the target BB in the template image obtained by the template image acquisition unit 1101 and the target BB in the search range image obtained by the search range image acquisition unit 1102.

[0061] The tracking target estimation unit 1104 estimates the tracking target based on the template image obtained by the template image acquisition unit 1101, the search range image obtained by the search range image acquisition unit 1102, and the tracking target BB obtained by the GT acquisition unit 1103.

[0062] The loss calculation unit 1105 calculates the loss based on the tracking result obtained by the tracking target estimation unit 1104 and the BB of the tracking target in the search range image obtained by the GT acquisition unit 1103.

[0063] The parameter update unit 1106 updates the CNN parameters based on the loss obtained in the loss calculation unit 1105. Here, the parameters are updated so that the loss value converges. When the sum of the loss values ​​converges, or when the loss value becomes smaller than a predetermined value, the parameter set is updated and training is terminated. The parameter storage unit 1107 stores the CNN parameters updated in the parameter update unit 1106 as learned parameters in the storage unit 211.

[0064] Figure 12 is a flowchart of the learning process.

[0065] In S1201, the template image acquisition unit 1101 acquires a template image. Figure 13 shows an example of a template image and a search area image. Object 1301 is the object to be tracked, rectangle 1302 is the BB of the object to be tracked obtained by the GT acquisition unit 1103, and rectangle 1303 is the region to be cut out as a template.

[0066] In S1202, the template image acquisition unit 1101 extracts a region to be used as a template from the template image and resizes it to a predetermined size. The size of the region to be extracted is determined based on the tracking target BB, such as by setting it as a constant multiple of the BB size.

[0067] In S1203, the tracking target estimation unit 1104 inputs the template image generated in S1202 into the CNN and obtains the CNN features of the template.

[0068] In S1204, the search range image acquisition unit 1102 acquires an image. Object 1304 represents the target to be tracked, rectangle 1305 represents the target BB, and rectangle 1306 represents the search range area.

[0069] In S1205, the search range image acquisition unit 1102 generates a search range image by cutting out the search range region from the image acquired in S1204 and resizing it. The size of the search range image is determined to be a constant multiple of the size of the tracking target BB, and is resized to match the resizing factor of the template in S1202. For example, it is resized so that the size of the tracking target in the resized template and the size of the tracking target in the search range image are approximately the same.

[0070] In S1206, the tracking target estimation unit 1104 inputs the search range image generated in S1205 into the CNN to obtain the CNN features of the search range.

[0071] In S1207, the tracking target estimation unit 1104 calculates the cross-correlation between the CNN features of the template obtained in S1206 and the CNN features of the search range obtained in S1206.

[0072] Figure 14(a) shows an example of a map (inference result) obtained by cross-correlation. Map 1401 is a map obtained by cross-correlation, and cells 1402 and 1403, shown in gray, indicate areas with high cross-correlation values. In this way, the cross-correlation values ​​are high at locations that are likely to be tracked.

[0073] Figure 14(b) shows the true value (cell 1405, which represents the correct tracking target position) obtained by the GT acquisition unit 1103. In other words, cell 1402 indicates the position of the tracking target, thus estimating a desirable value, but cell 1403, despite not being a tracking target, has a high cross-correlation value, indicating that an undesirable value has been estimated. In the learning process, the weights are updated so that the cross-correlation value is high at the tracking target position and low at positions other than the tracking target.

[0074] In S1208, the loss calculation unit 1105 calculates the loss related to the inferred position of the tracked target and the loss related to the size of the tracked target. The position loss is calculated so that the cross-correlation value of the tracked target's position is high. If the map 1401 obtained in S1207 is Cin and the GT map 1404 is Cgt, the loss function can be described as shown in equation (2).

[0075]

number

[0076] Equation (2) is the average of the squared differences between each pixel of map Cin and map Cgt. The loss is small when the tracked target is correctly estimated, and large when a non-tracked target is estimated to be tracked, or a tracked target is estimated to be non-tracked.

[0077] Similarly, the size loss is calculated according to equations (3) and (4).

[0078]

number

[0079]

number

[0080] LossW and LossH are the estimated losses related to the width and height of the tracked object, respectively. Wgt and Hgt are the width and height values ​​of the tracked object embedded in the position of the tracked object, respectively. By calculating the losses using equations (3) and (4), learning progresses so that the width and height of the tracked object are inferred to the position of the tracked object in both Win and Hin. Combining the three losses described above results in equation (5).

[0081]

number

[0082] Here, the loss is described in the form of Mean Squared Error (MSE), but the loss is not limited to MSE. Smooth-L1 or other methods may also be used. This does not limit the formula for calculating the loss. Furthermore, the loss function for position and the loss function for size may be different.

[0083] In S1209, the parameter update unit 1106 updates the CNN parameters based on the loss calculated in S1208. Parameter updates are performed using backpropagation, such as Momentum SGD. Although the output of the loss function for a single image has been described, in actual training, the loss value in equation (2) is calculated for the estimated scores of multiple different images. The coupling weight coefficients between layers of the training model are updated so that the loss values ​​for all of the multiple images are smaller than a predetermined threshold.

[0084] In S1210, the parameter storage unit 1107 stores the CNN parameters updated in S1209 in the storage unit 211. In the inference process, the parameters stored in S1210 are used to perform inference (Figure 3).

[0085] In S1211, it is determined whether to terminate the learning process. The termination of the learning process is determined, for example, when the loss value obtained in equation (2) falls below a predetermined threshold.

[0086] <Effects> Figure 15 illustrates object tracking using depth information. Specifically, it shows a situation where the target object and similar objects are tracked simultaneously using a defocus map.

[0087] Images 1501, 1502, and 1503 in Figure 15(a) show the images obtained at times t=0, t=1, and t=2, respectively. The images show person 1504 and person 1505, of which person 1504 is the target of tracking, and person 1505 is the similar object.

[0088] Images 1521, 1522, and 1523 in Figure 15(b) superimpose the defocus maps generated at times t=0, t=1, and t=2, respectively, onto the images at each time point in Figure 15(a). The defocus maps at each time point are shown in grids 1531, 1532, and 1533, respectively, and each cell within the grid is assumed to have a corresponding amount of defocus.

[0089] At time t=0, cells 1541 and 1542 are assumed to be the cells selected from the map as defocus values ​​indicating the depth of people 1504 and 1505. At time t=1, cell 1543 is assumed to be the cell selected as the defocus value indicating the depth of person 1508. Furthermore, at time t=2, cells 1544 and 1545 are assumed to be the cells selected as defocus values ​​indicating the depth of people 1511 and 1512. Figure 15(c) shows the feature quantities, BB, and defocus value information obtained at each time point.

[0090] First, let's consider the conventional method of tracking the target and similar objects using only image features and broadband (without using defocus information, which is depth information). In this case, at time t=0, tracking begins simultaneously for person 1504, the target, and person 1505, the similar object. At time t=1, person 1509 is occluded by person 1508. At this point, person 1508, which was detected as a candidate, will be matched with either person 1504 or person 1505 at t=0.

[0091] The feature quantity of the target being tracked at t=0 "f 1|t=0 " and the feature quantity "f" of similar objects 2|t=0Let's consider a case where the two objects are very similar, and it's almost impossible to find any difference in the BB (Big Bang) data. In such a case, the similarity calculation may result in a match between candidate 1510 and similar object 1505. As a result, at time t=1, person 1508 is selected as the tracking target, causing a switch from person 1509, which was the original tracking target. At time t=2, if person 1511 is matched as the tracking target, continuing from the previous time, person 1512, which was the original tracking target, cannot be tracked, resulting in a tracking failure.

[0092] On the other hand, let's consider tracking a target using a defocus amount according to the first embodiment described above. Assume that at time t=0, the target person 1504 is in the background and the similar object person 1505 is in the foreground. In that case, the respective defocus amounts "e 1|t=0 "e 2|t=0 The relationship between the magnitudes of the values ​​of "e 1|t=0 」<「e 2|t=0 ". Under conditions where a switch is likely to occur at time t=1, the amount of defocus of person 1508 is "e 1|t=1 " is analogous object 1505 at time t=0, "e 2|t=0 This results in a value similar to that of (indicating the amount of defocus on the near side). In other words, when the past difference value is taken, the difference with the similar object, person 1505, is smaller. Therefore, when the similarity is calculated according to equation (1), the similarity L1 between person 1508 and the previously tracked person 1504 is given by equation (6). Also, the similarity L2 between person 1508 and the previously similar object, person 1505, is given by equation (7).

[0093]

number

[0094]

number

[0095] When these are calculated, even when there are no differences in feature amounts or BB (that is, when there is almost no difference between the first and second terms in equations (6) and (7)), a significant difference occurs in the past difference value of the defocus amount of the third term. That is, the third term of the similarity degree L2 with the past similar object, person 1505, becomes a value close to 0, and the third term of the similarity degree L1 with the past tracking target, person 1504, becomes larger. As a result, L1 becomes a negative value, L2 is near 0, and L1 < L2. In the case of the similarity degree calculated according to equation (1), it will match the larger value. Therefore, person 1508 correctly matches with person 1505, who is a past similar object, and no switching of the tracking target occurs.

[0096] As described above, according to the first embodiment, by using depth information to track the object to be tracked, it is possible to suppress the occurrence of mis-tracking. In particular, in a case where the object to be tracked and a similar object are close to each other in the front-back direction in the depth direction, it is possible to suppress the occurrence of mis-tracking.

[0097] (Modification example) As a modification example, a method of further utilizing the reliability of depth information will be described. Depth information may not always be accurate information due to the influence of disturbances such as noise. The same applies to the defocus amount exemplified as depth information in the first embodiment. The distance measurement method described above with reference to FIG. 10 is based on the premise that the image shapes of the A image and the B image are the same. Therefore, reliability is defined based on the degree of coincidence of the image shapes of the A image and the B image.

[0098] In each cell of the aforementioned defocus map, if reliability information for the amount of defocus is obtained (reliability acquired), this information is used to determine how much to consider the amount of defocus. Specifically, if the reliability of the amount of defocus is high, the coefficient W3 of the past difference value of the amount of defocus, defined as the third term in equation (1), is set to a predetermined value. On the other hand, if the reliability of the amount of defocus is low, the coefficient W3 is set to a value lower than the predetermined value mentioned above. This reduces erroneous tracking due to incorrect depth information by suppressing the consideration of depth information when performing similarity calculations when the reliability of depth information is low. Note that if the reliability of the amount of defocus is less than a predetermined value, the coefficient W3 may be configured to be zero.

[0099] Figure 16 is a detailed flowchart of S306 when utilizing the reliability of depth information. Note that the explanation of parts that are the same as in the first embodiment 1 (Figure 9) is omitted.

[0100] In S1604, the tracking target identification unit 207 determines whether the reliability of the current candidate for the defocus amount is above a predetermined value. If the reliability is above the predetermined value, the process proceeds to S1606; otherwise, the process proceeds to S1607.

[0101] In S1606, the tracking target identification unit 207 updates the BB and features, defocus amount, and reliability of the defocus amount of the tracking target and its candidates, which were stored in the DB in S1605. On the other hand, in S1607, the tracking target identification unit 207 does not update the information on the defocus amount and reliability of the defocus amount of the tracking target and its candidates, which were stored in the DB in S1605 in S1617, but only updates the BB and features.

[0102] By performing this process, depth information at unreliable points in time is discarded, which reduces false tracking caused by incorrect depth information.

[0103] Regarding the setting of coefficients according to the reliability of depth information, in this embodiment, if the reliability is obtained as a discrete value, each coefficient is set empirically accordingly, but this is not limited to this. For example, if the reliability is a continuous quantity, the coefficient may be set as a continuous quantity accordingly. Alternatively, these coefficients according to the reliability of depth information may be obtained from the results of the tracking learning described above.

[0104] As explained above, the modified method further uses the reliability information of the depth information to identify the target to be tracked. This makes it possible to suppress false tracking even when the depth information is affected by external disturbances.

[0105] (Second Embodiment) In the second embodiment, a method for utilizing state information regarding the operating state of an imaging device connected to an information processing device will be described. In particular, a method for updating depth information by acquiring information on the focus lens drive amount as the operating state will be described.

[0106] Imaging devices (such as single-lens reflex cameras) are prone to sudden changes in their operating state when the photographer performs actions such as zooming, focusing, and framing during shooting. Such sudden changes in operating state can also affect the depth information obtained from the imaging device. In this embodiment, stable tracking can be achieved even in such situations by utilizing depth information.

[0107] Figure 17 shows the time change in the focus lens position of the imaging device. Here, imaging starts at time t0. From time t0 to t1, the focus lens position remains almost unchanged, and the lens movement is small. From time t1 to t2, the focus lens position changes significantly, and the lens movement is large. As mentioned above, referring to Figure 10, the amount of defocus is a deviation at the image plane, and therefore it is a relative value with respect to the lens position. For this reason, it is not possible to simply compare the amount of defocus before and after a large movement of the focus lens.

[0108] Therefore, a method can be used in which information regarding the amount of focus lens drive is obtained from the imaging device (state acquisition), and the weight W3 for the similarity calculation defined by equation (1) is set according to the amount of lens drive. For example, if the amount of lens drive is large, setting a smaller weight can reduce erroneous tracking due to incorrect depth information.

[0109] Alternatively, a method may be used in which the lens drive amount is multiplied by a predetermined conversion coefficient to convert it to image plane distance. Here, the conversion coefficient is a quantity that differs depending on the imaging optical system, such as the focusing lens. The calculated image plane distance can be converted to a defocus amount by dividing it by the product of the aperture F value and the allowable circle of confusion diameter δ [Fδ] in the imaging optical system. Based on this defocus converted value of the lens drive amount, the information on the defocus amount of the current or past candidate can be updated, and the difference value between the past and current defocus amounts can be obtained in the similarity calculation.

[0110] Figure 18 is a detailed flowchart of S306 in the second embodiment. Figure 18(a) shows the overall flow when updating defocus information based on the lens drive amount, and Figure 18(b) shows the detailed flow of S1801. Note that the explanation of parts that overlap with the flow of the first embodiment (Figure 9) is omitted. In S1801, the tracking target identification unit 207 preprocesses the defocus information based on the lens drive amount.

[0111] In S1811, the tracking target identification unit 207 obtains the lens drive amount from the imaging device. Information regarding the lens drive amount of the imaging device is obtained via the communication unit 107. In S1812, the tracking target identification unit 207 converts the lens drive amount into a defocus amount. In S1813, based on the converted defocus amount, the defocus amounts of past candidates or current candidates are added or subtracted to convert (correct) them.

[0112] From S901 onward, the procedure is the same as in the first embodiment, but the difference between the current candidate and past candidate for the defocus amount is calculated using the converted defocus amount (correction depth).

[0113] As described above, according to the second embodiment, the amount of defocus is calculated (converted) based on the amount of lens drive between frame images during tracking processing. This makes it possible to suppress the occurrence of false tracking even when the operating state of the imaging device changes significantly between frame images during tracking processing, which affects the accuracy of depth information.

[0114] (Third embodiment) In the third embodiment, a method for determining an appropriate subregion for acquiring feature quantities of candidate targets to be tracked using depth information will be described. In particular, a configuration that enables more accurate acquisition of feature quantities for each candidate when the target to be tracked and similar objects are positioned so that they overlap in the depth direction, front and back, will be described.

[0115] Figure 19 illustrates the updating of feature acquisition locations using depth information (region determination of subregions used for feature derivation). Map 1901 in Figure 19(a) shows a map obtained from the cross-correlation calculation results of CNN features between the template and the search range. Consider the case where, from this result, candidate person 1902 is estimated to be at the location of cell 1904, and person 1903 is estimated to be at the location of cell 1905. In this case, the location of cell 1904 includes a large portion of the region of person 1903, and if features are acquired from this result, there is a risk that the features of person 1902 will not be captured well.

[0116] Figure 19(b) illustrates the updating of the feature acquisition position for person 1902 using the defocus map 1911. Cells 1912 and 1913 in Figure 19(b) indicate the defocus amounts at the positions of cells 1904 and 1905 in Figure 19(a), respectively. The defocus amount for person 1902 is calculated from the average value of region 1914, which includes the surrounding cells of cell 1912, and the defocus amount for person 1903 is calculated from the average value of region 1915, which includes the surrounding cells of cell 1913. From these results, it can be determined that cells 1904 and 1905, which were originally the positions for acquiring the feature amounts of each candidate, are both near the defocus amount of person 1903.

[0117] Therefore, the feature acquisition position is updated to a position near the defocus amount of person 1902, which is calculated from the average value of region 1914. The appropriate position for updating is cell 1916, which is "a region near the defocus amount of person 1902" and "the position closest in distance to the position of cell 1904" in the defocus map 1911.

[0118] Figure 19(c) shows the result of updating the feature acquisition location for person 1902 to cell 1921. This process allows for the correct acquisition of features for each candidate.

[0119] Figure 20 is a detailed flowchart of S306 in the third embodiment. Figure 20(a) shows the overall flow of the feature information update process using the defocus map, and Figure 20(b) shows the detailed flow of S2001. Note that the explanation of parts that overlap with the flow of the first embodiment (Figure 9) is omitted. In S2001, the tracking target identification unit 207 performs the feature information update process.

[0120] In S2011, the tracking target identification unit 207 obtains a map of the results of the cross-correlation calculation of CNN features between the template and the search range, and a defocus map.

[0121] In S2012, the tracking target identification unit 207 obtains the amount of defocus for the estimated position (= position where feature information is obtained) of each candidate from the cross-correlation result map and defocus map obtained in S2011.

[0122] In S2013, the tracking target identification unit 207 acquires the defocus amount for each candidate. As described in the first embodiment, the defocus amount for each candidate is not limited to the method of acquiring it from the vicinity of the candidate range of each candidate as described above, but may also be an estimation result using a defocus inference unit. Furthermore, the defocus amount may not be a single point value, but rather a range value in which the subject exists, having a maximum and minimum value. In addition, if depth information is used, depth information of the region in which the subject exists may be obtained by depth estimation or the like.

[0123] In S2014, the tracking target identification unit 207 determines whether there is a difference between the amount of defocus for each candidate and the amount of defocus at the estimated position of each candidate. If the difference is greater than or equal to a predetermined value, the process proceeds to S2015; otherwise, the process terminates.

[0124] In S2015, the tracking target identification unit 207 updates the location from which feature information is acquired based on the estimated position of each candidate. Here, the conditions for the updated location are that the region is "within a predetermined value of the difference in the amount of defocus" and "closest to the original estimated position of each candidate." The conditions for the updated location are not limited to these; for example, the reliability information of the amount of defocus mentioned above may be used to limit the region to one with a certain level of reliability. Alternatively, the estimated width and height information of each BB for each candidate may be used to limit the region to within the BB area.

[0125] As described above, according to the third embodiment, depth information is used to update the appropriate position for acquiring feature quantities of candidate tracking targets. This makes it possible to acquire more accurate feature quantities of each candidate, even when the target being tracked and similar objects are positioned to overlap, thereby suppressing the occurrence of false tracking.

[0126] The disclosures herein include the following information processing devices, control methods, and programs. (Item 1) Image acquisition means for acquiring frame images contained in a video, Depth acquisition means for acquiring depth information relating to the depth in each region within the frame image, Tracking target setting means for setting the tracking target, Candidate detection means for detecting one or more tracking candidates similar to the tracking target from the frame image, A means for identifying the target to be tracked from the one or more tracking candidates, Equipped with, The identification means identifies the tracking target based on the similarity between the tracking target in a past frame image preceding the frame image and each of the one or more tracking candidates in the frame image. The aforementioned similarity is based at least on a first similarity between the depth of the tracking target in the past frame image and the depth of each of the one or more tracking candidates. An information processing device characterized by the following: (Item 2) The aforementioned similarity is based on a second similarity between the position and size of the bounding box (BB) of the target to be tracked in the past frame image and the position and size of the BB of each of the one or more tracking candidates, and a third similarity between the image features of the target to be tracked in the past frame image and the image features of each of the one or more tracking candidates. The information processing device described in item 1, characterized by the features described herein. (Item 3) The first similarity is based on the difference between the depth of the target being tracked in the past frame image and the depth of each of the one or more tracking candidates. An information processing device according to item 1 or 2, characterized by the above. (Item 4) The system further includes reliability acquisition means for acquiring information regarding the reliability of the depth information, The identifying means sets the weight of the first similarity in the calculation of the similarity based on the reliability. An information processing device according to any one of items 1 to 3, characterized by the above. (Item 5) The identifying means sets the weight of the first similarity to zero if the reliability is less than a predetermined value. The information processing device described in item 4, characterized by the features described herein. (Item 6) The system further comprises a state acquisition means for acquiring state information relating to the operation of the imaging device that generated the aforementioned moving image, The identifying means sets the first similarity weight in the similarity calculation based on the operation during the period between the past frame image and the current frame image. An information processing device according to any one of items 1 to 5, characterized by the above. (Item 7) The aforementioned state information is information about the amount of drive of the focus lens in the imaging device. The information processing device described in item 6, characterized by the features described herein. (Item 8) The aforementioned similarity is based at least on a first similarity between the corrected depth, which is obtained by correcting the depth of the tracking target in the past frame image based on the drive amount information, and the depth of each of the one or more tracking candidates. The information processing device described in item 7, characterized by the features described herein. (Item 9) The system further comprises region determination means for determining a subregion in the frame image used to derive image features for each of the one or more tracking candidates, based on the depth of each of the one or more tracking candidates. An information processing device according to any one of items 1 to 8, characterized by the above. (Item 10) The depth information is the amount of defocus obtained by the imaging device that generated the moving image. An information processing device according to any one of items 1 to 9, characterized by the above. (Item 11) A control method for an information processing device that tracks a target object within a moving image, The tracking target setting step involves setting the tracking target, The image acquisition step involves acquiring frame images included in the aforementioned video, A depth acquisition step is to acquire depth information relating to the depth in each region within the frame image, A candidate detection step of detecting one or more tracking candidates similar to the tracking target from the frame image, A selection step to identify the target to be tracked from the one or more tracking candidates, Includes, In the aforementioned specific step, the tracking target is identified based on the similarity between the tracking target in a past frame image preceding the frame image and each of the one or more tracking candidates in the frame image. The aforementioned similarity is based at least on a first similarity between the depth of the tracking target in the past frame image and the depth of each of the one or more tracking candidates. A control method characterized by the following: (Item 12) A program to cause a computer to execute the control method described in item 11.

[0127] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0128] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0129] 201 Input image acquisition unit; 202 Depth information acquisition unit; 203 Tracking target setting unit; 204 Tracking target template generation unit; 205 Candidate detection unit; 206 Feature extraction unit; 207 Tracking target identification unit; 211 Storage unit

Claims

1. Image acquisition means for acquiring frame images contained in a video, Depth acquisition means for acquiring depth information relating to the depth in each region within the frame image, Tracking target setting means for setting the tracking target, Candidate detection means for detecting one or more tracking candidates similar to the tracking target from the frame image, A means for identifying the target to be tracked from the one or more tracking candidates, Equipped with, The identification means identifies the tracking target based on the similarity between the tracking target in a past frame image preceding the frame image and each of the one or more tracking candidates in the frame image. The aforementioned similarity is based at least on a first similarity between the depth of the target being tracked in the past frame image and the depth of each of the one or more tracking candidates. An information processing device characterized by the following:

2. The aforementioned similarity is based on a second similarity between the position and size of the bounding box (BB) of the target to be tracked in the past frame image and the position and size of the BB of each of the one or more tracking candidates, and a third similarity between the image features of the target to be tracked in the past frame image and the image features of each of the one or more tracking candidates. The information processing apparatus according to feature 1.

3. The first similarity is based on the difference between the depth of the target being tracked in the past frame image and the depth of each of the one or more tracking candidates. The information processing apparatus according to feature 1.

4. The system further includes reliability acquisition means for acquiring information regarding the reliability of the depth information, The identifying means sets the weight of the first similarity in the calculation of the similarity based on the reliability. The information processing apparatus according to feature 1.

5. The aforementioned identification means sets the weight of the first similarity to zero if the reliability is less than a predetermined value. The information processing apparatus according to feature 4.

6. The system further comprises a state acquisition means for acquiring state information relating to the operation of the imaging device that generated the aforementioned moving image, The identifying means sets the first similarity weight in the similarity calculation based on the operation during the period between the past frame image and the current frame image. The information processing apparatus according to feature 1.

7. The aforementioned state information is information about the amount of drive of the focus lens in the imaging device. The information processing apparatus according to feature 6.

8. The aforementioned similarity is based at least on a first similarity between the corrected depth, which is obtained by correcting the depth of the tracking target in the past frame image based on the drive amount information, and the depth of each of the one or more tracking candidates. The information processing apparatus according to feature 7.

9. The system further comprises region determination means for determining a subregion in the frame image used to derive the image features of each of the one or more tracking candidates, based on the depth of each of the one or more tracking candidates. The information processing apparatus according to feature 1.

10. The depth information is the amount of defocus obtained by the imaging device that generated the moving image. The information processing apparatus according to feature 1.

11. A control method for an information processing device that tracks a target object within a moving image, The tracking target setting step involves setting the tracking target, The image acquisition step involves acquiring frame images included in the aforementioned video, A depth acquisition step is to acquire depth information relating to the depth in each region within the frame image, A candidate detection step of detecting one or more tracking candidates similar to the tracking target from the frame image, A selection step to identify the target to be tracked from the one or more tracking candidates, Includes, In the aforementioned specific step, the tracking target is identified based on the similarity between the tracking target in a past frame image preceding the frame image and each of the one or more tracking candidates in the frame image. The aforementioned similarity is based at least on a first similarity between the depth of the target being tracked in the past frame image and the depth of each of the one or more tracking candidates. A control method characterized by the following:

12. A program for causing a computer to execute the control method described in claim 11.

Images