Information processing device, information processing method

Abstract

To provide a technology for suppressing erroneous tracking of a tracking target.SOLUTION: An information processing apparatus obtains a first feature of an image of a tracking target, obtains a second feature of an image of a search area, uses the first feature and the second feature to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search area. Then, using the inference tensor, and obtains an inference map representing the position of the tracking target in the image of the search area.SELECTED DRAWING: Figure 2

Description

Technical Field

[0005]

[0001] The present invention relates to a tracking technique for a tracking target in an image.

Background Art

[0002] As techniques for tracking a specific subject in an image, there are those that utilize luminance and color information, template matching, etc. In recent years, techniques using Deep Neural Network (DNN) have attracted attention as highly accurate tracking techniques.

[0003] The technique described in Non-Patent Document 1 is one of the techniques for tracking a specific subject in an image. By inputting an image in which the tracking target appears and an image serving as a search area into a Convolutional Neural Network (CNN) with the same weight, and calculating the cross-correlation of the respective results obtained from the CNN, the position where the tracking target exists in the image of the search area is specified. In such a tracking method, while the position of the tracking target can be accurately predicted, when there is a similar object similar to the tracking target in the image, due to the increase in the cross-correlation value with the similar object, mis-tracking that erroneously tracks the similar object as the tracking target tends to occur.

[0004] In the technique described in Patent Document 1, when a similar object similar to the tracking target exists in the vicinity of the tracking target, an attempt is made to suppress mis-tracking to the similar object by predicting the positions of the tracking target and the similar object respectively. In the technique described in Non-Patent Document 1, the feature amounts of the tracking target and the search image are extracted respectively, and the similarity is calculated for each channel to detect the tracking target.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006] [Non-Patent Document 1] “Fully-Convolutional Siamese Networks for Object Tracking”,arXiv 2016 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, in the method described in Non-Patent Document 1, similarity is calculated independently for each part of the feature quantity, so false tracking occurs when similar objects that have some features similar to the target object, such as contours or colors, are in close proximity. The present invention provides a technique for suppressing false tracking of the target object. [Means for solving the problem]

[0008] One aspect of the present invention is a first calculation means for determining a first feature quantity of an image to be tracked, A second computation means for obtaining the second feature of the image in the search region, A third computation means that uses the first and second features to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search region, A fourth operation means that uses the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, 、 A transformation means that transforms the first feature by combining it with a pre-held feature, It is characterized by having the following features. [Effects of the Invention]

[0009] According to the present invention, it is possible to suppress false tracking of the target object. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A block diagram showing an example of the functional configuration of an information processing device. [Figure 3] Flowchart of the inference process. [Figure 4] Diagram showing details of the process at step S303. [Figure 5] Diagram for explaining the inference map 412. [Figure 6] Flowchart of the learning process of the CNN. [Figure 7] Diagram showing the learning process of the CNN. [Figure 8] Diagram showing a method for generating weight parameters of the CNN. [Figure 9] Diagram showing the operation when performing convolution processing multiple times. [Figure 10] Diagram showing the generation process of the tracking target template feature amount.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] First, an example of the hardware configuration of the information processing apparatus according to the present embodiment will be described using the block diagram of FIG. 1. Note that the hardware configuration shown in FIG. 1 is merely an example, and can be appropriately changed / modified.

[0013] The CPU 101 executes various processes using computer programs and data stored in the ROM 102 and the RAM 103. Thereby, the CPU 101 controls the operation of the entire information processing apparatus, and executes or controls each process described as being performed by the information processing apparatus.

[0014] The ROM 102 stores the setting data of the information processing apparatus, computer programs and data related to the startup of the information processing apparatus, computer programs and data related to the basic operations of the information processing apparatus, and the like.

[0015] The RAM 103 has an area for storing computer programs and data loaded from the ROM 102 or the storage unit 104, and an area for storing information received from an external device via the I / F 107. Further, the RAM 103 has a work area used when the CPU 101 executes various processes. Thus, the RAM 103 can appropriately provide various areas.

[0016] The storage unit 104 is a large-capacity information storage device such as a hard disk drive device. The storage unit 104 stores an OS (Operating System), computer programs and data for causing the CPU 101 to execute or control each process described as a process performed by the information processing apparatus, and the like. The computer programs and data stored in the storage unit 104 are appropriately loaded into the RAM 103 according to the control by the CPU 101 and become processing targets by the CPU 101.

[0017] Note that the storage unit 104 can be implemented by a medium (recording medium) and an external storage drive for realizing access to the medium. As such a medium, for example, a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, etc. are known. Also, the storage unit 104 may be a server device or the like that can be accessed by the information processing apparatus via a network.

[0018] The display unit 105 is a device having a liquid crystal screen, a touch panel screen, an organic EL display, etc., and displays the processing result by the CPU 101 as an image, characters, or the like. Note that the display unit 105 may be a projection device such as a projector that projects an image or characters.

[0019] The operation unit 106 is a user interface such as a keyboard, mouse, or touch panel, and allows the user to input various instructions to the CPU 101 through user operation. The operation unit 106 may also be an external device capable of communicating with the information processing device. Furthermore, the operation unit 106 may be a pen tablet, or the display unit 105 and the operation unit 106 may be combined to form a tablet. Thus, the implementation of the display function and the operation input function is not limited to any specific form.

[0020] I / F107 is a communication interface for data communication with external devices. The CPU101, ROM102, RAM103, storage unit104, display unit105, operation unit106, and I / F107 are all connected to the system bus108.

[0021] Next, an example of the functional configuration of the information processing device is shown in the block diagram of Figure 2. In this embodiment, of the functional units shown in Figure 2, the other units are the storage unit 208 and the other units. function All of these components are assumed to be implemented by computer programs. In the following, the functional components of Figure 2 (excluding the storage unit 208) may be described as the main processing unit, but in reality, the functions of these functional components are realized by the CPU 101 executing the corresponding computer program. Note that the other functional components shown in Figure 2, excluding the storage unit 208, are also described. function Part or all of the unit may be implemented in hardware. The memory unit 208 can be implemented using memory devices such as RAM 103, ROM 102, and memory unit 104.

[0022] Next, the process performed by the information processing device to output information indicating the position of the tracked target in the image as an inference map (inference process) will be explained according to the flowchart in Figure 3. Note that the process according to the flowchart in Figure 3 may be performed by the information processing device alone, or it may be performed by multiple devices, including the information processing device, working in coordination.

[0023] In step S301, the input unit 203 acquires an image containing various objects (people, cars, animals, buildings, trees, etc.). The method of acquiring the image is not limited to a specific method; for example, it may be acquired from the storage unit 208, or from an external device such as an imaging device or server connected via the I / F 107.

[0024] In step S302, the input unit 203 acquires an image within the image region to be tracked from the image acquired in step S301 as the image to be tracked. The method of acquiring the image to be tracked is not limited to a specific method. For example, the image acquired in step S301 may be displayed on the display unit 105, and the user may operate the operation unit 106 to select an image within the image region to be tracked and acquire it as the image to be tracked.

[0025] In step S303, the generation unit 204 performs calculations on the CNN with the target image as input to generate target template features representing the features of the target image as the output of the CNN. The transformation unit 205 then transforms the target template features to match the number of dimensions of the CNN weight parameters used by the detection unit 206.

[0026] In step S304, the input unit 201 acquires an input image and search area information indicating the range (image region) in the input image where the tracking target is searched. The method of acquiring the input image is not limited to a specific method; for example, it may be acquired from the storage unit 208, or from an external device such as an imaging device or server connected via the I / F 107. Similarly, the method of acquiring the search area information is not limited to a specific method; for example, it may be acquired from the storage unit 208, or from an external device such as a server connected via the I / F 107. Alternatively, information defining a region specified by the user operating the operation unit 106 may be acquired as search area information. The extraction unit 202 then acquires an image within the image region indicated by the search area information from the input image as a search image, and generates feature quantities of the search image (search area features) as the output of the CNN by performing calculations on the search image as input.

[0027] In step S305, the detection unit 206 inputs the search region features into a CNN, which has the tracked target template features transformed in step S303 set as weight parameters, and performs calculations on the CNN to generate an inference tensor representing the likelihood that a tracked target exists at each position in the search image, as the output of the CNN. Here, the weight parameters are parameters that represent the weight values ​​between layers in the CNN. The detection unit 206 then performs calculations on the CNN with the inference tensor as input to generate an inference map showing the positions of the tracked targets in the search image, as the output of the CNN.

[0028] The details of the processing in step S303 described above will be explained with reference to Figure 4. The generation unit 204 inputs the tracking target image 401 to the CNN 402 and performs calculations on the CNN 402 to generate the tracking target template feature 403 as the output of the CNN 402. The CNN 402 is a CNN that has been pre-trained to obtain a tracking target template feature that makes it easy to distinguish between tracking targets and non-tracking targets. In this embodiment, the tracking target template feature 405, which is obtained by transforming the tracking target template feature 403 to match the number of dimensions of the weight parameters of the CNN 409, is used as the weight parameter of the CNN 409. Therefore, the CNN 402 is a CNN with weights independent of the CNN 407. The dimension of the weight parameter of the CNN 409 is 4 dimensions, for example, if it is 3x3xCxY (C and Y are natural numbers), the dimension of the tracking target template feature 403 is 3x3xCY.

[0029] The transformation unit 205 transforms the 3x3xCY target template feature 403 into a 3x3xCxY target template feature 405 (transformation 404). Transformation methods include dimensional block partitioning and shuffle partitioning, and the same transformation method is used during inference and training.

[0030] Next, the details of the processing in steps S304 and S305 described above will be explained using Figures 4 and 5. The extraction unit 202 obtains an image within the image region indicated by the search region information from the input image as the search image 406. The extraction unit 202 then performs calculations on the CNN 407 with the search image 406 as input, thereby generating the search region feature quantity 408, which is a feature quantity of the search image 406, as the output of the CNN 407.

[0031] The detection unit 206 sets the target tracking template feature 405 as the weight parameter of the CNN 409. The detection unit 206 then inputs the search region feature 408 into the CNN 409, which has the target tracking template feature 405 set as the weight parameter, and performs an operation (convolution) on the CNN 409 to generate an inference tensor 410 that represents the likelihood that the target tracking exists at each position in the search image 406. Here, while Non-Patent Literature 1 calculates the cross-correlation of each feature of the target tracking image and the search region image by depthwise convolution, in this embodiment the CNN 409 is full convolution. If the dimension of the search region feature 408 is WxHxC and the dimension of the weight parameter of the CNN 409 is 3x3xCxY (W, H, C, and Y are natural numbers), then the dimension of the inference tensor 410 is WxHxY. At this time, the tracking target template feature 405 is trained so that the value of the inference tensor 410 corresponding to a location in the search image 406 with a high likelihood of the tracking target being present becomes large. The detection unit 206 then performs a CNN 411 operation (convolution operation) with the inference tensor 410 as input to generate an inference map 412 that shows the location of the tracking target in the search image 406.

[0032] The inference map 412 will be explained using Figure 5. For each position (each rectangle) in the search image 501, the "likelihood that the target being tracked 502 is located in that rectangle" is obtained. In Figure 5, the likelihood for rectangle 503, which is near the center of the target being tracked 502, is shown to be high. If this likelihood is above a threshold, it can be estimated that the target being tracked 502 is located at the position corresponding to rectangle 503.

[0033] Next, the processes performed by the information processing unit to train CNN402, CNN407, and CNN411, which appeared in Figure 4 (CNN training processes), will be explained according to the flowchart in Figure 6. Note that the processes according to the flowchart in Figure 6 may be executed by the information processing unit alone, or multiple devices, including the information processing unit, may work together to execute them.

[0034] In step S601, the input unit 203 acquires an image of the object to be trained (an object that can be tracked) as the tracking target image. As mentioned in the above explanation related to step S302, the method of acquiring the tracking target image is not limited to a specific method.

[0035] In step S602, the generation unit 204 performs calculations on the CNN 402 with the tracking target image acquired in step S601 as input, thereby generating tracking target template features representing the features of the tracking target image as the output of the CNN 402. The transformation unit 205 then transforms the tracking target template features to match the number of dimensions of the CNN weight parameters used by the detection unit 206.

[0036] In step S603, the input unit 201 acquires an input image and search area information indicating the range (image region) in which objects to be learned (objects that can be tracked) are searched within the input image. The method for acquiring the input image and search area information is not limited to a specific method, as mentioned in the above explanation relating to step S304. The extraction unit 202 acquires an image from the input image within the image region indicated by the search area information as a search image.

[0037] In step S604, the extraction unit 202 performs calculations on the CNN 407, which is input to the search image acquired in step S603, to generate the search region features, which are the features of the search image, as the output of the CNN 407.

[0038] In the flowchart of Figure 6, steps S603 and S604 are executed in parallel with steps S601 and S602. However, the flowchart is not limited to this, and these processes may be executed sequentially.

[0039] In step S605, the detection unit 206 sets the tracked target template features, which were transformed in step S602, as weight parameters for CNN409. The detection unit 206 then inputs the search region features generated in step S604 into CNN409, which has the tracked target template features set as weight parameters, and performs calculations on CNN409 to generate an inference tensor. This inference tensor represents the likelihood that the learning target exists at each position in the search image. The detection unit 206 then performs calculations on CNN411, which has the inference tensor 410 as input, to generate an inference map that shows the position of the learning target in the search image.

[0040] In the CNN training process, the goal is to update the weight parameters of each CNN so that in the inference map, values ​​corresponding to the target location are high, and values ​​at locations other than the target location are low.

[0041] In step S606, the learning unit 207 calculates the loss as the difference between the inference map generated in step S605 and the map (training data) in which "1" is assigned to the positions of the learning target and "0" to positions other than the learning target in the search image. A loss function such as Cross Entropy Loss or Smooth L1 Loss can be used to calculate the loss.

[0042] In step S607, the learning unit 207 updates the weight parameters of CNN402, CNN407, and CNN411 based on the loss obtained in step S606. At this time, transformation 404 performs the inverse transformation of the transformation in the above inference process and propagates the error to CNN402. The weight parameters of CNN409 do not need to be updated as the target template feature 405 is set. The weight parameter updates are performed based on backpropagation using methods such as Momentum SGD.

[0043] In this embodiment, for the sake of simplicity, the output of the loss function for a single image was described. However, when dealing with multiple images, the loss is calculated for the estimated scores of those multiple images. Then, the interlayer weight parameters of the CNN are updated so that the losses for all of the multiple images are smaller than a predetermined threshold.

[0044] In step S608, the learning unit 207 stores the weight parameters (weight parameters for CNN402, CNN407, and CNN411) updated in step S607 into the memory unit 208. Then, when using CNN402 in the inference process described above, CNN402 with the "CNN402 weight parameters" stored in the memory unit 208 is used. Similarly, when using CNN407 in the inference process described above, CNN407 with the "CNN407 weight parameters" stored in the memory unit 208 is used. Similarly, when using CNN411 in the inference process described above, CNN411 with the "CNN411 weight parameters" stored in the memory unit 208 is used.

[0045] In step S609, the learning unit 207 determines whether the learning termination conditions have been met. The learning termination conditions are not limited to specific conditions. Possible learning termination conditions include, for example, "the loss obtained in step S606 is less than or equal to a threshold," "the rate of change of the loss obtained in step S606 is less than or equal to a threshold," and "the number of repetitions of the processing in steps S601 to S608 is greater than or equal to a threshold."

[0046] Various modifications of this embodiment will be described below. In the description of each modification, the differences from the first embodiment will be explained, and unless otherwise specified, it will be assumed to be the same as the first embodiment.

[0047] <Example 1> In the first embodiment, during the CNN training process, the input unit 203 acquired images of the object to be trained (objects that can be tracked) as tracked images (positive examples). In this modified example, during the CNN training process, in addition to the tracked images, the input unit 203 acquires images of similar objects that are similar to the object to be trained as non-tracked images (negative examples).

[0048] The learning process of the CNN related to this modified example will be explained using Figure 7. The input unit 203 acquires the non-tracking image 701 in addition to the tracking target image 401. The method for acquiring the non-tracking image 701 is not limited to a specific method, similar to the method for acquiring the tracking target image 401.

[0049] The generation unit 204 then performs calculations on the CNN 402, which is input to a concatenated image obtained by concatenating the tracking target image 401 and the non-tracking target image 701, thereby generating a tracking target template feature quantity 403 corresponding to the concatenated image as the output of the CNN 402.

[0050] Furthermore, the non-tracking image 701 is not limited to images of similar objects that are similar to the object being trained; it can be any image of an object different from the object being trained, such as an image showing only the background. By doing so, features of similar objects and backgrounds can be learned as negative examples, thereby suppressing false tracking.

[0051] <Modification 2> In the first embodiment, the deformation unit 205 deformed the tracking target template features to match the number of dimensions of the CNN weight parameters used by the detection unit 206 and set them as the CNN weight parameters. In this modified example, the deformation unit 205 generates the CNN weight parameters by combining the tracking target template features with pre-prepared features. The method for generating the CNN weight parameters according to this modified example will be explained with reference to Figure 8.

[0052] The generation unit 204 inputs the tracking target image 401 into the CNN 402 and performs calculations on the CNN 402 to generate the tracking target template feature 803 as the output of the CNN 402. The transformation unit 205 concatenates (combines) the tracking target template feature 803 and the feature 806 that has been previously stored in the storage unit 208 to generate the tracking target template feature 805 (transformation 804). Instead of "concatenation (combination)", "reshape addition" may be applied. Here, the feature 806 is a weight that has been previously learned to detect the tracking target. In the example in Figure 8, some dimensions of the feature 806 are replaced with the tracking target template feature 803. For example, if the tracking target template feature 803 has dimensions of 3x3x16 and feature 806 has dimensions of 3x3x16x16, the transformation unit 205 replaces some of the dimensions of feature 806 with the tracking target template feature 803 to generate a tracking target template feature 805 with dimensions of 3x3x16x16. The memory unit 208 stores feature 806 for each category of object, such as people, cars, and animals. The transformation unit 205 retrieves the feature 806 corresponding to the category of the target object from the memory unit 208 and concatenates (combines) it with the tracking target template feature 803. In this way, by combining the weights calculated from the tracking target image 401 with weights that have been sufficiently trained in advance, the tracking accuracy can be improved.

[0053] <Variation 3> In Modification 2, CNN402 and CNN407 were separate CNNs with different weight parameters, but the same CNN may be applied to CNN402 and CNN407. By applying the same CNN to CNN402 and CNN407, for example, the image 401 to be tracked can be searched. image When using 406, the tracking target template feature 803 and the search area feature 408 are the same. Therefore, it is not necessary to calculate both the tracking target template feature 803 and the search area feature 408 (calculating one of them is sufficient), which speeds up the overall processing. Furthermore, since CNN409 incorporates the depthwise convolution described in Non-Patent Document 1, it can achieve tracking accuracy higher than that described in Non-Patent Document 1.

[0054] <Modification 4> In the first embodiment, the CNN operation (convolution) using weight parameters based on the tracking target template features calculated from the tracking target image 401 was performed only once, but it may be performed multiple times. The operation when such convolution is performed multiple times will be explained with reference to Figure 9.

[0055] The generation unit 204 inputs the tracking target image 401 into the CNN 902 and performs calculations on the CNN 902 to generate tracking target template features 403 and tracking target template features 903 as outputs of the CNN 902. Here, the dimensions of the search region feature 408 are WxHxCh, the dimensions of the tracking target template feature 403 are 3x3x(ChxY), the dimensions of the tracking target template feature 405 are 3x3xChxY, and the dimensions of the tracking target template feature 903 are 3x3x(YxOUT).

[0056] The deformation unit 205 deforms the target template feature 903 to match the number of dimensions of the weight parameters of CNN411 (deformed 904) to generate a target template feature 905 with dimensions of 3x3xYxOUT.

[0057] The detection unit 206 inputs an inference tensor 410 with dimensions WxHxY into a CNN 911, which has the tracking target template feature quantity 905 set as weight parameters, and performs operations on the CNN 911 to generate an inference map 412 with dimensions WxHxOUT.

[0058] In the training process, CNN911 does not update its weight parameters, similar to CNN409. By generating multiple weights and performing multiple stages of convolution in this way, it becomes possible to perform complex and non-linear processing according to the target image 401, thereby improving detection accuracy.

[0059] <Modification 5> In the first embodiment, the generation unit 204 generated tracking target template features only from the image acquired by the input unit 203. However, it is also possible to generate tracking target template features using previously generated tracking target template features in addition to the image. The tracking target template feature generation process according to this modified example will be explained with reference to Figure 10.

[0060] The generation unit 204 inputs the tracking target image 401 into the CNN 1002, performs calculations in the CNN 1002 to generate the tracking target template feature 1003 (dimensions: 3x3x16), and inputs the generated tracking target template feature 1003 into the CNN 1006. The generation unit 204 also reads the previously obtained tracking target template feature 1005 (dimensions: 3x3x(16xY)) from the storage unit 208 and inputs it into the CNN 1006. Then, the generation unit 204 performs calculations in the CNN 1006, which has the tracking target template feature 1003 and tracking target template feature 1005 as inputs, to generate the tracking target template feature 1007 (dimensions: 3x3x(16xY)).

[0061] The deformation unit 205 generates a target template feature 405 (dimensions: 3x3x16xY) by deforming the target template feature 1007 to match the number of dimensions of the weight parameters of CNN409 (deformation 404).

[0062] Alternatively, instead of directly inputting the previously obtained tracking target template feature 1005 into CNN1006, you could input the moving average of the previously obtained tracking target template feature 1005 into CNN1006.

[0063] Alternatively, GRU (Gated Recurrent Unit) or LSTM (Long Short-Term Memory) can be used instead of CNN1006.

[0064] Alternatively, instead of inputting the tracking target template feature 1005 into CNN1006, the tracking target template feature 1003 could be input and integrated into the calculation result of CNN1006 to generate the tracking target template feature 1007. Integration methods include calculating a moving average or concatenating and performing convolution. This approach helps to suppress a decrease in detection accuracy even when there are changes in the appearance of the tracking target or occlusion.

[0065] Thus, according to the first embodiment and its modifications, even when a similar object similar to the target being tracked is in close proximity to the target, false tracking, where the system tracks the similar object, can be suppressed.

[0066] [Second Embodiment] The information processing device described in the first embodiment and its modifications may be a PC (personal computer), a tablet device, a smartphone, or other device. In that case, the information processing device may consist of one such device or multiple such devices. In the latter case, each device does not have to have the same configuration, and in that case, the information processing device may be a collection of devices, each with its own role, such as one or more devices that execute processing according to the flowchart described above, and a device that functions as storage.

[0067] Furthermore, while the first embodiment and its variations describe a configuration in which the inference process, CNN training process, and the process are performed on the same device, each process may be performed on separate devices. In this case, the device performing the inference process uses the CNN generated by the device that performed the CNN training process.

[0068] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) structure (including number of dimensions), acquisition method, destination, source, and storage location used in each of the above embodiments and modifications are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0069] Furthermore, some or all of the embodiments and modified examples described above may be used in appropriate combinations. Alternatively, some or all of the embodiments and modified examples described above may be used selectively.

[0070] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0071] The inventions described herein include the following systems, information processing devices, information processing methods, and computer programs.

[0072] (Item 1) A first computation means for determining the first feature quantity of the image to be tracked, A second computation means for obtaining the second feature of the image in the search region, A third computation means that uses the first and second features to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search region, A fourth operation means that uses the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, An information processing device characterized by comprising:

[0073] (Item 2) moreover, The information processing apparatus according to item 1, characterized by comprising a transformation means for transforming the first feature quantity into a feature quantity with the same dimensionality as the third calculation means.

[0074] (Item 3) moreover, The information processing apparatus according to item 1, characterized in that it includes a deformation means for deforming the first feature by combining it with a feature that is held in advance.

[0075] (Item 4) moreover, The information processing apparatus according to item 1, characterized by comprising a transformation means for transforming a feature quantity obtained based on the first feature quantity and the first feature quantity previously determined by the first calculation means.

[0076] (Item 5) The first, second, third, and fourth computation means are Convolutional Neural Networks (CNNs), The third computation means is a CNN in which the first feature transformed by the transformation means is set as a weight parameter, and the inference tensor is obtained using the second feature as input. An information processing device according to any one of items 2 to 4, characterized by the above.

[0077] (Item 6) The information processing device according to item 5, characterized in that it is a CNN different from the first and second calculation means.

[0078] (Item 7) The information processing apparatus according to item 5, characterized in that the first calculation means and the second calculation means are the same CNN.

[0079] (Item 8) The information processing device according to any one of items 1 to 7, characterized in that the fourth calculation means uses the third feature quantity of the image of the target to be tracked obtained by the first calculation means and the inference tensor to obtain an inference map representing the position of the target to be tracked in the image of the search region.

[0080] (Item 9) moreover, An information processing device according to any one of items 1 to 8, characterized by comprising: an inference tensor representing the likelihood of the presence of a learning target at each position in an image of a search region, obtained using a first feature of the image of the learning target and a second feature of the image of the search region; an inference map representing the position of the learning target in the image of the search region, obtained using the inference tensor; and a learning means for training the first, second, third, and fourth calculation means using the inference map.

[0081] (Item 10) moreover, An information processing device according to any one of items 1 to 8, characterized by comprising: an inference tensor representing the likelihood of a target being present at each position in an image of a search region, obtained using a first feature of a concatenated image of the target being to be learned and an image of a different target, and a second feature of the image of the search region; an inference map representing the position of the target being learned in the image of the search region, obtained using the inference tensor; and a learning means for learning the first, second, third, and fourth calculation means using the inference map.

[0082] (Item 11) An information processing method performed by an information processing device, The first calculation means of the information processing device performs a first calculation step of determining a first feature quantity of the image to be tracked, The second calculation means of the information processing device performs a second calculation step of determining a second feature quantity of the image of the search region, The third processing means of the information processing device performs a third calculation step of obtaining an inference tensor representing the likelihood that the tracking target exists at each position of the image in the search region, using the first feature and the second feature, The fourth arithmetic means of the information processing device performs a fourth arithmetic step of using the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region. An information processing method characterized by comprising:

[0083] (Item 12) A computer program that causes a computer to function as one of the information processing devices described in any one of items 1 through 10.

[0084] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0085] 201: Input Unit 202: Extraction Unit 203: Input Unit 204: Generation Unit 205: Transformation Unit 206: Detection Unit 207: Learning Unit 208: Memory Unit

Claims

1. A first calculation means for determining the first feature quantity of the image to be tracked, A second computation means for obtaining the second feature quantity of the image in the search region, A third computation means that uses the first and second features to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search region, A fourth calculation means that uses the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, A transformation means that transforms the first feature by combining it with a pre-held feature, An information processing device characterized by comprising:

2. A first calculation means for determining a first feature quantity of the image to be tracked, A second computation means for obtaining the second feature quantity of the image in the search region, A third computation means that uses the first and second features to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search region, A fourth calculation means that uses the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, A transformation means for transforming a feature obtained based on the aforementioned first feature and the first feature previously obtained by the aforementioned first calculation means. An information processing device characterized by comprising:

3. The information processing apparatus according to claim 1 or 2, characterized in that the transformation means transforms the first feature quantity into a feature quantity of the third calculation means with respect to the number of dimensions.

4. The first calculation means, the second calculation means, the third calculation means, and the fourth calculation means are Convolutional Neural Network (CNN), The third calculation means is a CNN in which the first feature quantities transformed by the transformation means are set as weight parameters, and the inference tensor is obtained by performing a convolution operation of the CNN with the second feature quantities as input. The aforementioned CNN is a full convolution. The information processing apparatus according to claim 1 or 2.

5. The information processing apparatus according to claim 4, characterized in that the first calculation means and the second calculation means are different CNNs.

6. The information processing apparatus according to claim 4, characterized in that the first calculation means and the second calculation means are the same CNN.

7. A first calculation means for determining a first feature quantity of the image to be tracked, A second computation means for obtaining the second feature quantity of the image in the search region, A third computation means that uses the first and second features to obtain an inference tensor representing the likelihood that the tracking target exists at each position in the image of the search region, A fourth calculation means obtains an inference map representing the position of the tracked target in the image of the search region using the third feature quantity of the tracked target image obtained by the first calculation means and the inference tensor. An information processing device characterized by comprising:

8. moreover, The information processing apparatus according to claim 1 or 2, comprising: an inference tensor representing the likelihood of the presence of a learning target at each position in the image of the search region, obtained using a first feature of the image of the learning target and a second feature of the image of the search region; an inference map representing the position of the learning target in the image of the search region, obtained using the inference tensor; and a learning means for performing learning of the first, second, third, and fourth calculation means using the inference map.

9. moreover, The information processing apparatus according to claim 1 or 2, comprising: an inference tensor representing the likelihood of a target being present at each position in the image of the search region, obtained using a first feature of a concatenated image of the image of the target and an image of a different target, and a second feature of the image of the search region; an inference map representing the position of the target in the image of the search region, obtained using the inference tensor; and a learning means for performing learning of the first, second, third, and fourth calculation means using the inference map.

10. An information processing method performed by an information processing device, The first calculation means of the information processing device performs a first calculation step of determining a first feature quantity of the image to be tracked, The second calculation means of the information processing device performs a second calculation step of determining a second feature quantity of the image of the search region, The third processing means of the information processing device performs a third processing step of obtaining an inference tensor representing the likelihood that the tracking target exists at each position of the image in the search region, using the first feature and the second feature, The fourth arithmetic means of the information processing device performs a fourth arithmetic step of using the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, The deformation means of the information processing device includes a deformation step of deforming the first feature quantity by combining it with a feature quantity that has been previously held, and An information processing method characterized by comprising:

11. An information processing method performed by an information processing device, The first calculation means of the information processing device performs a first calculation step of determining a first feature quantity of the image to be tracked, The second calculation means of the information processing device performs a second calculation step of determining a second feature quantity of the image of the search region, The third processing means of the information processing device performs a third processing step of obtaining an inference tensor representing the likelihood that the tracking target exists at each position of the image in the search region, using the first feature and the second feature, The fourth arithmetic means of the information processing device performs a fourth arithmetic step of using the inference tensor to obtain an inference map representing the position of the tracked target in the image of the search region, The deformation means of the information processing device includes a deformation step of deforming a feature quantity obtained based on the first feature quantity and the first feature quantity previously determined by the first calculation means. An information processing method characterized by comprising:

12. An information processing method performed by an information processing device, The first calculation means of the information processing device performs a first calculation step of determining a first feature quantity of the image to be tracked, The second calculation means of the information processing device performs a second calculation step of determining a second feature quantity of the image of the search region, The third processing means of the information processing device performs a third processing step of obtaining an inference tensor representing the likelihood that the tracking target exists at each position of the image in the search region, using the first feature and the second feature, The fourth calculation means of the information processing device performs a fourth calculation step in which it uses the third feature quantity of the image of the target to be tracked obtained in the first calculation step and the inference tensor to obtain an inference map representing the position of the target to be tracked in the image of the search region. An information processing method characterized by comprising:

13. The inference tensor is obtained by inputting the second feature into a CNN that uses the first feature, which has been deformed by the deformation means, as weight parameters, and performing a convolution operation. The information processing method according to any one of claims 10 to 12, characterized in that the CNN is a full convolution.

14. A computer program for causing a computer to function as one of the means of an information processing device according to any one of claims 1, 2, or 7.

Images