Object tracking method and recording medium

Abstract

An object tracking method and a recording medium. The method comprises: a prediction step (S1), inputting time series data of posture into a posture predictor, predicting the distribution of each predicted posture position of the first object in the current frame, i.e., a predicted distribution; an estimation step (S2), inputting the current frame into a posture estimation unit, estimating the set of positions of each second object shown in the current frame; an output step (S3), matching the set of positions with the predicted distribution, obtaining and outputting identification information indicating which of the first objects the second object corresponds to, and the position information of the second object; and an update step (S4), updating the coordinates of each posture of the first object obtained from the predicted distribution predicted in the prediction step (S1) based on the identification information and position information obtained in the output step (S3), and adding them to the time series data of the posture.

Description

Technical Field

[0001] The present disclosure relates to an object tracking method and a recording medium. Background Art

[0002] Among object tracking technologies for tracking multiple target objects shown in an image, posture tracking technologies for tracking the postures of multiple people in an image have attracted attention. In addition, in posture tracking technologies, a two-stage solution is often used (for example, refer to non-patent document 1). Here, the two-stage solution is the following solution: 1) using deep CNN (Convolutional Neural Network) to estimate the posture of each frame, and 2) performing greedy bipartite matching for each frame.

[0003] In the posture tracking technology disclosed in non-patent document 1, the posture is estimated from the current frame of the input image, and the coordinates of the posture of the past frame are shifted according to the posture and optical flow of the past frame, thereby predicting the posture in the current frame. Then, the posture estimated from the current frame and the posture predicted from the past frame are matched, and the same ID is assigned to the matched pair. In addition, in this matching, the object key point similarity (OKS: Object Keypoint Similarity) of the estimated posture and the predicted posture is used.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: B. Xiao, H. Wu, and Y. Wei, "Simple baselines for human poseestimation and tracking," in European Conference on Computer Vision, 2018. Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the posture tracking technology disclosed in non-patent document 1 has the problem of being vulnerable to errors in posture prediction. For example, there are situations where the postures of multiple characters in an image change at a high speed, or disappear or reappear due to occlusion, etc., causing interference in the image. In the posture tracking technology disclosed in non-patent document 1, if such interference occurs, the posture in the current frame is incorrectly estimated and cannot be correctly matched with the posture predicted from past frames, etc. That is, in the posture tracking technology disclosed in non-patent document 1, if such interference occurs, it will lead to posture tracking errors, that is, tracking errors.

[0009] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an object tracking method and a recording medium capable of suppressing tracking errors.

[0010] Means for solving problems

[0011] In order to solve the above-mentioned problems, an object tracking method of one embodiment of the present invention is an object tracking method of an image performed by a computer, comprising: a prediction step, inputting time series data of a posture into a posture predictor, so that the posture predictor predicts the distribution of each predicted posture position of one or more first objects in a current frame, that is, a predicted distribution, the time series data of the posture including coordinates representing the postures of one or more objects respectively appearing in two or more past frames consecutive in the time series, the posture predictor including a first neural network and performing posture prediction taking uncertainty into account; an estimation step, inputting the current frame into a second neural network, so that the second neural network estimates the position of one or more second objects appearing in the current frame. A set of respective positions; an output step, by matching the set of respective positions of the one or more second objects with the predicted distribution, obtaining and outputting identification information and position information, wherein the identification information indicates which of the one or more first objects the one or more second objects corresponds to, and the position information is a position indicating the posture of each of the one or more second objects; and an updating step, based on the identification information and the position information obtained in the output step, appending data including coordinates indicating the posture of each of the one or more first objects obtained from the predicted distribution predicted in the prediction step to the time series data of the posture, thereby updating the time series data of the posture.

[0012] In addition, some of these specific aspects may be implemented using systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or may be implemented using any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0013] Effects of the Invention

[0014] According to the object tracking method and the like of the present disclosure, tracking errors can be suppressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a diagram showing an object tracking device in an embodiment.

[0016] Figure 2 This is a diagram showing an example of the hardware configuration of a computer that realizes the functions of the object tracking device in the embodiment by software.

[0017] Figure 3 This is a block diagram showing an example of the structure of an object tracking device in an embodiment.

[0018] Figure 4 Yes means Figure 3 A block diagram showing an example of a detailed structure of a posture predictor shown in FIG.

[0019] Figure 5 Yes means Figure 3 A diagram of an example of the architecture of a posture predictor is shown.

[0020] Fig. 6A : is a diagram showing the posture estimation result of the current frame in the comparative example.

[0021] Figure 6B : is a diagram showing the posture estimation result of the current frame in the comparative example.

[0022] Figure 6C : is a diagram showing the posture estimation result of the current frame in the comparative example.

[0023] Fig. 7A This is a diagram showing the posture prediction result of the current frame in Example 1.

[0024] Figure 7B This is a diagram showing the posture prediction result of the current frame in Example 1.

[0025] Figure 7C This is a diagram showing the posture prediction result of the current frame in Example 1.

[0026] Figure 8 This is a flowchart showing the operation of the object tracking device in the embodiment.

[0027] Fig. 9 This is a diagram showing an example of pseudo code of the object tracking method in Example 2.

[0028] Fig.10 This is a diagram conceptually showing the object tracking method in the second embodiment.

[0029] Fig.113 is a diagram showing the evaluation results of the object tracking performance in Example 3.

[0030] Fig.12 This is a diagram showing the results of a performance evaluation experiment when components of the posture predictor in Example 3 are disabled.

[0031] Fig.13 : is a figure which visually shows the object tracking result based on the object tracking method in this disclosure and the comparative example.

[0032] Fig.14 : is a figure which visually shows the object tracking result based on the object tracking method in this disclosure and the comparative example.

[0033] Description of Reference Numerals

[0034] 10 Object Tracking Device

[0035] 11 Pose Predictor

[0036] 12 Pose Estimation Unit

[0037] 13 Matching

[0038] 14 Output section

[0039] 15 Data Update Department

[0040] 111 Input processing unit

[0041] 112 1st Neural Network

[0042] 113 Circular NN

[0043] 114 Fully coupled NN

[0044] 115 Output Processing Unit DETAILED DESCRIPTION

[0045] An object tracking method according to one embodiment of the present invention is an image object tracking method performed by a computer, comprising: a prediction step, inputting time series data of a posture into a posture predictor, so that the posture predictor predicts the distribution of the predicted posture positions of one or more first objects in a current frame, that is, the predicted distribution, wherein the time series data of the posture includes coordinates representing the postures of one or more objects respectively appearing in two or more past frames consecutive in the time series, and the posture predictor includes a first neural network and performs posture prediction taking uncertainty into account; an estimation step, inputting the current frame into a second neural network, so that the second neural network estimates the positions of one or more second objects respectively appearing in the current frame. an output step, in which identification information and position information are obtained and output by matching the set of positions of each of the one or more second objects with the predicted distribution, wherein the identification information indicates which of the one or more first objects each of the one or more second objects corresponds to, and the position information indicates the position of the posture of each of the one or more second objects; and an updating step, in which data including coordinates indicating the posture of each of the one or more first objects obtained from the predicted distribution predicted in the prediction step are appended to the time series data of the posture based on the identification information and the position information obtained in the output step, thereby updating the time series data of the posture.

[0046] This makes it possible to implement an object tracking method that can suppress tracking errors.

[0047] Here, for example, it may also be that in the output step, by matching the set of positions of each of the one or more second objects with the predicted distribution, a group consisting of the predicted distribution and the set of positions having a matching score greater than a specified value is calculated, and the identification information and the position information are obtained and output based on the calculated group.

[0048] In addition, for example, in the output step, the group consisting of the predicted distribution and the set of positions having a matching score greater than a specified value is calculated by performing bipartite graph matching on the set of positions of the one or more second objects and the predicted distribution.

[0049] In addition, for example, in the prediction step, the first neural network uses one or more models for taking the uncertainty into consideration, and enables the one or more models to predict the predicted posture positions of one or more of the first objects in the current frame based on the time series data of the posture, thereby predicting the predicted distribution.

[0050] In addition, for example, the predicted distribution may be a distribution that takes the uncertainty into consideration and may be a distribution that includes the predicted predicted posture position and is expressed by a variance in a Gaussian distribution.

[0051] In addition, for example, the first neural network may include a recurrent neural network and a predetermined neural network that forms the one or more models.

[0052] In addition, for example, the one or more models may be formed by applying dropout to the predetermined neural network to deactivate one or more nodes constituting the predetermined neural network.

[0053] Hereinafter, an object tracking method according to one embodiment of the present invention will be described in detail with reference to the accompanying drawings. In addition, the embodiments described below all represent a specific example of the present invention. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements, etc. shown in the following embodiments are examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not recorded in the independent technical solutions representing the highest-level concepts are described as arbitrary constituent elements. In addition, in all embodiments, the various contents may also be combined.

[0054] (Implementation Method)

[0055] Hereinafter, an object tracking method and the like of the object tracking device 10 according to the embodiment will be described.

[0056] [Object tracking device 10]

[0057] Figure 1 1 is a diagram showing an object tracking device 10 according to the present embodiment.

[0058] In the object tracking device 10 of the present embodiment, an image, i.e., a current frame showing one or more objects as tracking targets, is input together with the time series data of posture. Then, the object tracking device 10 outputs the tracking result in the current frame including the identification information of each of the one or more objects as tracking targets and the position information representing the position of the posture of each of the objects. In addition, the object tracking device 10 also outputs the updated time series data of posture. As described later, the time series data of posture includes the coordinates representing the posture of each of the objects shown in two or more past frames that are continuous in the time series. In addition, when the object tracking device 10 is input with the next current frame together with the updated time series data of posture, it outputs the tracking result in the same manner. In this way, the object tracking device 10 tracks the posture of one or more objects shown in the input image. In the following, a person is cited as an example of an object for explanation, but it is not limited to this. It can be an animal with bones and posture, and as long as the key points described later can be detected, it is not limited to animals, and can be a living thing or an object.

[0059] [Hardware structure]

[0060] Next, use Figure 2 The hardware configuration of the object tracking device 10 in this embodiment will be described. Figure 2 1 is a diagram showing an example of the hardware configuration of a computer 1000 that realizes the functions of the object tracking device 10 in the embodiment by software.

[0061] like Figure 2 As shown, the computer 1000 is a computer including an input device 1001, an output device 1002, a CPU 1003, a built-in memory 1004, a RAM 1005, a reader 1007, a transceiver 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the built-in memory 1004, the RAM 1005, the reader 1007, and the transceiver 1008 are connected via the bus 1009.

[0062] The input device 1001 is a device that serves as a user interface such as an input button, a touch pad, or a touch panel display, and accepts user operations. In addition to accepting user contact operations, the input device 1001 may also be a structure that accepts operations using voice or remote operations using a remote controller.

[0063] The built-in memory 1004 is a flash memory, etc. In addition, the built-in memory 1004 may store in advance at least one of a program for realizing the functions of the object tracking device 10 and an application program utilizing the functional configuration of the object tracking device 10 .

[0064] RAM 1005 is a random access memory (Random Access Memory) and is used to store data and the like when a program or an application is executed.

[0065] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory, etc. The reading device 1007 reads the above-mentioned program and application from a recording medium recording the program and application, and stores the program and application in the built-in memory 1004 .

[0066] The transceiver 1008 is a communication circuit for communicating wirelessly or wired. The transceiver 1008 communicates with, for example, a server device connected to a network, downloads the above-mentioned programs and application programs from the server device, and stores them in the built-in memory 1004 .

[0067] The CPU 1003 is a central processing unit (Central Processing Unit), which copies the programs and application programs stored in the built-in memory 1004 to the RAM 1005, and sequentially reads out and executes the commands included in the programs and application programs from the RAM 1005.

[0068] [Structure of Object Tracking Device 10]

[0069] Next, the configuration of the object tracking device 10 in this embodiment will be described.

[0070] Figure 3 1 is a block diagram showing an example of the configuration of the object tracking device 10 according to the present embodiment.

[0071] The object tracking device 10 tracks the posture of one or more objects shown in the input image. Figure 3 As shown, the object tracking device 10 includes a posture predictor 11, a posture estimation unit 12, a matching unit 13, an output unit 14, and a data update unit 15. Hereinafter, each component will be described in detail.

[0072] <Posture Predictor 11>

[0073] The posture predictor 11 is composed of an architecture including a first neural network, and performs posture prediction taking uncertainty into consideration. The details of the architecture will be described later.

[0074] When the time series data of the posture is input, the posture predictor 11 predicts the distribution of the predicted posture positions of one or more first objects in the current frame, that is, the predicted distribution, wherein the time series data of the posture includes coordinates representing the postures of one or more objects respectively shown in two or more past frames continuous in the time series. Here, the predicted distribution predicted by the posture predictor 11 is a distribution that takes into account at least one of the two uncertainties described later, and is a distribution that includes the predicted predicted posture positions and is expressed as a variance in a Gaussian distribution. In addition, the time series data of the posture may include identifiers (IDs) for identifying one or more objects respectively in addition to the coordinates.

[0075] In this embodiment, the posture of a person as an object in a frame is predicted as a collection of locations of joints of the person in the frame. In addition, the joints are, for example, the head, shoulders, fingertips, elbows, wrists, etc., which are called key points. The time series data of the posture is maintained as a data structure such as a queue and is input to the posture predictor 11 through the data update unit 15 described later.

[0076] The posture predictor 11 predicts on which side of the current frame the positions of the key points constituting the postures of one or more characters in the current frame are located using a distribution (i.e., a distribution of the positions of the key points) based on the time series data of the postures. Such a distribution of the positions of the key points is equivalent to the above-mentioned predicted distribution. In this way, the posture predictor 11 does not predict where the positions of the key points constituting the postures of the characters in the current frame are located (one hypothesis), but predicts on which side (in other words, multiple hypotheses) the positions of the key points constituting the postures of the characters in the current frame are located in the form of a distribution (i.e., a predicted distribution).

[0077] <Posture Estimation Unit 12>

[0078] When the current frame is input, the posture estimation unit 12 estimates a set of positions of zero or more second objects shown in the current frame. In addition, the posture estimation unit 12 may not output or estimate when no object is shown.

[0079] In this embodiment, the RGB image of the current frame is input to the posture estimation unit 12, and the set of key points constituting the posture of one or more people shown in the current frame is estimated. In addition, the posture estimation unit 12 stops at being able to estimate the posture of certain people in the current frame. In other words, although the posture of the person shown in the current frame is estimated by the posture estimation unit 12, it is unknown who the person of the estimated posture is (no ID is given to the person of the estimated posture).

[0080] In addition, the posture estimation unit 12 is composed of a structure including a second neural network. Then, the posture estimation unit 12 inputs the current frame obtained from the image into the second neural network, so that the second neural network estimates the set of positions of each of the one or more second objects shown in the current frame. In the present embodiment, the second neural network is the deepCNN disclosed in non-patent document 1. In addition, the second neural network can be a DNN (Deep Neural Network) of any structure as long as it can estimate the set of positions of key points constituting the postures of one or more people shown in the current frame based on the current frame.

[0081] <Matching section 13>

[0082] The matching unit 13 matches the set of positions of the one or more second objects estimated by the posture estimation unit 12 with the predicted distribution predicted by the posture predictor 11. The matching unit 13 obtains and outputs identification information indicating which of the one or more first objects the one or more second objects correspond to, and coordinates indicating the posture of the one or more second objects, i.e., position information, through matching. More specifically, the matching unit 13 calculates a group consisting of a set of predicted distributions and positions having a matching score greater than a specified value through matching, and obtains and outputs identification information and position information based on the calculated group. The matching unit 13 may also calculate the group by performing bipartite graph matching on the set of positions of the one or more second objects estimated by the posture estimation unit 12 and the predicted distribution predicted by the posture predictor 11.

[0083] In the present embodiment, the matching unit 13 uses the object key point similarity (OKS) to perform bipartite graph matching on the distribution of the key point positions predicted by the posture predictor 11 and the set of key point positions estimated by the posture estimation unit 12. Here, the object key point similarity (OKS) refers to an evaluation index used to represent the similarity between the estimated posture and the correct posture. The object key point similarity (OKS) is a value representing the average similarity between the estimated coordinates and the correct coordinates of the annotated joint points, and represents a value of 1 when the estimated posture of the character is completely consistent with the correct posture. In addition, bipartite graph matching is also called bipartite graph matching, which is a method of finding multiple groups that are valid pairs between two categories each containing multiple elements.

[0084] More specifically, the matching unit 13 calculates the matching score using the object key point similarity (OKS) for all the groups of the distribution of the key point positions predicted by the posture predictor 11 and the set of the key point positions estimated by the posture estimation unit 12. The matching unit 13 obtains the group with the matching score exceeding the threshold value among all the groups as a valid group. The group with the matching score exceeding the threshold value is composed of the pairing of the key point position predicted by the posture predictor 11 and the key point position in the distribution and the key point position in the set estimated by the posture estimation unit 12.

[0085] The matching unit 13 obtains position information indicating the posture of one or more persons shown in the estimated current frame based on the group of matching scores that are greater than the threshold value, and assigns an ID to the person in the estimated posture. In addition, the position information indicating the posture of one or more persons is, for example, the position coordinates of a plurality of key points constituting the posture of one or more persons. In addition, the information related to the ID assigned to the person in the estimated posture is equivalent to the above-mentioned identification information.

[0086] Then, the matching unit 13 outputs the acquired position information and identification information to the output unit 14 .

[0087] <Output section 14>

[0088] The output unit 14 reflects the position information and identification information output from the matching unit 13 in the current frame and outputs the tracking result. In addition, the output unit 14 outputs the position information and identification information output from the matching unit 13 to the data updating unit 15 .

[0089] <Data updating section 15>

[0090] The data updating unit 15 adds data including coordinates indicating the posture of each of the one or more first objects obtained from the predicted distribution predicted by the posture predictor 11 to the posture time series data based on the acquired identification information and position information, thereby updating the posture time series data.

[0091] In this embodiment, the data update unit 15 adds only the objects with good probability to the time series data of the posture based on the matching result of the matching unit 13 in the predicted distribution predicted by the posture predictor 11. Here, as described above, the time series data of the posture is maintained in a data structure such as a queue. Therefore, the data update unit 15 deletes the earliest data of the time series data of the posture by adding (pushing) only the objects with good probability in the predicted distribution to the time series data of the posture, thereby being able to update the time series data of the posture.

[0092] In more detail, the data update unit 15 may also use a particle filter to update the time series data of the posture. Here, the particle filter is also called a particle filter or a sequential Monte Carlo method, which is a prediction method for time series data based on probability distribution. The particle filter is also understood as a Monte Carlo approximation of a Bayesian filter. Here, it is sufficient to capture the particle (or particle) as a posture consisting of a set of key points. Therefore, when the data update unit 15 uses a particle filter, the data update unit 15 only needs to update the queue of the time series data and perform sampling based on the matching score calculated by the bipartite graph matching. In this way, the data update unit 15 can update the time series data of the above-mentioned posture.

[0093] [Detailed Structure of Posture Predictor 11, etc.]

[0094] Next, the detailed structure of the posture predictor 11 will be described.

[0095] Figure 4 Yes means Figure 3 FIG. 2 is a block diagram showing an example of a detailed configuration of the posture predictor 11 shown in FIG. Figure 5 Yes means Figure 3 FIG. 1 is a diagram showing an example of the architecture of the posture predictor 11 shown in FIG. Figure 5 , it is also shown that the first neural network 112 performs error back propagation and calculation of the error function of the log-likelihood only during learning.

[0096] As described above, the posture predictor 11 is configured by a structure including the first neural network 112 , and predicts on which side the positions of key points constituting the posture of the person in the current frame are located using the posture distribution as a posture prediction taking uncertainty into consideration.

[0097] In this embodiment, the posture predictor 11 can predict multiple postures (also called multiple hypotheses) with probabilistic behaviors using posture distribution by considering two types of uncertainty: epistemic uncertainty and heteroscedastic aleatoric uncertainty.

[0098] Epistemic uncertainty is the uncertainty caused by insufficient knowledge or information about the model and the parameters used in the model, also known as model uncertainty. In the case of using DNN for reasoning, epistemic uncertainty means the uncertainty caused by differences in learning models due to insufficient learning data. In addition, epistemic uncertainty is improved when there is sufficient learning data.

[0099] Chance uncertainty is uncertainty caused by differences or changes that exist in observations, that is, uncertainty caused by noise inherent in observations. In other words, chance uncertainty is accidental uncertainty caused by observation errors or interference. In the case of posture prediction, this uncertainty is caused by sudden changes in a person's pace, panning and pitching of a high-speed camera, or posture estimation errors based on the deepCNN model. In addition, no matter how much learning is done, chance uncertainty will not improve. Heteroscedastic chance uncertainty is uncertainty when the amount of noise inherently contained in observations varies depending on the input.

[0100] like Figure 4 As shown, in this embodiment, the posture predictor 11 includes an input processing unit 111 , a first neural network 112 , and an output processing unit 115 .

[0101] <Input Processing Unit 111>

[0102] The input processing unit 111 extracts and outputs, from the time series data of postures input to the posture predictor 11, the coordinates of the postures of one or more characters appearing in past frames within a specified time such as 10 frames, and a visible flag indicating whether the coordinates were visible in the past frames.

[0103] exist Figure 5 In the example shown, the input processing unit 111 extracts the position coordinates of the set of key points constituting the posture in the past frames of the number of frames included in the parameter L indicating the time length from the time series data of the posture input to the posture predictor 11, and calculates the difference in the time direction. The input processing unit 111 outputs the calculated difference to the first neural network 112.

[0104] <First neural network 112>

[0105] The first neural network 112 uses one or more models for considering uncertainty, and makes one or more models predict the predicted posture positions of one or more first objects in the current frame based on the time series data of the posture, thereby predicting the predicted distribution. The first neural network 112 has a recurrent neural network and a predetermined neural network that forms one or more models. Here, the one or more models are formed by applying dropout to the predetermined neural network to deactivate one or more nodes constituting the predetermined neural network.

[0106] In this embodiment, if Figure 4 As shown, the first neural network 112 includes a recurrent NN (Neural Network) 113 and a fully coupled NN (Neural Network) 114 .

[0107] The recurrent NN 113 is a neural network that has a recursive coupling of the output of a certain layer and inputs back and forth, and processes time series data whose values ​​change over time. The recurrent NN 113 may be, for example, an LSTM (Long Short Term Memory). In addition, the recurrent NN 113 is not limited to LSTM, and may be a GRU (Gated Recurrent Unit) or a QRNN (Quasi-Recurrent Neural Networks).

[0108] Here, LSTM is a model that can learn long-term time series data and can use the short-term memory inside the network for a long time. GRU is a model that makes LSTM a little simpler, but like LSTM, it can learn long-term time series data. QRNN is a model that simulates the structure of RNN with CNN. QRNN is also understood as an algorithm that speeds up LSTM.

[0109] exist Figure 5 In the example shown, the recurrent NN 113 is composed of an LSTM with 64 units.

[0110] The fully coupled NN114 is a neural network composed of fully coupled layers, also known as a fully coupled neural network (feedforward). The fully coupled NN114 consists of an input layer, multiple intermediate layers, and an output layer, and all intermediate layers become fully coupled layers. The fully coupled NN114 applies discarding or ensemble, and outputs multiple hypotheses (various postures) that take into account cognitive uncertainty in a distributed form.

[0111] Here, discarding is a method of promoting learning, etc. while deactivating (also called invalidating) certain nodes of a neural network. In the present embodiment, discarding is applied to the fully coupled NN114 during prediction to model cognitive uncertainty. Thus, the fully coupled NN114 when a part of the intermediate layer has been deactivated can be treated as one model. That is, the fully coupled NN114 uses discarding to randomly deactivate a part of the intermediate layer, so that one posture prediction can be output according to each model while performing probability sampling (also called probability model) on more than one different model. In addition, the fully coupled NN114 approximates the result of the posture prediction as a Gaussian distribution and outputs it.

[0112] In this way, the fully coupled NN 114 can approximate the multiple postures obtained from the probability model to Gaussian distribution and output them as prediction results. In addition, Gaussian distribution is an example, and it can be approximated by categorical distribution, triangular distribution or Cauchy distribution.

[0113] exist Figure 5In the example shown, the fully coupled NN 114 is composed of, for example, 40 intermediate layers, i.e., hidden nodes, and Leaky-ReLU is used as the activation function of the hidden nodes. In addition, the probability of discarding the hidden nodes is 0.3. In this way, the fully coupled NN 114 can model cognitive uncertainty by using discarding during prediction. In addition, the number of hidden nodes is not limited to 40, and the activation function of the hidden nodes is not limited to Leaky-ReLU. The probability of discarding is also not limited to 0.3, and can be set appropriately.

[0114] In addition, Figure 5 , an example of using discarding in prediction by the fully coupled NN 114 is shown, but it is not limited to this, and integration can also be used. In this case, the fully coupled NN 114 has more than one model, and the result of fusing the posture predictions obtained from more than one model can be output, for example, in a manner similar to a Gaussian distribution.

[0115] As described above, since the fully coupled NN114 probabilistically samples various models, it is able to output multiple hypotheses (predicted multiple postures) based on one result (input) output from LSTM113, and output the output in the form of a distribution (i.e., predicted distribution) to the output processing unit 115.

[0116] <Output processing unit 115>

[0117] The output processing unit 115 further generates a plurality of hypotheses (predicted plurality of postures) based on the distribution (the above-mentioned predicted distribution) output from the first neural network 112. Then, the distribution including the distribution output from the first neural network 112 and the generated plurality of hypotheses is output as the predicted distribution output by the posture predictor 11 to the matching unit 13 and the data updating unit 15.

[0118] In the present embodiment, by causing the output processing unit 115 to further add variance to the output of the first neural network 112 (the above-mentioned prediction distribution) and output the result, heteroscedastic random uncertainty can be modeled.

[0119] More specifically, the output processing unit 115 outputs the distribution including the distribution output from the first neural network 112 and the generated multiple hypotheses as the predicted distribution output by the posture predictor 11 in the form of adding the variance to the output of the first neural network 112. Thus, the output processing unit 115 can output the predicted distribution output by the posture predictor 11 in the form of processing as particles of the sample set constituting the particle filter.

[0120] exist Figure 5In the example shown, the output processing unit 115 calculates a plurality of values ​​corresponding to a plurality of hypotheses by averaging the Gaussian distribution output from the first neural network 112 and multiplying the values ​​by a random number. In addition, the output processing unit 115 calculates a variance based on the Gaussian distribution output from the first neural network 112 and the sum of the coordinates of the most recent eye-visible key point extracted from the time series data of the posture. Then, the output processing unit 115 outputs the Gaussian distribution obtained by adding the calculated plurality of values ​​to the variance value as a predicted distribution. In addition, in Figure 5 In it, σ represents the mean of Gaussian distribution, μ represents the variance, and N(0, 1) represents the standard Gaussian distribution.

[0121] In this way, the output processing unit 115 adaptively changes the diversity of particles, that is, the variance value, according to the situation, and outputs a Gaussian distribution that depends on the output of the first neural network 112. In addition, the Gaussian distribution is an example, and it can be approximated by a categorical distribution, a triangular distribution, or a Cauchy distribution.

[0122] In addition, the output processing unit 115 outputs a Gaussian distribution that disperses particles over a wider range, for example, when a person shown in the image moves at a high speed. On the other hand, the output processing unit 115 outputs a Gaussian distribution that concentrates particles in a narrower area, for example, when a person shown in the image moves at a low speed. Such an action helps to effectively utilize the limited particles used in the sample set of the particle filter.

[0123] (Example 1)

[0124] In Example 1, the use Figure 5 The following describes a case where the posture predictor 11 predicts the prediction distribution of the current frame to obtain a posture prediction result, and a case where the posture tracking technology disclosed in non-patent document 1 is used as a comparative example to estimate the posture of the current frame and obtain a posture estimation result. Here, the posture tracking technology disclosed in non-patent document 1 is also referred to as optical flow posture estimation.

[0125] Figure 6A to Figure 6C : is a diagram showing the posture estimation result of the current frame in the comparative example. Figure 7A to Figure 7C is a diagram showing the posture prediction result of the current frame in Example 1. Fig. 6A and Fig. 7A , Figure 6B and Figure 7B ,as well as Figure 6C and Figure 7C In , the pose estimation result and the pose prediction result for the same current frame are shown respectively. Figure 6A to Figure 6CIn , the hollow circles represent the positions of the key points estimated in the previous frame, the hollow triangles represent the positions of the key points estimated in the current frame, and the hollow stars represent the positions of the real (i.e. correct) key points in the current frame. Figure 7A to Figure 7C In , the dotted area represents the distribution of the positions of the predicted key points in the current frame (equivalent to the predicted distribution mentioned above), represents the average position of the predicted distribution, and the hollow stars represent the real (i.e. correct) positions of the key points in the current frame.

[0126] like Fig. 6A As shown in , for people with little movement, the positions of the real key points are consistent with the estimated key points, and a highly reliable pose estimation result is obtained. Figure 6B and Figure 6C As shown, for a body part or a person that moves quickly, the positions of the true key points are separated from the positions of the estimated key points, resulting in a pose estimation result with low reliability.

[0127] In contrast, when using Figure 5 The pose predictor 11 shown in FIG. 1 considers two uncertainties to predict the prediction distribution of the current frame, as Figure 7A to Figure 7C As shown in FIG. 1 , the true position of the key point is included in the predicted distribution shown in the dotted area. Furthermore, since the true position of the key point is relatively consistent with the average position of the predicted distribution, it can be seen that a highly reliable posture prediction result is obtained.

[0128] [Operation of Object Tracking Device 10, etc.]

[0129] Next, the operation and the like of the object tracking device 10 configured as above will be described.

[0130] Figure 8 : is a flowchart showing an operation example of the object tracking device 10 in this embodiment.

[0131] First, the object tracking device 10 inputs the time series data of the posture to the posture predictor 11 to predict the predicted distribution in the current frame (S1). More specifically, the object tracking device 10 inputs the time series data of the posture to the posture predictor 11 including the first neural network and performing posture prediction taking into account uncertainty to predict the distribution of the predicted posture positions of one or more first objects in the current frame, i.e., the predicted distribution. In addition, the time series data of the posture includes the coordinates of the postures of one or more objects respectively shown in two or more past frames continuous in time series in the image input to the posture estimation unit 12. In addition, in the present embodiment, the object is a person.

[0132] Next, the object tracking device 10 inputs the image to the posture estimation unit 12, and estimates the set of positions of each object shown in the current frame (S2). More specifically, the object tracking device 10 inputs the RGB image of the current frame to the second neural network constituting the posture estimation unit 12, and estimates the set of positions of one or more second objects shown in the current frame.

[0133] Next, the object tracking device 10 obtains and outputs the identification information and position information of the object in the current frame by performing matching (S3). More specifically, the object tracking device 10 matches the set of positions estimated in step S2 with the predicted distribution predicted in step S1. Thus, the object tracking device 10 obtains and outputs identification information indicating which of the one or more first objects one or more second objects respectively corresponds to, and coordinates indicating the posture of each of the one or more second objects, i.e., position information. In addition, in the present embodiment, the object tracking device 10 uses bipartite graph matching as matching. In addition, the identification information of the object in the current frame is information related to the ID assigned to the object in the current frame. The position information of the object in the current frame is the position coordinates of multiple key points constituting the posture of the object in the current frame.

[0134] Next, the object tracking device 10 updates the posture time series data by adding data including coordinates indicating the posture of the object in the current frame obtained from the predicted distribution predicted in step S1 based on the identification information and position information acquired in step S3 ( S4 ).

[0135] In this way, the object tracking device 10 can track the posture of an object such as a person shown in the image by performing steps S1 to S4 on the input image.

[0136] (Example 2)

[0137] In the second embodiment, the object tracking method of the pose predictor 11 and the object tracking device 10 using a particle filter for predicting the prediction distribution of the current frame in consideration of two uncertainties will be described. In the second embodiment, the following description assumes that the object is a person.

[0138] Fig. 9 This is a diagram showing an example of pseudo code of the object tracking method in Example 2. Fig.10 is a diagram conceptually showing the object tracking method in Example 2. In addition, Fig. 9The pseudo code shown is implemented using TensorFlow, a tool for performing matrix operations, and the repetition of loops other than the outermost loop and all independent threads are executed in parallel by the GPU. As a GPU, for example, by using a single NVIDIA RTX2080 GPU, it is possible to track about 10 poses at the same time at 30fps. Fig. 9 In the pseudo code shown, the algorithm represented by "(1) Pose Prediction" is equivalent to the pose prediction processing of the pose predictor 11 of the object tracking device 10. The algorithm represented by "(2) Pose Estimation" is equivalent to the pose estimation processing of the pose estimation unit 12 of the object tracking device 10. The algorithm represented by "(3) Greedy Matching" is equivalent to the matching processing of the matching unit 13 of the object tracking device 10. The algorithm represented by "(4) Particle Update" is equivalent to the update processing of the data update unit 15 of the object tracking device 10.

[0139] Here, let C be one or more characters composed of a plurality of postures estimated from the current frame t at time t. t , and set the temporary index representing each of the multiple characters to j. Fig. 9 The pseudo code shown in the figure performs the object tracking method using the maximum F max Filters to track multiple poses in multiple people.

[0140] In addition, each of the plurality of postures has a track ID k that uniquely represents a person, and is represented by P particles. The particle n of the filter k includes a particle storing the past posture z (k，n) t-L：t-1 During the tracking process for a given moment, only F t (<F max ) The filter is activated and executed. The activation and deactivation of the filter is controlled by a life count l that manages the appearance (disappearance) of the character in the frame. k control.

[0141] First, in Fig. 9 In the pseudo code shown, when Algorithm 1 starts, all filters become inactive as shown in the order of line 1.

[0142] Next, the posture prediction process is performed in the order of lines 3 and 4. More specifically, first, in the step of line 3, the past posture z is collected from the activation filter. (k；n) t-L：t-1 F tNext, in the step of row 4, the collected posture time series data is input to the posture predictor 11, and the output F t ×P predicted pose z (k； n) t .

[0143] In addition, when time t=1, that is, for the first current frame t constituting the image or when the filter is not activated, the steps of the third and fourth rows are not executed. t represents the set of currently valid activated filters. In addition, Fig.10 In , the time series data of the posture is represented as coordinates representing the predicted distribution of multiple past frames and an ID assigned to the character.

[0144] Next, the pose estimation process is performed according to the sequence of line 5. More specifically, in the step of line 5, the RGB image of the current frame is input to the deep CNN disclosed in the above-mentioned non-patent document 1 to estimate the pose of the person C. t The estimated pose x (j) t In addition, Fig.10 , the RGB image of the current frame is input to the posture estimation unit 12, and the estimation results of multiple postures of the person shown in the current frame are obtained using key points, and even the person's identity (ID to be assigned) is unknown.

[0145] Next, the matching process is performed according to the steps of lines 6 to 17. More specifically, first, in the step of line 6, the greedy method is used to match the F t ×P predicted pose z (k；n) t and C t The estimated pose x (j) t , as a whole, C t ×F t ×P OKS value d (j，k，n) OKS Then, in step 7, C t ×F t The shape of the OKS tensor (Japanese: テンソル) of ×P is transformed into C by taking the weighted average along the n-axis t ×F t , calculate the matrix of matching scores. Then, in the step of line 8, the calculated matrix of matching scores is used to perform bipartite graph matching processing to combine reasonable pairs. Through this processing, pairs with matching scores below the threshold (i.e., the estimated posture and the predicted posture are configured separately) are deleted, thereby preventing inappropriate matching. The variable j' shown in the step of line 8k and k' j The variables j' represent the indices of the objects corresponding to the filter k and the posture j, respectively. In addition, when the corresponding object is not assigned due to insufficient or excessive activation of the filter or the threshold, the variable j' k and k' j Takes a negative value. In addition, {j' k} k represents the ID of the matching person observed from a certain filter k, {k' j} j represents the matched filter observed from the detected person j.

[0146] Next, in the steps of lines 10 and 11, the variable k' j If the posture j has a valid value, (x (j) t , k' j ) is output to the output unit 14 as a tracking result. Fig.10 In FIG. 10 , the tracking result of the group overlapped with the current frame t outputted by the output unit 14 (not shown) is shown. j >-1 means there is a variable k' j The filter that has a valid value and a pairing is established for posture j. On the other hand, in the steps of lines 12 to 17, the variable k' j If the new filter k has an invalid value for posture j, new is activated. Then, the new index is equal to the estimated pose x (j) t The signals are output to the output unit 14 together.

[0147] Next, in the steps of lines 15 and 16, since the posture of the past frame is unknown, the queue of time series data representing the posture is set to 0, the key points are not displayed and initialized, and the predicted posture is set to x (j) t , thereby initializing.

[0148] Next, the update process is performed in the steps of lines 18 to 26. More specifically, first, in the step of line 18, the latest predicted posture z is (k，n) t Push to the queue of time series data representing postures, and delete the earliest predicted posture. Then, in the steps of lines 20 and 21, in the variable j' k With a valid value, the queue of filter k performs probabilistic resampling of reselected particles based on the likelihood in the previous state. In addition, the steps in lines 20 and 21 are in j'k If <0, it will not be executed.

[0149] Next, in steps 22 and 24, the lifetime count l k Increase or decrease according to the existence of the object corresponding to the filter k. In the steps of lines 25 and 26, when the lifetime count l k When it becomes zero, it is considered that the person tracked by filter k has completely disappeared, and filter k is deactivated.

[0150] (Example 3)

[0151] In Example 3, the results of the performance evaluation experiment of the object tracking device 10 using the Pose-Track2018 Data set are described. In addition, in the Pose-Track2018 Data set, multiple characters in the video are annotated with locations including 17 individual key points and unique track IDs. In Example 3, the following description assumes that the object is a person.

[0152] In this embodiment, the Pose-Track2018 Data set is used to Figure 5 The pose predictor 11 shown in FIG. 1 is learned and evaluated. The learning data is created from the annotation data for learning of the Pose-Track2018 Data set. In addition, the pose predictor 11 is learned using the created learning data using the Adam optimizer. Here, the learning rate is 10 -3 , the mini-batch size is 30.

[0153] Fig.11 is a diagram showing the evaluation results of the object tracking performance in Example 3. Fig.11 In this paper, the object tracking accuracy (MOTA) is calculated as the object tracking performance by using the Pose-Track2018 Data set and known evaluation tools. Here, MOTA is the abbreviation of Multiple Object Tracking Accuracy. Fig.11 In the present disclosure, the MOTA of the posture tracking method of the posture predictor 11 is shown. Fig.11 In FIG. 1 , the MOTA of the posture tracking method disclosed in Non-Patent Document 1 is shown as a comparative example, and the MOTA of other existing posture tracking methods are shown as Reference Examples 1 to 5.

[0154] like Fig.11 As shown, it can be seen that in the present disclosure, a score of 66.2 was achieved, which is an improvement over the score of 65.4 in the comparative example.

[0155] Next, we clarified Figure 5 An experiment was conducted to determine which component of the posture predictor 11 shown contributes to the improvement. Figure 5 The performance evaluation experiment of the posture predictor 11 shown in the figure is performed in the case where 1) both or one of the two uncertainties are invalidated, and 2) the parameter L representing the time length of the time series data is changed. Figure 5 The discarding and deactivation of the fully coupled NN 114 shown in FIG. 1 achieves a situation where epistemic uncertainty is not modeled. Figure 5 The average value (ie, the value of σ) used in the output processing unit 115 shown is fixed, thereby achieving a situation where the heteroscedastic random uncertainty is not modeled.

[0156] Fig.12 1 is a diagram showing the results of a performance evaluation experiment in which the components of the posture predictor 11 in Example 3 are disabled. Fig.12 In the table shown, the test is performed when epistemic uncertainty is enabled, that is, when epistemic uncertainty is modeled. Similarly, the test is performed when heteroscedasticity aleatory uncertainty is enabled, that is, when heteroscedasticity aleatory uncertainty is modeled. Fig.12 In the above equation, epistemic uncertainty is expressed as E.Uncertainty and heteroskedastic aleatoric uncertainty is expressed as HAUncertainty.

[0157] In addition, Fig.12 In , as an indicator of performance evaluation, num_Switches, which is an intermediate variable for calculating MOTA, is used instead of MOTA. In addition, MOTA is composed of three variables: num_Switches, num_misses, and num_false_positives. num_Switches is an indicator that represents the total number of tracking errors, such as a key point that is recognized as a person such as A in a certain frame, but is mistakenly recognized as a key point of a person such as B in other frames. num_misses is an indicator that represents the total number of pose estimation errors in which the position of a key point that should be recognized is not recognized in a certain frame. num_false_positives is an indicator that represents the total number of pose estimation errors in which a key point that should not be recognized is mistakenly recognized in a certain frame. In addition, in Fig.12 In FIG. 1 , num_Switches of the posture tracking method disclosed in non-patent document 1 is shown as a comparative example, and the MOTA score is also shown for reference.

[0158] exist Fig.12In [the present disclosure], if the values of num_switches of the present disclosure and the comparative example are compared, it can be seen that Fig.11 the MOTA score of the pose predictor 11 shown in Fig.11 is achieved not by more accurate pose estimation but by pose prediction based on the pose predictor 11. Furthermore, as Fig.12 shown in Fig.12 , it can be seen that in the present disclosure and the comparative example, in the value of num_Switches, an improvement of approximately 50% is achieved.

[0159] In addition, from Fig.12 it can also be seen that the tracking performance is improved by the participation of two types of uncertainties. Additionally, in the pose predictor 11 that takes into account two types of uncertainties, by referring to a longer context where the value of the parameter L representing the length of the time series data of the pose is 10 or 15, it can be seen that the MOTA score becomes higher. That is, by referring to a long context of the time series data of the pose, in the pose predictor 11 that takes into account two types of uncertainties, more and more reasonable hypotheses (predicted pose positions) can be predicted, and as a result, it can be seen that the final performance can be improved. Furthermore, the parameter L affects the computational complexity and learning stability such as the memory size and calculation time of the sequential transmission of the LSTM, so it needs to be carefully determined. In addition, in the previous description, by using a queue, the time series information of the pose is clearly retained as a history, but the method of processing the time series information is not limited to this. For example, the internal variables of the LSTM can also be retained and used as an alternative to the queue (Japanese: キューの代替).

[0160] [Effects of the object tracking device 10, etc.]

[0161] As described above, according to the object tracking device 10 of the present disclosure, by using the pose predictor 11 that performs pose prediction considering the uncertainty of the prediction result to perform pose tracking of the person shown in the image, tracking errors can be suppressed. In the present embodiment, the pose predictor 11 that assembles two types of uncertainties to perform pose prediction and a particle filter are used to perform pose tracking of the person shown in the image. Here, in the pose predictor 11, two types of uncertainties, namely, epistemic uncertainty and heteroscedastic aleatoric uncertainty, are modeled and assembled. As a result, the pose predictor 11 can predict diverse and reasonable hypotheses (i.e., predicted pose positions) through the prediction distribution of the pose of the person shown in the current frame (i.e., the distribution of the positions of the key points). In addition, by introducing a recurrent neural network into the pose predictor 11, long-term context information can be utilized, so even if the person shown in the image is occluded, tracking errors can be suppressed.

[0162] Here, Fig.13 and Fig.14 are used to explain the effects of the object tracking device 10.

[0163] Fig.13 and Fig.14 is a diagram visually showing the object tracking results based on the object tracking method in the present disclosure and the comparative example. Fig.13 , the posture tracking result is shown when three people shown in the image are moving at high speed, that is, the postures of multiple people are changing at high speed. Fig.14 , shows the result of posture tracking when one person is obscured by others in the second frame and reproduced in the third frame among the three time-series frames constituting the image, that is, the posture of the person shown in the image is obscured. Fig.13 (a) and Fig.14 In (a), as a comparative example, the posture tracking result based on the posture tracking technology disclosed in Non-Patent Document 1 is shown. Fig.13 (b) and Fig.14 In (b), the posture tracking result of the object tracking device 10 in this embodiment is shown as the present disclosure.

[0164] More specifically, in Fig.13 In (a), it can be seen that in the later frame of the time series, the key point represented by the thick circle line is attached to a different person from the previous frame, and the different people are mistakenly recognized as the same person, resulting in a tracking error. Fig.13 In (b), in the previous and next frames in the time series frames, the key points indicated by the thick circle lines are added to the same person, so it can be seen that correct recognition is possible and tracking errors are suppressed.

[0165] exist Fig.14 As can be seen from (a), in the first and last frames of the time series, key points represented by circular lines and dotted circles are added to the occluded person, and the same person is mistakenly identified as a different person, resulting in tracking errors. Fig.14 In (b), it can be seen that even in the first frame and the last frame in the time series frame, key points represented by circular lines are added to the occluded person, and the person can be correctly identified as the same person, thereby suppressing tracking errors.

[0166] As described above, it can be seen that according to the object tracking device 10 in the present embodiment, even if disturbances such as a posture changing at a high speed or a posture disappearing or reappearing due to occlusion occur, tracking errors can be suppressed.

[0167] (Other possible implementations)

[0168] The object tracking method of the present disclosure is described above in the embodiments, but there is no particular limitation on the subject and device that implement each process. It can be processed by a processor (described below) assembled in a specific device configured locally. In addition, it can also be processed by a cloud server configured in a place different from the local device.

[0169] In addition, the present disclosure is not limited to the above-mentioned embodiments. For example, the constituent elements described in this specification may be arbitrarily combined, and other embodiments implemented by excluding several constituent elements may also be embodiments of the present disclosure. In addition, the above-mentioned embodiments are implemented without departing from the main purpose of the present disclosure, that is, the scope of the meaning indicated by the sentences recorded in the claims, and the modified examples obtained by implementing various modifications conceived by those skilled in the art are also included in the present disclosure.

[0170] For example, in order to model the aleatoric uncertainty, α-divergence dropout and neural network ensemble can also be used. In addition, in order to model the aleatoric uncertainty, the prediction distribution can be represented by a mixed Gaussian distribution instead of the above Gaussian distribution by introducing a mixture density network.

[0171] In addition, the object tracking method disclosed in the present invention utilizes a particle filter, posture prediction that takes into account two types of uncertainty, and context information. Therefore, the object tracking method disclosed in the present invention has the possibility of being applied to various SMC (Sequential Monte Carlo)-based robotics tasks such as SLAM (Simultaneous Localization and Mapping). In addition, the object tracking method disclosed in the present invention may not be limited to two-dimensional human postures, but may be applied to three-dimensional human postures.

[0172] The present disclosure also includes the following cases.

[0173] (1) Specifically, the above-mentioned device is a computer system composed of a microprocessor, ROM, RAM, hard disk unit, display unit, keyboard, mouse, etc. A computer program is stored in the RAM or hard disk unit. The microprocessor operates according to the computer program, and each device realizes its function. Here, the computer program is composed of a combination of multiple command codes representing instructions to the computer in order to realize a specified function.

[0174] (2) Some or all of the components constituting the above-mentioned device may also be constituted by a system LSI (Large Scale Integration). System LSI is a super-multifunctional LSI manufactured by integrating multiple structural parts on a single chip. Specifically, it is a computer system composed of a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The microprocessor operates according to the computer program, and the system LSI realizes its functions.

[0175] (3) A part or all of the components constituting the above-mentioned device may also be constituted by an IC card or a single module that can be loaded and unloaded on each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may also include the above-mentioned super-multifunctional LSI. The IC card or the module realizes its function by the microprocessor operating according to the computer program. The IC card or the module may also be tamper-resistant.

[0176] (4) In addition, the present disclosure may be the methods described above. In addition, the present disclosure may be a computer program that implements the methods on a computer, or a digital signal composed of the computer program.

[0177] (5) In addition, the present disclosure may also record the computer program or the digital signal in a computer-readable recording medium, such as a floppy disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. In addition, the digital signal may be recorded in these recording media.

[0178] In addition, the present disclosure may also transmit the computer program or the digital signal via a telecommunication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, or the like.

[0179] Furthermore, the present disclosure may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program and the microprocessor operates according to the computer program.

[0180] Furthermore, the program or the digital signal may be recorded in the recording medium and transferred, or may be transferred via the network or the like, so that the program or the digital signal may be implemented by another independent computer system.

[0181] Industrial Applicability

[0182] The present disclosure can be utilized in a non-temporary computer-readable recording medium recording an object tracking method and a program, and in particular can be utilized in a non-temporary computer-readable recording medium recording an object tracking method and a program for tracking two-dimensional postures of multiple persons shown in an image.

Claims

1. An object tracking method is a method for tracking an object in an image performed by a computer, wherein: include: A prediction step, inputting the time series data of the posture into a posture predictor, so that the posture predictor predicts the distribution of the predicted posture positions of one or more first objects in the current frame, that is, the predicted distribution, wherein the time series data of the posture includes coordinates representing the postures of the one or more objects respectively shown in two or more past frames continuous in the time series, the posture predictor is composed of a structure including a first neural network and performs posture prediction, wherein the posture prediction takes uncertainty into account; An estimation step of inputting the current frame into a second neural network so that the second neural network estimates a set of positions of one or more second objects shown in the current frame; an output step of obtaining and outputting identification information and position information by matching a set of positions of each of the one or more second objects with the predicted distribution, wherein the identification information indicates which of the one or more first objects each of the one or more second objects corresponds to, and the position information is a position indicating a posture of each of the one or more second objects; and An updating step, based on the identification information and the position information obtained in the output step, appends data including coordinates representing the postures of each of the one or more first objects obtained from the predicted distribution predicted in the prediction step to the time series data of the postures, thereby updating the time series data of the postures.

2. The object tracking method according to claim 1, wherein: In the output step, by matching the set of positions of the one or more second objects with the predicted distribution, a group consisting of the predicted distribution and the set of positions having a matching score greater than a specified value is calculated, and the identification information and the position information are obtained and output based on the calculated group.

3. The object tracking method according to claim 2, wherein: In the output step, the set of positions of each of the one or more second objects is subjected to bipartite graph matching with the predicted distribution, thereby calculating the group consisting of the predicted distribution and the set of positions having a matching score greater than a predetermined value.

4. The object tracking method according to any one of claims 1 to 3, wherein: In the prediction step, the first neural network uses one or more models for considering the uncertainty, and predicts the predicted distribution by allowing the one or more models to predict the predicted posture positions of one or more of the first objects in the current frame based on the time series data of the posture.

5. The object tracking method according to claim 4, wherein: The predicted distribution is a distribution that takes the uncertainty into consideration, includes the predicted predicted posture position, and is expressed as a variance in a Gaussian distribution.

6. The object tracking method according to claim 4, wherein: The first neural network includes a recurrent neural network and a predetermined neural network forming the one or more models.

7. The object tracking method according to claim 6, wherein: The one or more models are formed by applying dropout to the predetermined neural network to deactivate one or more nodes constituting the predetermined neural network.

8. A recording medium, which is a non-transitory computer-readable recording medium having a program recorded thereon, wherein the program causes a computer to execute the following steps: A prediction step, inputting the time series data of the posture into a posture predictor, so that the posture predictor predicts the distribution of the predicted posture positions of one or more first objects in the current frame, that is, the predicted distribution, wherein the time series data of the posture includes coordinates representing the postures of the one or more objects respectively shown in two or more past frames continuous in the time series, the posture predictor is composed of a structure including a first neural network and performs posture prediction, wherein the posture prediction takes uncertainty into account; An estimation step of inputting the current frame into a second neural network so that the second neural network estimates a set of positions of one or more second objects shown in the current frame; an output step of obtaining and outputting identification information and position information by matching a set of positions of each of the one or more second objects with the predicted distribution, wherein the identification information indicates which of the one or more first objects each of the one or more second objects corresponds to, and the position information is a position indicating a posture of each of the one or more second objects; and An updating step, based on the identification information and the position information obtained in the output step, appends data including coordinates representing the postures of each of the one or more first objects obtained from the predicted distribution predicted in the prediction step to the time series data of the postures, thereby updating the time series data of the postures.

Images