Information processing device, information processing method, and program

Abstract

The system correctly tracks identical objects, taking into account changes in appearance due to movement or changes in orientation. [Solution] In the information processing device, the trajectory fragment generation means generates trajectory fragments that show at least a portion of the trajectory of an object moving in a time-series frame image, and include object information such as the time, coordinates, and appearance features of the object in each frame image. The correlation calculation means calculates the correlation of appearance features for object pairs created by taking one object each from a first trajectory fragment and a second trajectory fragment, which are included in a trajectory fragment pair, which is a pair of trajectory fragments. The appearance similarity calculation means calculates the appearance similarity of the trajectory fragment pair based on the correlation and the similarity between the appearance features of the object pair. The fragment pair similarity calculation means calculates the fragment pair similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, using the appearance similarity. The object trajectory calculation means calculates the object trajectory for the same object by combining a plurality of trajectory fragment pairs based on the fragment pair similarity.

Description

[Technical Field]

[0001] This disclosure relates to the tracking of objects in video footage. [Background technology]

[0002] Methods have been proposed for tracking objects in images captured by cameras and other devices. For example, Patent Document 1 describes a method for tracking objects by linking the trajectories of objects detected in the image. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-194857 [Overview of the project] [Problems that the invention aims to solve]

[0004] The method described in Patent Document 1 determines whether or not to link pairs of trajectories based on the similarity of the objects included in the trajectories. However, even the same object can change in appearance due to movement, changes in orientation, etc. Therefore, simply comparing the similarity of appearances is not sufficient to correctly determine the identity of objects.

[0005] One objective of this disclosure is to provide an information processing device that can correctly track the same object while taking into account changes in appearance due to the movement or change in orientation of the object. [Means for solving the problem]

[0006] From one perspective of this disclosure, the information processing device is A trajectory fragment generation means generates a trajectory fragment that shows at least a portion of the trajectory of an object moving in a time-series frame image, and includes object information showing the time, coordinates, and appearance features of the object in each frame image, For an object pair created by extracting one object from each of a first trajectory fragment and a second trajectory fragment included in a pair of trajectory fragments that are a pair of the trajectory fragments, correlation calculation means for calculating the correlation of the appearance feature amounts; Appearance similarity calculation means for calculating the appearance similarity of the pair of trajectory fragments based on the correlation and the similarity between the appearance feature amounts of the object pair; Fragment pair similarity calculation means for calculating a fragment pair similarity that is the similarity between the first trajectory fragment and the second trajectory fragment using the appearance similarity; Object trajectory calculation means for calculating an object trajectory for the same object by combining a plurality of pairs of trajectory fragments based on the fragment pair similarity; comprising.

[0007] In another aspect of the present disclosure, an information processing method executed by a computer is generating a trajectory fragment including object information indicating at least a part of a trajectory along which an object included in a time-series frame image moves, the time, coordinates, and appearance feature amount of the object in each frame image of the object; For an object pair created by extracting one object from each of a first trajectory fragment and a second trajectory fragment included in a pair of trajectory fragments that are a pair of the trajectory fragments, calculating the correlation of the appearance feature amounts; calculating the appearance similarity of the pair of trajectory fragments based on the correlation and the similarity between the appearance feature amounts of the object pair; calculating a fragment pair similarity that is the similarity between the first trajectory fragment and the second trajectory fragment using the appearance similarity; calculating an object trajectory for the same object by combining a plurality of pairs of trajectory fragments based on the fragment pair similarity.

[0008] In still another aspect of the present disclosure, a program is generating a trajectory fragment including object information indicating at least a part of a trajectory along which an object included in a time-series frame image moves, the time, coordinates, and appearance feature amount of the object in each frame image of the object; For a pair of objects created by taking one object each from the first and second trajectory fragments included in the pair of trajectory fragments, the correlation of the aforementioned appearance features is calculated. Based on the correlation and the similarity between the appearance features of the object pair, the appearance similarity of the trajectory fragment pair is calculated. Using the aforementioned visual similarity, the fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, is calculated. Based on the aforementioned fragment pair similarity, the computer is instructed to combine multiple trajectory fragment pairs and perform a process to calculate the object trajectory for the same object. [Effects of the Invention]

[0009] According to this disclosure, it becomes possible to correctly track the same object while taking into account changes in appearance due to the movement or change in orientation of the object. [Brief explanation of the drawing]

[0010] [Figure 1] The overall configuration of an information processing device relating to an example of this disclosure is shown. [Figure 2] This is a block diagram showing the hardware configuration of an information processing device. [Figure 3] This is a block diagram showing the functional configuration of an information processing device. [Figure 4] This is a flowchart of the object trajectory calculation process. [Figure 5] An example of a behavioral management system using the information processing device disclosed herein is shown. [Figure 6] This is a block diagram showing the functional configuration of other information processing devices. [Figure 7] This is a flowchart of the processing performed by other information processing devices. [Modes for carrying out the invention]

[0011] Preferred embodiments of the present disclosure will be described below with reference to the drawings. In the following description, variables with a symbol above them are indicated by superscripting that symbol for convenience of notation. For example, a variable "X" with the symbol "~" above it is indicated as "X ~ It is written as "".

[0012] <First Embodiment> [Overall structure] Figure 1 shows the overall configuration of an information processing device according to an example of the present disclosure. The information processing device 100 tracks an object contained in a video and outputs an object trajectory showing the trajectory of that object.

[0013] The information processing device 100 receives video footage captured by a camera or the like. The input video is a time-series frame image, which is a sequence of multiple frame images arranged in chronological order. The information processing device 100 may receive the captured video footage directly from the camera, or it may receive video footage stored in a database or the like.

[0014] In short, the information processing device 100 first generates trajectory fragments for each object from the frame images. A trajectory fragment is data that arranges object information of the same object contained in several frame images in chronological order, and is also called a tracklet. Then, the information processing device 100 connects multiple trajectory fragments corresponding to the same object from the obtained multiple trajectory fragments to generate and output an object trajectory that shows the overall trajectory of the object. By correctly connecting individual trajectory fragments that can be detected with relatively high accuracy, the information processing device 100 can detect the complete trajectory of an object with high accuracy.

[0015] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the information processing device 100. As shown in the figure, the information processing device 100 comprises a processor 11, an interface (IF) 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a database (DB) 15, and a recording medium 16. Each component is connected to the others, for example, via a bus 18.

[0016] The processor 11 is a computer such as a CPU (Central Processing Unit) and controls the entire information processing device 100 by executing a pre-prepared program. Specifically, the processor 11 can be a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof.

[0017] Furthermore, the processor 11 loads the program stored in the ROM 13 and recording medium 16 into the RAM 14 and executes each process coded in the program. The processor 11 functions as part or all of the information processing device 100. Specifically, the processor 11 executes the object trajectory calculation process described later.

[0018] IF12 transmits and receives data to and from external devices. Specifically, the information processing device 100 receives time-series frame images through IF12. The information processing device 100 also outputs the calculated object trajectory to a display device or other external devices through IF12.

[0019] ROM 13 stores various programs executed by processor 11. RAM 14 is used as working memory while processor 11 is executing various processes.

[0020] DB15 stores various algorithms, data, machine learning models, etc., that the information processing device 100 uses when it performs the object trajectory calculation process described later.

[0021] The recording medium 16 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory. The recording medium 16 may be configured to be detachable from the information processing device 100. The recording medium 16 stores various programs executed by the processor 11.

[0022] In addition to the above, the information processing device 100 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the operator of the information processing device 100.

[0023] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the information processing device 100. As shown in the figure, the information processing device 100 includes a trajectory fragment generation unit 110, a fragment-to-similarity calculation unit 120, an appearance similarity calculation unit 130, a coordinate similarity calculation unit 140, a similarity integration unit 150, and an optimal trajectory calculation unit 160. The appearance similarity calculation unit 130 also includes a correlation calculation unit 131 and a similarity aggregation unit 132.

[0024] In the above configuration, the trajectory fragment generation unit 110 is an example of a trajectory fragment generation means, the correlation calculation unit 131 is an example of a correlation calculation means, the similarity aggregation unit 132 and the appearance similarity calculation unit 130 are examples of appearance similarity calculation means, the coordinate similarity calculation unit 140 is an example of a coordinate similarity calculation means, and the fragment-to-similarity calculation unit 120 and the similarity integration unit 150 are examples of fragment-to-similarity calculation means.

[0025] The trajectory fragment generation unit 110 receives a time-series frame image V as input. The trajectory fragment generation unit 110 generates trajectory fragments from the time-series frame image V. A "trajectory fragment" is a fragment of a trajectory formed by concatenating object information obj related to the same object detected in different frames.

[0026] The trajectory fragment generation unit 110 generates a trajectory fragment for each object included in the input time-series frame image V. For example, if the index of an object is "i", the trajectory fragment of object i is generated by arranging the object information obj of object i (physically the same object but detected on different frame images) included in the time-series frame image V in chronological order, (obj i,1 ,...,obj i,Li ) is expressed as follows: Here, each object information obj i,j The index (time) of the frame image in which it was detected is t i,j And the coordinate (bounding box) information B in that frame image. i,j And the external feature quantity F of that object i,j This includes,

[0027] In one specific example, the trajectory fragment generation unit 110 performs object detection on each frame image included in the time-series frame image V and detects the bounding box of the object in each frame image. The trajectory fragment generation unit 110 then obtains the time t of that frame image, the position coordinates B of its bounding box, and the appearance feature quantity F of the object contained within that bounding box as object information obj for that frame. The appearance feature quantity F is a feature quantity that represents the appearance of the object contained within the detected bounding box, and is also called a feature vector. The appearance feature quantity F is a numerical representation of the visual elements of the object, such as its color, shape, and texture. For example, when using object detection such as R-CNN, the feature quantity extracted using a convolutional neural network (CNN) from the proposed region (RoI) extracted from the input frame image can be used as the appearance feature quantity F. The trajectory fragment generation unit 110 generates trajectory fragments for each object by arranging the object information obj corresponding to the same object in time series from the object information obj in each frame thus obtained.

[0028] The trajectory fragment generation unit 110 generates a set S of trajectory fragments for N objects included in the input time-series frame image V, and outputs it to the fragment-to-similarity calculation unit 120.

[0029]

number

[0030] The fragment pair similarity calculation unit 120 takes a set S of trajectory fragments as input and calculates the similarity of all pairs of trajectory fragments (hereinafter referred to as "trajectory fragment pairs"). Specifically, the fragment pair similarity calculation unit 120 calculates the similarity of each trajectory fragment pair by integrating the similarity of its appearance features (hereinafter referred to as "appearance similarity") and the similarity calculated from the changes in time and coordinates (hereinafter referred to as "coordinate similarity"). The appearance similarity is calculated by the appearance similarity calculation unit 130, and the coordinate similarity is calculated by the coordinate similarity calculation unit 140.

[0031] First, the calculation of appearance similarity will be described. The fragment pair similarity calculation unit 120 inputs the trajectory fragment pair P1(i,j) shown in FIG. 3 to the appearance similarity calculation unit 130. The appearance similarity calculation unit 130 extracts objects one by one from the input trajectory fragment pair P1(i,j) and inputs them to the correlation calculation unit 131. The correlation calculation unit 131 generates the correlation of the objects (hereinafter referred to as "object correlation") for all combinations of the objects. Next, when the object correlation is obtained for all combinations of the objects, the similarity aggregation unit 132 aggregates the appearance feature quantity pair P3(i,j) and the object correlation to calculate the appearance similarity a ij as follows. This will be described in detail below.

[0032] First, the correlation calculation unit 131 calculates the following correlation s i,k;j,l for all object pairs extracted from the trajectory fragment pair P1(i,j).

[0033]

Equation

[0034] The correlation s i,k;j,l is a real value representing the degree to which each object pair contributes to the appearance similarity of the trajectory fragment pair. The correlation calculation unit 131 calculates the correlation s i,k;j,l based on the object information (time, coordinates, appearance feature quantity) included in the input trajectory fragment pair P1(i,j).

[0035] As described above, even for the same object, the appearance features change due to the movement or orientation change of the object. For example, when the target object is a person, the correlation between objects in a state of walking from left to right in the video is large. On the other hand, suppose a person walking to the right changes direction and walks towards the front in the video. In this case, the correlation between the person walking to the right and the person walking towards the front becomes small because the appearance features change. Therefore, by calculating the appearance similarity considering the correlation value, it is possible to reduce the influence on the appearance similarity when the object moves or the orientation of the object changes and the appearance becomes dissimilar in the video.

[0036] In one example, the correlation calculation unit 131 calculates the correlation s as follows: i,k;j,l This can be calculated as the dot product of the external features.

[0037]

number

[0038] In other examples, the correlation calculation unit 131 calculates the correlation s as follows: i,k;j,l This can be calculated as the cosine similarity of the appearance features.

[0039]

number

[0040] In another example, the correlation calculation unit 131 uses a pre-trained neural network, specifically a multilayer perceptron (MLP), to calculate the correlation s as follows: i,k;j,l This can be determined. In this case, the MLP parameters are determined by prior training.

[0041]

number

[0042] Then, the correlation calculation unit 131 calculates the correlation s obtained by one of the above methods. i,k;j,l The results are output to the similarity calculation unit 132.

[0043] The similarity calculation unit 132 calculates the appearance similarity of the fragment pairs based on the correlation calculated for all object pairs and the sequence of appearance features. Specifically, the similarity calculation unit 132 calculates the correlation s calculated for all object pairs from the correlation calculation unit 131. i,k;j,l The system receives the fragment pair similarity calculation unit 120 and the appearance feature pair P3(i,j). The similarity aggregation unit 132 then calculates the appearance similarity a for each fragment pair. ij Calculate.

[0044] Now, let's assume d(F i,k ,F j,l ) for appearance features vs F i,k ,F j,l Let the similarity between them be {r i,k;j,l}=H({s i,k;j,l Let}) be the correlation modulated using a certain modulation function H. In this case, the similarity aggregation unit 132 calculates the appearance similarity a using the following formula. ij We seek.

[0045]

number

[0046] This results in visual similarity a ij This uses the correlation value as a weight for the appearance feature versus F. i,k ,F j,l The similarity between the two is weighted and added together. Therefore, as mentioned above, for pairs of objects whose external features are no longer similar due to movement or changes in orientation, the similarity between the external features becomes the final external similarity a. ij This becomes less likely to be reflected, and changes in the movement or orientation of objects affect the appearance similarity a ij This can reduce the impact on [the subject].

[0047] In one example, the modulation function H can be a function that outputs the maximum value, as follows:

[0048]

number

[0049] In other examples, the modulation function H can be the following softmax function.

[0050]

number

[0051] The similarity aggregation unit 132 calculates the appearance similarity a ijThe results are output to the fragment-to-similarity calculation unit 120.

[0052] Next, the calculation of coordinate similarity will be explained. The fragment pair similarity calculation unit 120 outputs the coordinate information pair P2(i,j) to the coordinate similarity calculation unit 140. The coordinate information pair P2(i,j) is the coordinate information obj ~ i,j Includes coordinate information obj ~ i,j This is object information obj i,j From the external features F i,j This is what is left excluding. That is, coordinate information obj ~ i,j This is the frame index (time) t i,j And coordinate (bounding box) information B i,j This includes.

[0053] The coordinate similarity calculation unit 140 calculates coordinate similarity using the coordinate information pair P2(i,j). Specifically, the coordinate similarity calculation unit 140 assumes that the center coordinates of the bounding boxes of each trajectory fragment constituting the trajectory fragment pair are moving in a straight line at a constant velocity, and extrapolates the bounding boxes up to an intermediate time between the two trajectory fragments. At this time, it is assumed that the shape (width and height) of the bounding boxes does not change at the endpoints of the trajectory fragments. Next, the coordinate similarity calculation unit 140 calculates IoU (Intersection over Union) between the bounding boxes extrapolated from both trajectory fragments at the intermediate time, and calculates the coordinate similarity b ij The coordinate similarity calculation unit 140 then calculates the coordinate similarity b ij The results are output to the fragment-to-similarity calculation unit 120.

[0054] The similarity integration unit 150 receives the appearance similarity a from the fragment-to-segment similarity calculation unit 120. ij and coordinate similarity b ij And, as such, their linear combination c ij The following calculation is performed:

[0055]

number

[0056] Then, the similarity integration unit 150 calculates the linear combination c ij This is output to the fragment-to-similarity calculation unit 120 as an integrated similarity score.

[0057] The fragment-to-similarity calculation unit 120 calculates the integrated similarity c ij Based on this, the similarity C of the following pairs of trajectory fragments is output to the optimal trajectory calculation unit 160.

[0058]

number

[0059] The optimal trajectory calculation unit 160 calculates the optimal combination of trajectory fragments based on the similarity C of the trajectory fragment pairs and outputs the optimal trajectory, i.e., the set T of complete object trajectories for each object. Specifically, the optimal trajectory calculation unit 160 finds the optimal trajectory by solving the following constrained optimization problem using a mathematical optimization solver.

[0060] (1) Variable For each trajectory fragment pair P1(i,j), the variable x takes a value of either 0 or 1. ij We define whether the two trajectory fragments constituting the trajectory fragment pair P1(i,j) are temporally adjacent trajectory fragments in a single trajectory (i.e., x ij =1), or not (i.e., x ij Let x = 0 represent this. ij Cases where = 0 include cases where the two trajectory fragments belong to the trajectories of different objects, and cases where the two trajectory fragments belong to a single trajectory (they are from the same object), but another trajectory fragment exists between the two trajectory fragments on that trajectory.

[0061] (2) Constraints If temporal consistency is not established (i and j have a temporal overlap): x ij(=0), objects do not divide or combine. That is, a given trajectory segment is connected to at most one trajectory segment at an earlier time. This can be written mathematically as follows:

[0062]

number

[0063] Furthermore, a given trajectory fragment connects to at most one trajectory fragment at a later time.

[0064] (3) Optimization objective function The following function is the objective function.

[0065]

number

[0066] Thus, the optimal trajectory calculation unit 160 outputs a set T of complete object trajectories for each object based on the similarity C of the trajectory fragment pairs.

[0067] (Estimation process) Next, the process of calculating the object trajectory by the information processing device 100 will be described. Figure 4 is a flowchart of the object trajectory calculation process. This process is realized when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 3.

[0068] First, the trajectory fragment generation unit 110 generates trajectory fragments from the input time-series frame images (step S11). Next, the correlation calculation unit 131 calculates object correlations based on the trajectory fragment pairs input from the fragment-pair similarity calculation unit 120 (step S12). Next, the similarity aggregation unit 132 aggregates the object correlations and the appearance feature pairs input from the fragment-pair similarity calculation unit 120 to calculate appearance similarity (step S13).

[0069] Furthermore, the coordinate similarity calculation unit 140 calculates coordinate similarity based on the coordinate information pairs input from the fragment pair similarity calculation unit 120 (step S14). Note that steps S12 to S13 and step S14 may be performed in reverse order, or they may be performed in parallel in time.

[0070] Next, the similarity integration unit 150 integrates the coordinate similarity and appearance similarity to generate similarity between pairs of trajectory fragments (step S15). Then, the optimal trajectory calculation unit 160 calculates and outputs the optimal trajectory based on the similarity between pairs of trajectory fragments (step S16). The object trajectory calculation process then ends.

[0071] [Examples of application] The information processing methods disclosed herein can be applied, for example, to the management of the actions of people, robots, and other entities in industrial settings. Specifically, the methods disclosed herein can be used for automating warehouses in the logistics industry, improving store efficiency in the retail industry, streamlining site management in the construction industry, and automating inspections in the manufacturing industry.

[0072] Figure 5 shows an example of an action management system to which the information processing device of the present disclosure is applied. The action management system 200 comprises a camera 210, the information processing device 100 described above, an action estimation device 220, and a management DB 230. The camera 210 is installed at the site to be managed and captures video of the site and transmits it to the information processing device 100. The information processing device 100 tracks a person working at the site using the method described above and transmits the person's trajectory to the action estimation device 220.

[0073] The behavior estimation device 220 estimates what actions and tasks each person is performing based on the input person's trajectory. The behavior estimation device 220 can, for example, use a deep learning model that has been pre-trained to estimate the actions of people in the video from the input person's trajectory. The behavior estimation device 220 then associates the estimated actions of each person with the time and location at the site and records them as an action history in the management DB 230. This allows the site manager to manage workers based on the action history of each person recorded in the management DB 230.

[0074] <Second Embodiment> Figure 6 is a block diagram showing the functional configuration of an information processing device according to another example of the present disclosure. The information processing device 70 comprises a trajectory fragment generation means 71, a correlation calculation means 72, an appearance similarity calculation means 73, a fragment-to-similarity calculation means 74, and an object trajectory calculation means 75.

[0075] Figure 7 is a flowchart of the processing performed by the information processing device described above. The trajectory fragment generation means 71 generates a trajectory fragment that shows at least a portion of the trajectory of an object moving in a time-series frame image and includes object information indicating the time, coordinates, and appearance features of the object in each frame image (step S71). The correlation calculation means 72 calculates the correlation of the appearance features for an object pair created by taking one object each from a first trajectory fragment and a second trajectory fragment, which are a pair of trajectory fragments (step S72). The appearance similarity calculation means 73 calculates the appearance similarity of the trajectory fragment pair based on the correlation and the similarity between the appearance features of the object pair (step S73).

[0076] The fragment pair similarity calculation means 74 calculates a fragment pair similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, using the appearance similarity (step S74). Then, the object trajectory calculation means 75 combines a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate the object trajectory for the same object (step S75).

[0077] According to the information processing device 70 described above, it becomes possible to correctly track the same object by taking into account changes in its appearance due to movement or changes in orientation.

[0078] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0079] (Note 1) A trajectory fragment generation means generates a trajectory fragment that shows at least a portion of the trajectory of an object moving in a time-series frame image, and includes object information showing the time, coordinates, and appearance features of the object in each frame image, Correlation calculation means for calculating the correlation of the appearance features for an object pair formed by taking one object each from a first trajectory fragment and a second trajectory fragment included in a pair of trajectory fragments, An appearance similarity calculation means for calculating the appearance similarity of the trajectory fragment pair based on the correlation and the similarity between the appearance features of the object pair, A fragment-to-fragment similarity calculation means calculates a fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, using the aforementioned appearance similarity. An object trajectory calculation means that calculates an object trajectory for the same object by combining a plurality of trajectory fragment pairs based on the aforementioned fragment pair similarity, An information processing device equipped with the following features.

[0080] (Note 2) The system includes a coordinate similarity calculation means for calculating coordinate similarity that indicates the consistency of time and coordinates contained in each of the first and second trajectory fragments, The information processing apparatus described in Appendix 1, wherein the fragment pair similarity calculation means calculates the fragment pair similarity based on the appearance similarity and the coordinate similarity.

[0081] (Note 3) The information processing apparatus according to Appendix 1 or 2, wherein the appearance similarity calculation means uses the correlation as a weight and calculates the appearance similarity for a plurality of object pairs by weighting and adding the similarities between the appearance features of the object pairs.

[0082] (Note 4) The aforementioned appearance similarity calculation means is an information processing apparatus as described in Appendix 3, wherein the weight of the maximum value of the correlation is set to 1, and the weight of correlations other than the maximum value is set to 0.

[0083] (Note 5) The aforementioned appearance similarity calculation means is an information processing device according to Appendix 3, which inputs the correlation value into a softmax function to calculate the weight.

[0084] (Note 6) The information processing apparatus according to any one of the appendices 1 to 5, wherein the correlation calculation means calculates the correlation as the dot product or cosine similarity of the appearance features of the object pair.

[0085] (Note 7) The correlation calculation means is an information processing device according to any one of the appendices 1 to 5, which calculates the correlation by inputting the object information of the object pair into a pre-trained neural network.

[0086] (Note 8) The object trajectory calculation means is an information processing device according to any one of the appendices 1 to 7, which calculates the object trajectory by connecting a plurality of temporally adjacent pairs of trajectory fragments in a single trajectory of the same object.

[0087] (Note 9) A method of information processing performed by a computer, This generates a trajectory fragment that shows at least a portion of the trajectory of an object moving within a time-series frame image, and includes object information such as the time, coordinates, and appearance features of the object in each frame image. For a pair of objects created by taking one object each from the first and second trajectory fragments included in the pair of trajectory fragments, the correlation of the aforementioned appearance features is calculated. Based on the correlation and the similarity between the appearance features of the object pair, the appearance similarity of the trajectory fragment pair is calculated. Using the aforementioned visual similarity, the fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, is calculated. An information processing method for calculating an object trajectory for the same object by combining multiple pairs of trajectory fragments based on the aforementioned fragment pair similarity.

[0088] (Note 10) This generates a trajectory fragment that shows at least a portion of the trajectory of an object moving within a time-series frame image, and includes object information such as the time, coordinates, and appearance features of the object in each frame image. For a pair of objects created by taking one object each from the first and second trajectory fragments included in the pair of trajectory fragments, the correlation of the aforementioned appearance features is calculated. Based on the correlation and the similarity between the appearance features of the object pair, the appearance similarity of the trajectory fragment pair is calculated. Using the aforementioned visual similarity, the fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, is calculated. A program that causes a computer to perform a process of calculating the object trajectory of the same object by combining multiple pairs of trajectory fragments based on the aforementioned fragment pair similarity.

[0089] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 10 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0090] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of Symbols]

[0091] 11 processors 100 Information Processing Devices 110 Trajectory fragment generation unit 120 Fragment-to-similarity calculation unit 130 Appearance Similarity Calculation Unit 131 Correlation Calculation Unit 132 Similarity Calculation Department 140 Coordinate similarity calculation unit 150 Similarity integration part 160 Optimal Trajectory Calculation Unit

Claims

1. A trajectory fragment generation means generates a trajectory fragment that shows at least a portion of the trajectory of an object moving in a time-series frame image, and includes object information showing the time, coordinates, and appearance features of the object in each frame image, A correlation calculation means for calculating the correlation of the appearance features for an object pair formed by taking one object each from a first trajectory fragment and a second trajectory fragment included in a pair of trajectory fragments, An appearance similarity calculation means for calculating the appearance similarity of the trajectory fragment pair based on the correlation and the similarity between the appearance features of the object pair, A fragment-to-fragment similarity calculation means calculates a fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, using the aforementioned appearance similarity. An object trajectory calculation means that calculates an object trajectory for the same object by combining a plurality of trajectory fragment pairs based on the aforementioned fragment pair similarity, An information processing device equipped with the following features.

2. The system includes a coordinate similarity calculation means for calculating coordinate similarity that indicates the consistency of time and coordinates contained in each of the first and second trajectory fragments, The information processing apparatus according to claim 1, wherein the fragment pair similarity calculation means calculates the fragment pair similarity based on the appearance similarity and the coordinate similarity.

3. The information processing apparatus according to claim 1, wherein the appearance similarity calculation means uses the correlation as a weight and calculates the appearance similarity for a plurality of object pairs by weighting and adding the similarities between the appearance feature quantities of the object pairs.

4. The information processing apparatus according to claim 3, wherein the appearance similarity calculation means sets the weight of the maximum value of the correlation to 1 and the weight of correlations other than the maximum value to 0.

5. The information processing apparatus according to claim 3, wherein the appearance similarity calculation means inputs the correlation value into a softmax function to calculate the weight.

6. The information processing apparatus according to claim 1, wherein the correlation calculation means calculates the correlation as the dot product or cosine similarity of the appearance features of the pair of objects.

7. The information processing apparatus according to claim 1, wherein the correlation calculation means calculates the correlation by inputting the object information of the object pair into a pre-trained neural network.

8. The information processing apparatus according to claim 1, wherein the object trajectory calculation means calculates the object trajectory by connecting a plurality of temporally adjacent pairs of trajectory fragments in a single trajectory of the same object.

9. A method of information processing performed by a computer, The system generates a trajectory fragment that shows at least a portion of the trajectory of an object moving within a time-series frame image, and includes object information such as the time, coordinates, and appearance features of the object in each frame image. For a pair of objects created by taking one object each from the first and second trajectory fragments included in the pair of trajectory fragments, the correlation of the appearance features is calculated. Based on the correlation and the similarity between the appearance features of the object pair, the appearance similarity of the trajectory fragment pair is calculated. Using the aforementioned appearance similarity, the fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, is calculated. An information processing method for calculating an object trajectory for the same object by combining multiple pairs of trajectory fragments based on the aforementioned fragment pair similarity.

10. The system generates a trajectory fragment that shows at least a portion of the trajectory of an object moving within a time-series frame image, and includes object information such as the time, coordinates, and appearance features of the object in each frame image. For a pair of objects created by taking one object each from the first and second trajectory fragments included in the pair of trajectory fragments, the correlation of the appearance features is calculated. Based on the correlation and the similarity between the appearance features of the object pair, the appearance similarity of the trajectory fragment pair is calculated. Using the aforementioned appearance similarity, the fragment-to-fragment similarity, which is the similarity between the first trajectory fragment and the second trajectory fragment, is calculated. A program that causes a computer to perform a process of calculating the object trajectory of the same object by combining multiple pairs of trajectory fragments based on the aforementioned fragment pair similarity.

Images