Tracking device

Abstract

To provide a device which performs accurate tracking or an object DB generation device which facilitates generation processing even when accuracy of tracking is suspicious, not limited to a state in which a subject overlaps with another subject.SOLUTION: In a tracking device, moving object extraction means 2 acquires respective frame images of a moving image obtained by capturing a player as a moving object, and extracts the player from the respective acquired frame images. Tracking means 4 tracks movements of players between the respective frames to generate tracking trajectories, player by player. Cutting means 6 determines that accuracy of tracking of a frame OCL where occlusion occurs is suspicious and cuts the tracking trajectory. Connection means 8 refers to an object DB 10 to determine which player's trajectory tracking trajectories CL1, CL2, CL3, and CL4 before and after breaking are. Connection means 8 connects tracking trajectories determined as tracking trajectories that the same player makes.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a device for tracking a moving object. [Background technology]

[0002] By recording the movements of players on the field, it is possible to check the effectiveness of tactics and whether each player is moving in accordance with the tactics, etc. This allows for improvements to be made to tactics and the movements of each player.

[0003] Furthermore, such recording of movements is not limited to athletes, but is also carried out for other moving objects in general, such as vehicles, depending on the purpose.

[0004] Devices are used to capture images of moving objects such as athletes using cameras and track them. For example, there are devices that extract feature points of the subject in each frame, extract the subject using those feature points, and acquire its position. Because images are captured using a camera, tracking can be performed without the need to attach transmitters to athletes.

[0005] However, such tracking devices have a problem in that the accuracy of tracking decreases when a subject overlaps with another subject (called occlusion).

[0006] To solve this problem, for example, the device described in Patent Document 1 predicts the position of the feature points of the subject in the next frame. When occlusion occurs and tracking becomes impossible, the predicted feature points are used to estimate the position of the subject, and tracking continues based on the estimated position. This makes it possible to continue tracking even when occlusion occurs. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2016-173795 Summary of the Invention [Problem to be solved by the invention]

[0008] However, in conventional technologies such as those disclosed in Patent Document 1, tracking is performed based on the similarity of feature points between adjacent frames, and the tracking accuracy is insufficient, and it is not possible to handle cases where occlusion continues for a long time. Furthermore, similar problems arise not only in cases of occlusion, but also in cases where the accuracy of tracking is questionable (such as when tracking is performed across frames where the target object could not be detected).

[0009] Furthermore, the process of generating an object DB that associates objects with their images and is used to identify the objects is complicated.

[0010] Taking into consideration any of the above problems, the present invention aims to provide a device that can perform accurate tracking even when the accuracy of tracking is in doubt, not limited to occlusion, or an object DB generation device that is easy to generate. [Means for solving the problem]

[0011] Some independent features of the present invention are listed below. Each feature is independent and can be combined in any combination.

[0012] (1)(2) The tracking device of this invention is a tracking device that tracks the movement of moving objects in a video in which the moving objects are captured, and is equipped with: a moving object extraction means that acquires each frame image of a video in which multiple moving objects are captured and extracts the moving objects in each frame; a tracking means that tracks the movement of each extracted moving object between frames and generates tracking connection information; a disconnection means that, when it is determined that the accuracy of the tracking of the moving objects between frames by the tracking means is in doubt, disconnects the tracking connection information at the frame in which it is determined that the accuracy is in doubt; and a connection means that, for the tracking connection information disconnected by the disconnection means, compares images of the moving objects in a series of frames in each connection information with multiple moving objects registered in advance, identifies the moving objects in each connection information, and connects the tracking connection information of the same moving object.

[0013] Therefore, connection information that was cut off due to doubts about accuracy can be accurately connected to obtain connection information for each moving object.

[0014] (3) The tracking device of the present invention is characterized in that the connection means compares images of moving objects in a series of frames in each connection information, even for tracking connection information in which tracking by the tracking means has been interrupted, with multiple moving objects registered in advance, to identify the moving object in each connection information, and connects the tracking connection information of the same moving object.

[0015] Therefore, accurate tracking can be performed not only when the accuracy of tracking is doubtful, but also when tracking is not possible.

[0016] (4) The tracking device according to the present invention is characterized in that the moving object is a player on a stadium.

[0017] Therefore, the movement of the players can be tracked.

[0018] (5) The tracking device according to the present invention is characterized in that the connection means connects the tracking connection information in consideration of the position, speed or moving direction of the player, which is the moving object.

[0019] Therefore, the connection information can be connected with higher accuracy.

[0020] (6) The tracking device of the present invention is characterized in that the connection means identifies the player of the tracking connection information by providing images of the player, which is a moving object, in a series of frames of the tracking connection information to a trained person identification model that has been trained using distance deep learning to output similar features for the same player and dissimilar features for different players.

[0021] Therefore, the players can be identified with high accuracy.

[0022] (7) The tracking device of the present invention is characterized in that the learned person identification model is obtained by extracting each player using a moving object extraction means from multiple frames of images taken just before or just after the start of a match, identifying the player name of each extracted player based on correspondence data that associates the player name with the position, and learning the person identification model based on the images in which the player name has been identified.

[0023] Therefore, it is possible to generate a trained person identification model from captured video, which is efficient.

[0024] (8) The tracking device of the present invention is characterized in that the connection means identifies the moving object in the tracking connection information by comparing images of the moving object in a series of frames in the tracking connection information with a plurality of moving objects registered in advance, even for tracking connection information that has been tracked across all frames.

[0025] Therefore, it is possible to efficiently associate moving objects with connection information.

[0026] (9)(10) The object DB generation device of the present invention includes an object extraction means for acquiring each frame image of a moving image of an object and extracting the object image in each frame, an object ID assignment means for assigning an object ID for distinguishing the object to the object image extracted in each frame based on the position of the object or the tracking of the object or both, and a generation means for generating an object DB based on the object image of each object assigned the object ID.

[0027] Therefore, it is possible to track objects to which IDs have been assigned and generate a highly accurate player DB.

[0028] (11) The object DB generation device of the present invention is characterized in that the object DB is a trained object identification model that is trained using object images through deep distance learning to output similar features for the same object and dissimilar features for different objects.

[0029] Therefore, a highly accurate object DB can be generated.

[0030] (12) The object DB generating device according to the present invention is characterized in that the player ID assigning means identifies the player name by referring to data indicating the position of the player immediately before or immediately after the start of the game.

[0031] Therefore, it is possible to generate an object DB that specifies the names of players.

[0032] (a) The tracking device of this invention is a tracking device that tracks the movement of moving objects in a video in which the moving objects are captured, and is equipped with a moving object extraction means that acquires each frame image of a video in which multiple moving objects are captured and extracts the moving objects in each frame, a tracking means that tracks the movement of each extracted moving object between frames and generates tracking connection information, and a connection means that, for tracking connection information in which tracking by the tracking means has been interrupted, compares images of the moving object in a series of frames in each connection information with multiple moving objects registered in advance, identifies the moving object in each connection information, and connects the tracking connection information of the same moving object.

[0033] Therefore, even if tracking is interrupted, accurate tracking can be performed.

[0034] In the embodiment, step S4 corresponds to the "moving object extraction means."

[0035] In this embodiment, step S5 corresponds to the "tracking means."

[0036] In the embodiment, step S9 corresponds to the "cutting means."

[0037] In the embodiment, step S12 corresponds to the "connecting means."

[0038] In this embodiment, step S24 corresponds to the "player extraction means."

[0039] In this embodiment, step S25 corresponds to the "player ID assigning means."

[0040] In the embodiment, step S27 corresponds to the "generation means."

[0041] The concept of "device" includes not only what is constituted by one computer, but also what is constituted by multiple computers connected via a network, etc. Therefore, when the means of the present invention (or even a part of the means) is distributed among multiple computers, these multiple computers correspond to the device.

[0042] The term "program" is a concept that includes not only programs that can be executed directly by a CPU, but also programs in source format, compressed programs, encrypted programs, and programs that work in conjunction with an operating system to perform their functions. [Brief explanation of the drawings]

[0043] [Figure 1] 1 illustrates a functional configuration of a tracking device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of a tracking device. [Figure 3] 10 is a flowchart of a tracking program. [Figure 4] FIG. 2 is a diagram showing frames of a captured video. [Figure 5] FIG. 10 is a diagram showing an image in which players are extracted. [Figure 6] FIG. 10 is a diagram showing the tracking trajectory of a player. [Figure 7] FIG. 10 is a diagram illustrating determination of occlusion. [Figure 8] FIG. 10 is a diagram illustrating determination of occlusion. [Figure 9] FIG. 10 is a diagram illustrating a cut in a tracking trajectory. [Figure 10] FIG. 10 is a diagram showing a tracking trajectory arranged on a frame. [Figure 11] FIG. 10 is a diagram showing learning of a player DB. [Figure 12] FIG. 10 is a diagram showing a histogram of player IDs generated by a player DB. [Figure 13] FIG. 10 is a diagram for explaining player identification of each tracking trajectory based on a histogram. [Figure 14]FIG. 10 is a diagram for explaining player identification of each tracking trajectory based on a histogram. [Figure 15] 10 shows the functional configuration of a player DB generation device according to a second embodiment. [Figure 16] 10 is a flowchart of a player DB generation program. DETAILED DESCRIPTION OF THE INVENTION

[0044] 1. First embodiment 1.1 Functional configuration Figure 1 shows the functional configuration of a tracking device according to one embodiment of the present invention. Moving object extraction means 2 acquires each frame image of a moving image of a player, which is a moving object. Players are extracted from each acquired frame image. Tracking means 4 tracks the movement of players between frames and generates a tracking trajectory for each player. In Figure 1, there is a frame OCL in which occlusion occurred, so three tracking trajectories CL1, CL2, and CL3 are formed.

[0045] The cutting means 6 determines that the accuracy of tracking is doubtful for a frame OCL where occlusion has occurred. Then, the cutting means 6 cuts the tracking trajectory in this frame OCL. Therefore, the tracking trajectory CL1 is cut into CL1 and CL4. Furthermore, the trajectory of the occluded part of the tracking trajectory CL2 is deleted. Note that the cutting means 6 also cuts when the accuracy of tracking is doubtful for reasons other than occlusion (frames where a moving object could not be extracted).

[0046] The connection means 8 determines which player's trajectory each of the tracking trajectories CL1, CL2, CL3, and CL4 before and after the interruption belongs to by referring to the object DB 10. Images of each player are registered in the object DB 10. The connection means 8 refers to the images in the object DB 10 based on the series of player images in each frame included in each tracking trajectory CL1, CL2, CL3, and CL4, and identifies which player the trajectory belongs to based on the degree of similarity.

[0047] The connection means 8 connects tracking trajectories that are determined to belong to the same player. In FIG. 1, for example, tracking trajectories CL1 and CL4 are connected, and tracking trajectories CL2 and CL3 are connected. In this way, it is possible to clarify the connection relationship before and after a point where the reliability of tracking is doubtful.

[0048] As described above, the system references the object DB to identify the player from the series of player images included in the disconnected tracking trajectories, and then connects the disconnected tracking trajectories together, thereby improving the overall accuracy of tracking.

[0049] 1.2 Hardware configuration The hardware configuration of the tracking device is shown in Figure 2. Connected to a CPU 20 are a memory 22, a display 24, a communication circuit 26, an SSD 28, a DVD-ROM drive 30, an I / O port 32, and a keyboard / mouse 34. The communication circuit 26 is a circuit for connecting to the Internet.

[0050] The SSD 28 stores an operating system 36, a tracking program 38, and video data 40. The tracking program 38 functions in cooperation with the operating system 36. These programs were originally recorded on a DVD-ROM 42 and were installed on the SSD 28 via the DVD-ROM drive 30.

[0051] The video data 40 is footage of players during a match. The data was recorded on a portable recording medium 44 and then imported into the SSD 28 via the I / O port 32. The video data 40 may also be imported via the Internet using the communication circuit 26.

[0052] 1.3 Tracking process 3 shows a flowchart of the tracking program 38. The CPU 20 reads video data 40 of the players from the SSD 28 and stores it in the memory 22 (step S21). If the volume of the video data 40 is large, it is stored in the memory 22 in sequence as the process progresses.

[0053] Next, CPU 20 acquires the first frame of the video data as a target frame (step S3). An example image of the target frame is shown in Figure 4. In this embodiment, video data of the entire soccer field captured by a camera installed on the soccer field is used.

[0054] The CPU 20 extracts a person from the target frame (step S4). This process can be performed using a trained model that has been trained by enclosing the person image in the image in a rectangular frame (given the coordinates of the top left and bottom right of the rectangular frame).

[0055] For example, an untrained estimation model using the YOLO algorithm may be replaced with a trained model trained on people, or a trained model trained to estimate the type and position of objects including people using the YOLO algorithm may be used.

[0056] The trained model outputs the coordinates of the top left and bottom right of the rectangle (boundary box) surrounding the person as the result of person extraction. This is shown as an image in Figure 5. Each player is extracted and surrounded by a boundary box.

[0057] Substitute players, spectators, coaches, etc. will also be extracted. In this embodiment, the boundary boxes of people outside the predefined playing area in the image are deleted based on the coordinates of the center point of the bottom edge of the boundary box (the position of the person's feet). This means that only people inside the playing area will be extracted as players.

[0058] Next, the CPU 20 generates a tracking trajectory of each player as tracking connection information by associating it with the player extracted in the frame immediately preceding the target frame (step S5). However, since there is no frame immediately preceding the first frame, this process is performed from the next frame onwards. This player extraction and tracking process is performed for all frames (steps S2 and S6).

[0059] Matching of players between frames is performed based on the positional relationship between the boundary boxes in the previous frame and the target frame and the characteristics of the player images within both boundary boxes. That is, the player's possible movement range from the previous frame is estimated based on the player's past trajectory, and the corresponding boundary box in the target frame is found and matched. In addition, the image features of the player in the boundary box in the previous frame are compared with the image features of the player in the boundary box in the target frame, and similar images are matched. In this embodiment, the tracking described above is performed using a rigid body tracking method such as CSRT or an MOT method such as ByteTrack. Of course, other methods may also be used.

[0060] FIG. 6A shows a schematic diagram of player tracking. The figure shows the positions of extracted players P1 and P2 in frames F1 to F9. The frames progress in the order of F1, F2, etc. By tracking the players between frames, a tracking trajectory CL1 for player P1 and a tracking trajectory CL2 for player P2 can be obtained. Here, CL1 and CL2 are tracking trajectory IDs assigned to the tracking trajectories.

[0061] As shown in Figure 6B, even if player P1 cannot be extracted in frames F3 and F4, the tracking trajectory CL1 is formed as much as possible. That is, the players in frames F2 and F5 are associated (skipping frames) based on the degree of match with the predicted position three frames after frame F2 and the similarity between the image features of the player in frame F2 and the image features of the player in frame F5. This allows the tracking trajectory CL1 to be obtained without interruption.

[0062] 6B, if player P2 cannot be extracted in frames F4 and F5, the above-described correspondence may not be possible. If player P2's movement is unexpected and the position predicted in frame F3 differs significantly from the position in frame F6, or if the player's posture is abnormal (e.g., he is lying down) and the image features in frame F3 and frame F6 differ significantly, correspondence cannot be made. In this case, a tracking trajectory CL3 is generated in addition to tracking trajectory CL2.

[0063] 6C, when players overlap in the same frame (occlusion OCL), making it impossible to extract the players, and when matching by skipping the above-mentioned frames is also impossible, tracking trajectories CL2 and CL3 are generated. Note that even when occlusion OCL occurs, if it is possible to extract the players, a connected trajectory like tracking trajectory CL1 is generated.

[0064] Next, the CPU 20 finds a point for each player where the accuracy of tracking is questionable, cuts off the tracking trajectory at that point, and performs processing to reassign a tracking trajectory ID (steps S7 to S10). The following will explain this step by step.

[0065] First, the CPU 20 determines whether or not there is significant occlusion in each frame (step S7). This is because areas where significant occlusion occurs have overlapping images of players, raising doubts about the accuracy of tracking.

[0066] The determination of accuracy doubt due to occlusion in this embodiment will be described using Figure 7. Figure 7A shows two player boundary boxes BB1 and BB2 in the same frame in a state where no occlusion occurs. Note that the player boundary boxes BB1 and BB2 are extracted by the processing in step S4.

[0067] 7B shows a case where two player boundary boxes BB1 and BB2 are slightly occluded. In this embodiment, it is determined that this degree of overlap does not reduce the accuracy of player extraction and does not raise doubts about the accuracy of tracking.

[0068] 7C shows a case where two player boundary boxes BB1 and BB2 are significantly occluded. In this embodiment, it is determined that such a large overlap reduces the accuracy of player extraction and casts doubt on the accuracy of tracking.

[0069] In this embodiment, whether or not there is any doubt about the accuracy is determined according to the degree of overlap. For example, a case where occlusion occurs as shown in Fig. 8A will be described. In this case, the degree of overlap IOU is calculated by the following formula.

[0070] IOU=O / W Here, W is the area of ​​the entire region surrounding the outlines of the boundary boxes BB1 and BB2, as shown in Figure 8B. O is the area of ​​the overlapping portion of the boundary boxes BB1 and BB2, as shown in Figure 8C. If the overlap degree IOU exceeds a predetermined value (e.g., 0.1), it is determined that a severe occlusion has occurred, which casts doubt on the accuracy of tracking.

[0071] The CPU 20 determines whether or not there is a significant occlusion in each frame as described above, and then cuts off the tracking trajectory in the area of ​​the significant occlusion, even if the tracking trajectory is formed.

[0072] For example, it is assumed that tracking trajectories CL1, CL2, and CL3 are formed as shown in Fig. 9A by the processing of steps S2 to S6, and the CPU 20 has determined that serious occlusion has occurred in frames F5 and F6.

[0073] In this case, as shown in Fig. 9B, the CPU 20 cuts the tracking trajectory CL2 at the location where the severe occlusion occurred, and creates two tracking trajectories CL2 and CL4. That is, the tracking trajectory CL2 between frames F4 and F5, the tracking trajectory CL2 between frames F5 and F6, and the tracking trajectory CL2 between frames F6 and F7 are deleted, and the tracking trajectories are reassigned IDs (step S10). The tracking trajectory CL1 between frames F4 and F5 is also deleted.

[0074] Next, we will explain the case where it is determined that the accuracy of tracking is questionable because a player has not been extracted. The CPU 20 identifies a portion of each frame where selection and extraction has not been performed (step S8). Even if a tracking trajectory has been formed for that portion, the tracking trajectory is cut off (step S9).

[0075] For example, suppose that a tracking trajectory CL5 as shown in Fig. 9C is formed by the processing of steps S1 to S6. Here, player recognition fails in frame F6, but the tracking trajectory CL5 continues between frames F5 and F7.

[0076] The CPU 20 identifies a frame F6 from which a player has not been extracted (step S8), and cuts off the tracking trajectory CL5 at the frame F6. The cut tracking trajectory CL5 is reassigned a tracking trajectory ID to become tracking trajectories CL5 and CL6. In this way, the cut tracking trajectories CL5 and CL6 shown in FIG. 9D are obtained.

[0077] Next, the CPU 20 connects the tracking trajectories of the same player to complete the tracking trajectory for each player. This process is performed as follows.

[0078] 10 shows an example in which parts of tracking trajectories are arranged in association with parts of frames. For example, tracking trajectories CL1, CL2, and CL3 are continuous from frames F1 to F9. Tracking trajectory CL4 is continuous from frames F1 to F3. Tracking trajectory CL7 is continuous from frames F8 to F9, tracking trajectory CL5 is continuous from frames F1 to F5, and tracking trajectory CL6 is continuous from frames F5 to F9.

[0079] First, the CPU 20 refers to the player DB for each of these tracking trajectories and calculates which player the tracking trajectory belongs to (step S11). As shown in Fig. 11, this player DB is configured using a model that has undergone deep distance learning, whereby images of many postures of each target player (players who will participate in the game and be photographed) are provided and the same player is assigned a close vector and different players are assigned a distant vector.

[0080] Therefore, when an image of a player is input to this player DB, a vector is generated and a registered player (player used for learning) with a nearby vector can be output. For example, the ReID model can be used as such a model.

[0081] The CPU 20 provides the player image of each frame of each tracking trajectory in FIG. 10 to this player DB (trained ReID model) and calculates which player the image belongs to. For example, for tracking trajectory CL1, first, as shown in FIG. 12, the player image of frame F1 is provided to the player DB. This makes it possible to obtain player IDs from the registered player images in order of similarity. If player IDs range from 1 to 22 (assuming there are 22 players), a string of player IDs can be obtained as shown in the query ID array q1 in FIG. 12.

[0082] It shows that the image most similar to the given player image is the registered image of the player with ID "21", the next most similar is the registered image of the player with ID "21", the next most similar is the registered image of the player with ID "20", etc. In this embodiment, the top 10 similar player IDs are obtained.

[0083] The above process is also performed for subsequent frames F2 to F9. As a result, query ID arrays q2 to q9 can be obtained. The CPU 20 calculates the player IDs included in the query ID arrays q1 to q9 as a histogram (appearance frequency of each player). The histogram shown in FIG. 12 indicates that this tracking trajectory CL1 is most likely to be associated with player ID "21," followed by player ID "19," which is then most likely to be associated with player ID "20," and so on.

[0084] The CPU 20 generates histograms in the same manner for all other tracking trajectories CL2 and below. After generating histograms for identifying players for all tracking trajectories in this manner, the CPU 20 identifies players for each of the tracking trajectories CL1 to CL7 and connects tracking trajectories of the same player (step S12).

[0085] This process will be described with reference to Fig. 13. Fig. 13 shows histograms calculated for each of the tracking trajectories CL1 to CL7. The CPU 20 first obtains the player ID ranked first in the tracking trajectory CL1, and determines this as the player of the tracking trajectory CL1. Similarly, the CPU 20 determines players for the tracking trajectories CL2, CL3, and CL4. Therefore, as shown in Fig. 13, the tracking trajectory CL1 can be determined to have the player ID "21," the tracking trajectory CL2 to have the player ID "19," the tracking trajectory CL3 to have the player ID "18," and the tracking trajectory CL4 to have the player ID "20."

[0086] Next, the CPU 20 acquires the player ID at the top of the tracking trajectory CL5. As shown in Fig. 14, this is "18." This is the same player ID as the tracking trajectory CL3.

[0087] The CPU 20 determines whether the tracking trajectories CL3 and CL5 overlap in time. If they overlap in time, they do not coexist as the same player ID, and if they do not overlap in time, they coexist as the same player ID. As shown in FIG. 10, the tracking trajectories CL3 and CL5 overlap in time in frames F1 to F5. Therefore, the player determination for one of the tracking trajectories is incorrect.

[0088] In this embodiment, the player with the higher frequency is treated as the correct player. In the above example, the frequency of player ID "18" on tracking trajectory CL3 is 199, and the frequency of player ID "18" on tracking trajectory CL5 is 147. Therefore, tracking trajectory CL3 is set to have player ID "18."

[0089] For tracking trajectory CL5, the next ranked (second place) player ID "16" is determined.

[0090] Next, the CPU 20 obtains the player ID that is ranked first on the tracking trajectory CL6. In this case, the player ID "18" is obtained. This is the same as the tracking trajectory CL3, and since the tracking trajectories CL3 and CL6 overlap in time as shown in FIG. 10, they cannot coexist with the same player ID. Because the frequency of the tracking trajectory CL3 is higher, the player ID of the tracking trajectory CL6 is changed.

[0091] The next (second) player ID for tracking trajectory CL6 is "21." This is the same as tracking trajectory CL1. Moreover, as shown in FIG. 10, tracking trajectories CL1 and CL6 overlap in time, so they cannot coexist with the same player ID. Because tracking trajectory CL1 has a higher frequency, the player ID for tracking trajectory CL6 will be changed.

[0092] The next (third) player ID for tracking trajectory CL6 is "19." This is the same as tracking trajectory CL2. Moreover, as shown in FIG. 10, tracking trajectories CL2 and CL6 overlap in time, so they cannot coexist with the same player ID. Because tracking trajectory CL2 has a higher frequency, the player ID for tracking trajectory CL6 will be changed.

[0093] The next (fourth) player ID for tracking trajectory CL6 is "20," which is the same as tracking trajectory CL4. As shown in FIG. 10, tracking trajectories CL4 and CL6 do not overlap in time and can coexist. Therefore, as shown in FIG. 14, the player ID for tracking trajectory CL6 is determined to be "20."

[0094] Next, the CPU 20 acquires the player ID "16" at the top of the tracking trajectory CL7. This is the same as the tracking trajectory CL5. As shown in FIG. 10, the tracking trajectories CL5 and CL7 do not overlap in time and can coexist. Therefore, as shown in FIG. 14, the player ID of the tracking trajectory CL7 is determined to be "16."

[0095] After associating each of the tracking trajectories CL1 to CL7 with a player ID as shown in Fig. 14, the CPU 20 then connects tracking trajectories with the same player ID. In the case of Fig. 14, the tracking trajectories CL4 and CL6 are connected. Similarly, the tracking trajectories CL5 and CL7 are connected.

[0096] However, if the tracking trajectory to be connected is separated by more than a predetermined number of frames (for example, more than several tens of frames), it is likely that the player was off the field due to injury, rather than being a missed detection. In this case, the connection will not be performed.

[0097] In the player DB, player IDs and player names are recorded in association with each other, so that the tracking trajectory of each player can be obtained.

[0098] 1.4 Variations (Other) (1) In the above embodiment, the accuracy of tracking in frames where occlusion occurs is suspected. However, if tracking is performed across frames in which a person (target) cannot be recognized, the accuracy may be suspected and the tracking may be cut off.

[0099] (2) In the above embodiment, when the accuracy of tracking is in doubt due to occlusion or the like, tracking is temporarily disconnected and then reconnected.

[0100] If tracking is interrupted as a result of an interruption in filming itself, or if tracking is not possible due to blurred images caused by bad weather such as rain, connection may be established in the same manner as above. In this case, since tracking has already been interrupted, there is no need to disconnect.

[0101] (3) In the above embodiment, when it is determined that the players are the same, the connection is made by a straight trajectory. However, the trajectory of the disconnected part may be estimated based on the trajectories before and after the disconnection (interruption). Furthermore, the position of the object may be estimated by deep learning or the like, and the trajectory may be determined or corrected.

[0102] (4) In the above embodiment, the person as the target object is surrounded by a rectangle as shown in Fig. 5. However, it may also be surrounded by an oval or a shape that follows the outline of the person.

[0103] (5) In the above embodiment, soccer players are tracked. However, the present invention can also be used to track players in games such as rugby and American football.

[0104] (6) In the above embodiment, athletes are tracked as moving objects. However, people (spectators, visitors, etc.) moving in a given location may also be tracked as moving objects. Furthermore, machines other than people, such as automobiles, motorcycles, and motorboats, may also be tracked as moving objects.

[0105] In addition to athletes, the present invention can also be applied to non-rigid objects such as spectators and animals, and rigid objects such as vehicles.

[0106] Furthermore, people (spectators, visitors, etc.) moving in a predetermined location may be tracked as moving objects. Also, machines other than people, such as automobiles, motorcycles, and motorboats, may be tracked as moving objects.

[0107] (7) In the above embodiment, the tracking trajectory is used as the tracking connection information. However, any information that allows association of moving objects between frames can be used as the tracking connection information.

[0108] (8) In the above embodiment, the tracking device is configured as a standalone PC. However, the tracking device may be configured as a server device on the Internet. In this case, captured video can be uploaded from a terminal device, and the tracking results can be returned from the server device.

[0109] (9) In the above embodiment, YOLO is used to extract people. However, people may also be extracted using background subtraction or other methods.

[0110] (10) In the above embodiment, tracking is performed based on location prediction and image features using an MOT technique such as ByteTrack. However, tracking may be performed based only on location prediction or only on image features.

[0111] (11) In the above embodiment, the tracking trajectories that were not disconnected are also identified using the player DB. However, for the tracking trajectories that were not disconnected, players may be identified using another method, and only the tracking trajectories that were disconnected may be identified using the player DB and connected.

[0112] (12) In the above embodiment, players are identified based on image features and their tracking trajectories are connected. However, in addition to this, the decision on whether to connect players may be made based on factors such as whether the predicted movement direction of the player matches, whether the position of the player to be connected is within a reasonable distance depending on the number of frames, whether the difference in movement speed of the players to be connected is within a reasonable range, or whether the players to be connected are from the same team based on their uniforms.

[0113] (13) In the above embodiment, the learned ReID is used as the player DB. However, any player DB may be used as long as it outputs which of the pre-registered player images resembles the input player image.

[0114] (14) In the above embodiment, cases where the accuracy of tracking is in doubt are described as when a player cannot be extracted or when occlusion occurs. However, cases where the accuracy of tracking is in doubt may also be used, such as when the probability of player extraction falls below a predetermined value.

[0115] (15) The above-described embodiments and their modifications can be implemented in combination with each other, and can also be implemented in combination with other embodiments.

[0116] 2. Second embodiment 2.1 Functional configuration FIG. 15 shows the functional configuration of a player DB generation device according to the second embodiment. Player extraction means 20 receives images of players taken just before or just after the start of a match, extracts the players, and obtains player images. Just before or just after the start of a match, the positions of each player are roughly determined by their position, and the individual players are positioned apart. This makes it easy to distinguish between the players.

[0117] The player ID assigning means 22 assigns a player ID to the player image of each frame based on the position and / or tracking of each player in the captured video. The generating means 24 generates the person DB 26 based on the player images to which the player IDs have been assigned.

[0118] Therefore, an accurate person DB can be easily generated based on the video captured just before or just after the start of the match.

[0119] 2.2 Hardware configuration The hardware configuration is the same as that shown in Fig. 2 in the first embodiment, except that a player DB generation program is recorded in the SSD 28 instead of the tracking program 38.

[0120] 2.3 Person DB generation process 16 shows a flowchart of the player DB generation program. Immediately after (or immediately before) the start of a match, the CPU 20 reads video data 40 of the players from the SSD 28 and stores it in the memory 22 (step S21). If the volume of the video data 40 is large, it is stored in the memory 22 sequentially as the process progresses. The start of the match is a concept that includes the time when the match resumes after a break, etc.

[0121] Next, CPU 20 acquires the first frame of the video data as a target frame (step S23). An example image of the target frame is shown in Fig. 4. In this embodiment, video data captured of the entire soccer field by a camera installed on the soccer field is used.

[0122] The CPU 20 extracts a person from the target frame (step S24). This process can be performed using a trained model that has been trained by enclosing the person image in the image in a rectangular frame (given the coordinates of the top left and bottom right of the rectangular frame).

[0123] For example, an untrained estimation model using the YOLO algorithm may be replaced with a trained model trained on people, or a trained model trained to estimate the type and position of objects including people using the YOLO algorithm may be used.

[0124] The trained model outputs the coordinates of the top left and bottom right of the rectangle (boundary box) surrounding the person as the result of person extraction. This is shown as an image in Figure 5. Each player is extracted and surrounded by a boundary box.

[0125] Substitute players, spectators, coaches, etc. will also be extracted. In this embodiment, the boundary boxes of people outside the predefined playing area in the image are deleted based on the coordinates of the center point of the bottom edge of the boundary box (the position of the person's feet). This means that only people inside the playing area will be extracted as players.

[0126] Next, the CPU 20 assigns a player ID to each boundary box (i.e., player image) based on the entire frame image (step S25). As shown in Fig. 5, immediately after (or just before) the start of the game, the players are in predetermined positions separated from each other, so that the player IDs can be reliably assigned.

[0127] The above process is performed for all frames of the video captured immediately after the start of the match (for example, 3 minutes after the start of the match) or before the start of the match (for example, 30 seconds before the start of the match). In step S25, the player IDs assigned to the first frame are used to track the boundary boxes (players) in the following frames using ByteTrack or similar.

[0128] Therefore, a large number of player images with player IDs attached can be obtained. Furthermore, since the poses of players change over time, player images in various poses can be obtained.

[0129] Next, the CPU 20 provides the player image with the player ID to the ReID model for learning (step S27). That is, the player image is provided to the ReID model, and learning is performed using the player ID provided to the player image as correct answer data. The ReID model is a model that performs deep metric learning so that players with the same ID are assigned close vectors and players with different IDs are assigned distant vectors.

[0130] The ReID model learned as described above can be obtained as a player DB.

[0131] The player DB generated in this way can be used, for example, in the tracking process in the first embodiment. In this case, the player DB is generated from the early frames of a video of a game, and can be used for tracking in later frames of the video.

[0132] 2.4 Variations (Other) (1) In the above embodiment, a player DB for soccer players has been described. However, the present invention can also be used to generate a player DB for games in which the initial positions of players are fixed, such as rugby and American football.

[0133] Furthermore, a player DB can be similarly generated by assigning player IDs to images taken during the game.

[0134] Furthermore, the present invention can be applied not only to people but also to moving objects in general.

[0135] (2) In the above embodiment, in step S25, tracking is performed and a player ID is assigned to the player image. However, tracking may not be performed, and a player ID may be assigned based only on the position of the player image within the entire frame image. If the time of the target frame is short, the player's movement distance is short, so this method still allows for accurate assignment of a player ID.

[0136] (3) In the above embodiment, the learned ReID is generated as the player DB. However, any player DB may be generated as long as it outputs which pre-registered player images an input player image resembles.

[0137] (4) The above-described embodiments and their modifications can be implemented in combination with each other, and can also be implemented in combination with other embodiments.

Claims

1. A tracking device that tracks the movement of a moving object in a moving image captured of the moving object, a moving object extraction means for acquiring each frame image of a moving image in which a plurality of moving objects are captured, and extracting a moving object in each frame; a tracking means for tracking the movement of each extracted moving object between frames and generating tracking connection information; a disconnection means for disconnecting the tracking connection information at a frame where the accuracy of the tracking of the moving object between frames by the tracking means is determined to be doubtful, when the accuracy of the tracking of the moving object between frames by the tracking means is determined to be doubtful; a connecting means for comparing images of a series of frames of a moving object in each piece of tracking connection information disconnected by the disconnecting means with a plurality of moving objects registered in advance, thereby identifying the moving object in each piece of tracking connection information, and connecting the tracking connection information of the same moving object; A tracking device comprising:

2. A tracking program for realizing a tracking device that tracks the movement of a moving object in a moving image captured by the moving object, by a computer, the tracking program comprising: a moving object extraction means for acquiring each frame image of a moving image in which a plurality of moving objects are captured, and extracting a moving object in each frame; a tracking means for tracking the movement of each extracted moving object between frames and generating tracking connection information; a disconnection means for disconnecting the tracking connection information at a frame where the accuracy of the tracking of the moving object between frames by the tracking means is determined to be doubtful, when the accuracy of the tracking of the moving object between frames by the tracking means is determined to be doubtful; A tracking program that compares images of moving objects in a series of frames in each piece of tracking connection information that has been disconnected by the disconnection means with multiple moving objects registered in advance, identifies the moving object in each piece of connection information, and functions as a connection means that connects tracking connection information of the same moving object.

3. In the device of claim 1 or the program of claim 2, The connection means compares images of moving objects in a series of frames in each piece of tracking connection information for which tracking by the tracking means has been interrupted with multiple moving objects registered in advance, identifies the moving object in each piece of connection information, and connects the tracking connection information for the same moving object.

4. In the device of claim 1 or the program of claim 2, The device or program is characterized in that the moving object is a player on a stadium.

5. The device or program of claim 4, The connection means connects the tracking connection information while taking into consideration the position, speed, or direction of movement of the player, which is the moving object.

6. The device or program of claim 4, The connection means is a device or program characterized in that it identifies players in the tracking connection information by providing images of players, which are moving objects, in a series of frames of the tracking connection information to a trained person identification model that has been trained using distance deep learning to output similar features for the same player and dissimilar features for different players.

7. The device or program of claim 6, The trained person identification model is extracting each player from a plurality of frames of the captured image taken immediately before or after the start of the game using a moving object extraction means; Identifying the player name of each of the extracted players based on correspondence data that associates player names with positions; A device or program characterized in that it learns and obtains a person identification model based on an image in which the player's name is identified.

8. In the device of claim 1 or the program of claim 2, The connection means is a device or program characterized in that, even for tracking connection information that has been tracked across all frames, the connection means compares images of moving objects in a series of frames in the tracking connection information with multiple moving objects registered in advance to identify the moving objects in the tracking connection information.

9. A device for generating a moving object DB, a moving object extraction means for acquiring each frame image of a moving image of a moving object and extracting a moving object image from each frame; a moving object ID assigning means for assigning a moving object ID for distinguishing the moving object to the moving object image extracted in each frame based on the position of the moving object, the tracking of the moving object, or both; a generating means for generating a moving object DB based on moving object images of each moving object to which the moving object ID is assigned; A moving object DB generation device comprising:

10. A moving object DB generation program for realizing a moving object DB generation device by a computer, the program comprising: a moving object extraction means for acquiring each frame image of a moving image of a moving object and extracting a moving object image from each frame; a moving object ID assigning means for assigning a moving object ID for distinguishing the moving object to the moving object image extracted in each frame based on the position of the moving object, the tracking of the moving object, or both; a moving object DB generation program for functioning as a generation means for generating a moving object DB based on moving object images of each moving object to which the moving object ID is assigned;

11. In the device of claim 9 or the program of claim 10, The moving object DB is a device or program characterized in that it is a trained person identification model that uses moving object images and deep distance learning to output similar features for the same moving object and dissimilar features for different moving objects.

12. In the device of claim 9 or the program of claim 10, The moving object ID assigning means refers to data indicating the positions of players immediately before or immediately after the start of a game, and identifies the names of the players.

Images