Tracking Device

Abstract

To provide a tracking device capable of accurately tracking a moving body such as a sports player.SOLUTION: Target feature point extraction means 14 extracts a feature point (a target feature point) of an image in a search frame of a target frame image FT in a moving image. Reference feature point acquisition means 12 acquires a recorded feature point (a reference feature point) regarding a reference frame FS in which a mobile frame has been already set and the feature point has been extracted. Correspondence forming means 16 determines similarity of surrounding pixels of each reference feature point with surrounding pixels of each target feature point to associate each reference feature point with each target feature point. Movement line calculation means 18 calculates a movement line connecting the corresponding reference feature point and the target feature point. The reference feature point is specified as a starting point and the target feature point as an end point. Movement calculation means 20 calculates a movement amount based on the movement line. Mobile frame setting means 26 moves the mobile frame of the reference frame based on the calculated movement amount and sets the mobile frame of a target frame. Thus, tracking can be performed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a tracking device for tracking a moving object. [Background technology]

[0002] Devices that capture moving objects using cameras and track them are used. For example, as in Patent Document 1, a method has been proposed in which a person is identified for each frame of a video, a person frame is generated, and the movement of the person frame is tracked to perform tracking. An algorithm such as YOLO is used to identify people.

[0003] Figure 19 shows an example of detecting a person in one frame of a video using YOLO. The rectangular frame in the figure is the detected person frame 2. By repeating this process for each frame, the movement of the person can be tracked.

[0004] The detected human frame is associated with the closest human frame in the previous frame, thereby tracking the person. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2015-070354 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned conventional technology has the following problem. In order to accurately recognize a moving object such as a person from an image, the moving object needs to be captured in a relatively large size. For this reason, as shown in FIG. 19, there are cases where a moving object 4 is not recognized, or a moving object frame 8 is recognized in a position different from that of a moving object 6.

[0007] If the moving object frame cannot be set accurately, tracking of the moving object will not work properly, so it was necessary to avoid this situation.

[0008] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a tracking device capable of solving the above problems and accurately tracking a moving object. [Means for solving the problem]

[0009] Independently applicable features of the present invention are listed below.

[0010] (1)(2) A tracking device according to the present invention is a tracking device that tracks the movement of a moving object in a video in which the moving object is captured, and includes: a target feature point extraction means that acquires a target frame image of the video in which the moving object is captured, and extracts feature points of the moving object in a search frame set based on a moving object frame of a reference frame prior to the target frame; a reference feature point acquisition means that acquires feature points within the moving object frame of the reference frame; a correspondence formation means that matches each reference feature point with each target feature point based on surrounding pixels of the reference feature point and surrounding pixels of the target feature point; a movement line calculation means that calculates, for each feature point, a movement line that has a feature point in the reference frame as a starting point and a corresponding feature point in the target frame as an end point; a movement calculation means that calculates the amount of movement of the moving body based on the movement line; and a moving object frame setting means that sets the moving object frame of the target frame based on the calculated amount of movement using the moving object frame of the reference frame as a reference.

[0011] Since the moving object frame is moved for each frame to track the moving object, the moving object can be tracked reliably.

[0012] (3) The tracking device according to the present invention is characterized in that the movement calculation means calculates the amount of movement of the moving body based on movement lines whose end points are within a predetermined range, excluding movement lines whose end points are outside the predetermined range, when the starting points of the movement lines are set to the same point.

[0013] Therefore, even if the shape of the moving object on the image changes due to a change in the posture of the moving object, tracking can be performed with high accuracy.

[0014] (4) The tracking device according to the present invention is characterized in that the movement calculation means calculates the amount of movement of the moving body based on a movement line whose end point falls within a predetermined radius centered on the center of gravity position of each end point when the starting point of each movement line is set to the same point.

[0015] Therefore, even if the shape of the moving object on the image changes due to a change in the posture of the moving object, tracking can be performed with high accuracy. Also, the processing is easy.

[0016] (5) The tracking device according to the present invention is characterized in that the moving object frame setting means sets the moving object frame of the target frame to a position to which the moving object frame of the reference frame is moved in accordance with the calculated amount of movement.

[0017] Therefore, tracking can be performed accurately.

[0018] (6) A tracking device according to the present invention is characterized in that the moving object frame setting means corrects the size of the moving object frame based on the position on the screen of the set moving object frame.

[0019] Therefore, the moving object frame can be set in accordance with the size of the captured image according to the distance from the camera.

[0020] (7) The tracking device of the present invention is characterized in that the moving object frame setting means generates a plurality of candidate moving object frames based on the calculated amount of movement, and sets the one of these candidate moving object frames that contains the greatest number of feature points as the moving object frame.

[0021] Therefore, it is possible to set a moving object frame more accurately and perform tracking.

[0022] In this invention, the "reference feature point acquisition means" corresponds to step S10 in the embodiment.

[0023] In the embodiment, step S9 corresponds to the "target feature point extraction means."

[0024] In this embodiment, step S105 corresponds to the "correspondence forming means."

[0025] In this embodiment, step S11 corresponds to the "movement line calculation means."

[0026] In this embodiment, step S14 corresponds to the "moving object frame setting means."

[0027] The term "program" is a concept that includes not only programs that can be executed directly by a CPU, but also programs in source format, compressed programs, encrypted programs, programs that work in conjunction with an operating system to perform functions, and the like. [Brief description of the drawings]

[0028] [Figure 1] 1 illustrates a functional configuration of a tracking device according to an embodiment. [Diagram 2] 1 shows the hardware configuration of a tracking device. [Diagram 3] 1 is a flow chart of the tracking program. [Figure 4] 1 is a flow chart of the tracking program. [Figure 5a] 1 is an example of a captured image. [Figure 5b] 13 is an example of a captured image from which the background has been removed. [Figure 6] FIG. 13 is a diagram showing a state in which person frames 60a and 60b have been set. [Figure 7] FIG. 13 is a diagram showing feature points P1, P2, etc. that are set on an image of a player. [Figure 8] FIG. 13 is a diagram illustrating a method for calculating a BRISK feature amount. [Figure 9] FIG. 13 is a diagram showing a search frame 61b set in a target frame. [Figure 10] FIG. 2 is a diagram showing feature points of a reference frame and feature points of a target frame. [Figure 11] FIG. 11 is a diagram for explaining a process of selecting a movement line. [Figure 12] FIG. 13 shows selected movement lines and an average movement line. [Figure 13] 13 is a diagram showing a player frame 62b set in the target frame. FIG. [Figure 14] FIG. 13 is a diagram showing a player frame in a reference frame and a player frame in a target frame. [Figure 15] 13 illustrates a functional configuration of a tracking device according to a second embodiment. [Figure 16] 1 is a flowchart of a tracking program. [Figure 17] 1 is a flowchart of a tracking program. [Figure 18] FIG. 13 is a diagram showing a process of selecting a person frame. [Figure 19] FIG. 1 is a diagram illustrating a conventional tracking process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] 1. First embodiment 1.1 Functional configuration Fig. 1 shows the functional configuration of a tracking device according to an embodiment of the present invention. Target feature point extraction means 14 extracts image feature points (target feature points) within a search frame of a target frame image FT in a moving image. This search frame is set based on a moving object frame that indicates the position of a moving object in a reference frame image prior to the target frame image FT (for example, a frame twice the size of the moving object frame).

[0030] The reference feature point acquisition means 12 acquires the recorded feature points (reference feature points) for the reference frame FS in which the moving body frame has already been set and the feature points have already been extracted. In other words, the means 12 acquires the feature points within the recorded moving body frame.

[0031] The correspondence forming means 16 judges the similarity between the surrounding pixels of each reference feature point and the surrounding pixels of each target feature point, and associates each reference feature point with each target feature point. In this way, the reference feature point of the moving body in the reference frame corresponds to the target feature point of the moving body in the target frame.

[0032] The movement line calculation means 18 calculates a movement line connecting the corresponding reference feature point and target feature point, with the reference feature point being the start point and the target feature point being the end point.

[0033] The movement line extraction means 22 extracts movement lines whose end points are within a predetermined range, excluding movement lines whose end points are outside the predetermined range when the start points of all standard movement lines are the same. The movement amount calculation means 24 calculates an average line of these movement lines, and calculates the movement amount of the moving body based on the average line. In this embodiment, the movement line extraction means 22 and the movement amount calculation means 24 constitute the movement calculation means 20.

[0034] The moving frame setting means 26 moves the moving body frame of the reference frame based on the calculated amount of movement, and sets the moving body frame of the target frame.

[0035] When the above process is completed, the target frame is set as a reference frame, and the frame following the target frame is set as a new target frame, and the above process is repeated. This makes it possible to track the moving body as the movement of the moving body frame.

[0036] In this embodiment, the moving object frame of the reference frame is moved to set the moving object frame of the target frame, so that the moving object can be captured reliably.

[0037] 1.2 Hardware Configuration The hardware configuration of the tracking device is shown in Figure 2. Connected to the CPU 30 are a memory 32, a display 34, a communication circuit 36, an SSD 38, a DVD-ROM 40, an input / output interface 42, and a mouse / keyboard 44. The communication circuit 36 ​​is a circuit for connecting to the Internet.

[0038] An operating system 50, a tracking program 52, and video data 54 are recorded in the SSD 38. The tracking program 52 performs its functions in cooperation with the operating system 50. These programs were recorded on a DVD-ROM 56 and installed in the SSD 38 via the DVD-ROM drive 40.

[0039] Video data 54 to be analyzed is also recorded in the SSD 38. This video data 54 is captured by a camera (not shown) and recorded on a portable recording medium 58, and then imported into the SSD 38 via the input / output interface 42. Note that the data may also be imported directly from the camera via the Internet or the like.

[0040] 1.3 Tracking process 3 and 4 show a flowchart of the tracking program 52. The CPU 30 acquires the start frame of the video data 54 from the SSD 38. Note that the video data 54 here is taken as an example of an entire soccer field captured by a fixed camera.

[0041] The CPU 30 removes the background from the video data to produce an image of only the players (step S1). Figure 5a is the captured image, and Figure 5b is the image with the background removed. The background can be removed from the video data using a background subtraction method. Note that the background image may be captured in advance without the players present, or the background image may be generated by analyzing all frames of the video.

[0042] Next, the CPU 30 acquires the start frame of the video data showing only the players. The start frame does not necessarily have to be the first frame, and may be an appropriate frame that allows the individual players to be easily distinguished. In this embodiment, the operator plays back the video data and selects the start frame while checking it on the display 34.

[0043] CPU 30 displays the selected start frame on display 34 (step S1). An example of the start frame is shown in Fig. 5b. The operator operates mouse 44 on the image of the start frame displayed on display 34 to set rectangular character frames for each player. Fig. 6 shows the state in which character frames 60a, 60b, etc. have been set for all players in the competition.

[0044] The operator operates the mouse 44 to change the size of the character frame, and sets it so that it generally surrounds the player. Note that in this embodiment, the size of the character frame can be changed, but the aspect ratio is fixed (for example, 2:1).

[0045] Next, the CPU 30 extracts image feature points for the image of the player in the person frame (step S3). Image feature points include corners and edges. In this embodiment, the feature points are extracted using the FAST method. In the FAST method, the brightness of pixels (about 16 pixels) on the circumference of a circle (radius, about several pixels) centered on a target pixel is obtained. If the brightness of all pixels (a predetermined number or more) on the circumference is brighter (darker) than the brightness of the target pixel, the target pixel is determined to be a feature point. In addition to brightness, color information may be used.

[0046] Such feature points are preferable for tracking players because they can be identified even if the player moves. As shown in Fig. 7, many feature points P1, P2, ... are extracted for each player.

[0047] Furthermore, the CPU 30 calculates a feature amount for each extracted feature point. The feature amount is used to characterize and distinguish each feature point, and is characterized by, for example, the image surrounding the feature point. Even if a player moves, the image surrounding the feature point does not change, so multiple feature points can be distinguished.

[0048] It is also possible to record an image of the surroundings for each feature point and distinguish the feature points based on the image of the surroundings. However, this would result in redundant processing, so in this embodiment, the BRISK feature is used.

[0049] The BRISK feature is calculated from multiple pixels RP surrounding the target pixel TP, as shown in Figure 8. 512 pairs (first and second points) of two predetermined pixels FIRST and SECOND are created. Starting from the first pair PAIR1, if the brightness of the first point FIRST is greater than that of the second point SECOND, then it is set to "1"; otherwise, it is set to "0". A "1" or "0" is calculated for each pair. In this way, a 512-bit feature (features of a predetermined number of bits can be used) can be obtained.

[0050] Such feature amounts represent the features of the image surrounding the feature point, and each feature point can be distinguished and characterized by the feature amount.

[0051] Note that the above feature values ​​change even for the same feature points due to changes in the player's posture. Therefore, in the BRISK feature, the direction of the overall density gradient of pixel RP in Figure 8 (for example, the density increases overall in the direction of 15 degrees to the upper right) is calculated, and the image is rotated so that the direction of this density gradient is at a specified angle (for example, 0 degrees), and the above feature values ​​are calculated with the orientation aligned.

[0052] In this manner, features (512-bit BRISK features) are calculated for each feature point of each player.

[0053] Next, the CPU 30 records the start frame as a reference frame together with the person frame in the SSD 38 (step S3). Furthermore, the CPU 30 obtains the next frame (containing only the player image) and sets this as the target frame (step S4).

[0054] An example of a target frame is shown in Fig. 9. The CPU 30 detects players in this reference frame and sets a person frame 60 for each player (steps S6 to S14). In this embodiment, on the premise that the range in which a player can move during one frame is limited, the same player is found from the target frame within a predetermined range of the player frame 60 of the reference frame and tracked.

[0055] For example, the process for the person frame 60b in the reference frame in Fig. 6 will be described below. First, the CPU 30 generates a search frame 61b by multiplying the frame by a predetermined factor (twice in this embodiment) without changing the center point of this person frame 60b (assuming that the reference frame and the target frame are at the same coordinate position). The generated search frame 61b is set as the target frame. The target frame in which the search frame 61b is set is shown in Fig. 9.

[0056] As shown in the target frame of Figure 9, the player's position has moved from the reference frame of Figure 6, but the search frame 61b is large enough that the player fits within this search frame 61b. Therefore, it is preferable that the size of the search frame is large enough that the player fits within the search frame in the target frame even if the player moves in either direction at maximum speed during one frame.

[0057] The CPU 30 extracts feature points within the search frame 61b and calculates feature amounts (step S9). The feature point extraction and feature amount calculation are performed using the FAST algorithm and the BRISK algorithm, as in step S3. As a result, the coordinates of the feature point and a 512-bit feature amount characterizing the feature point are calculated for each feature point.

[0058] 10 shows the feature points P1 to Pn of a player in the reference frame and the feature points T1 to Tn of a player in the target frame superimposed on the same coordinates. As is clear from the figure, the player moves to the upper right during one frame.

[0059] Next, the CPU 30 acquires feature points (reference feature points) P1 to Pn in the reference frame recorded in the SSD 38 (step S10). Furthermore, feature points (target feature points) T1 to Tn in the target frame corresponding to the feature points (reference feature points) P1 to Pn in the reference frame are found (step S105). For this correspondence, the BRISK feature amount of each feature point is used, and the reference feature point and the target feature point having the closest BRISK feature amount that have a dissimilarity level below a predetermined threshold (for example, using the Hamming distance) are associated with each other.

[0060] However, depending on the posture and position of the player (or if other players are partially in the frame), it is not always possible to find target feature points that match all of the reference feature points. For example, there may be cases where a feature point that was captured in the reference frame is hidden by a change in posture and is not captured in the target frame. In such cases, processing is performed assuming that there are no target feature points with a dissimilarity below a certain threshold.

[0061] Next, for the associated reference feature point and target feature point, the CPU 30 calculates a movement line M starting from the reference feature point and ending at the target feature point (step S11). In Fig. 10, a movement line M1 connecting the reference feature point P1 and the target feature point T1, and a movement line M2 connecting the reference feature point P2 and the target feature point T2 are shown.

[0062] The average value of the movement lines calculated in this way (other representative values ​​such as the median and the mode may be used) can be grasped as the movement amount of the player. Therefore, the person frame 60b in the reference frame can be moved by this movement amount to set the person frame 61b. By repeating this process, the movement of the player can be tracked sequentially.

[0063] However, if the amount of movement is calculated based on the average value of the movement lines calculated above, there is a possibility that an error will occur in the amount of movement due to feature points such as the fingertips that move differently from the movement of the player as a whole. Therefore, in this embodiment, the error in the amount of movement is reduced as follows.

[0064] The CPU 30 causes the start points of the movement lines M1 to Mn shown in Fig. 11A to coincide with one point as shown in Fig. 11B. Furthermore, as shown in Fig. 11C, the center of gravity G of the end points EP1 to EPn of the movement lines M1 to Mn is calculated, and movement lines having end points EPk, EPm, and EPq that are a predetermined distance or more away from the center of gravity G (peculiar movement lines) are deleted (step S12). This is because end points that are far from the center of gravity G are likely to reflect local movements such as postures rather than the movements of the player as a whole. Note that peculiar movement lines may be deleted using other methods such as histograms and statistical methods.

[0065] CPU 30 returns the remaining movement lines (selected movement lines) to their original positions, excluding the deleted movement lines. This state is shown in Fig. 12A. Next, as shown in Fig. 12B, the average of the selected movement lines (average movement distance in the X-axis direction and average movement distance in the Y-axis direction) is calculated, and the movement distances in the X-axis and Y-axis directions are calculated (step S13).

[0066] Next, the CPU 30 sets a person frame 62b in the target frame based on the calculated movement distance, as shown in Fig. 13. The setting of the person frame 62b is performed as follows.

[0067] As shown in Fig. 14, the person frame 60b in the reference frame is moved based on the calculated movement distance to set the person frame 62b in the target frame. Next, the coordinate position of the center point PG of the bottom side of the person frame 62b in the target frame in the entire screen is obtained. That is, the Y coordinate position of the center point PG in the screen in Fig. 13 is obtained.

[0068] Furthermore, the size of the person frame 62b (the center point PG is not moved and the aspect ratio is maintained) is changed according to the acquired Y coordinate position. This is because the person appears relatively small when viewed toward the top of the screen (farther from the camera). In this embodiment, the size of the person frame is determined in advance according to the Y coordinate and used.

[0069] The size of the person frame according to the Y coordinate can be calculated by recording a calculation formula in advance so that the size of the person frame according to the Y coordinate is proportionally allocated based on, for example, the largest person frame 60a and the Y coordinate of its base midpoint PG, and the smallest person frame 60n and the Y coordinate of its base midpoint PG in the screen of Figure 6.

[0070] The coordinates (e.g., the coordinates of the top left point and the coordinates of the bottom right point) of the person frame of the target frame finally determined as described above are recorded. Also, the coordinates and feature amounts of the feature points included in this person frame are recorded in association with the person frame.

[0071] In this way, the movement of the player can be tracked as a movement from the person frame 60b of the reference frame to the person frame 62b of the target frame. The above process is performed for all person frames, and the person frame of the reference frame and the person frame of the target frame for each player are set, and all players are tracked.

[0072] Next, the CPU 30 sets the target frame on which the above processing has been performed as a new reference frame (step S16). Furthermore, the CPU 30 obtains the next frame and sets it as a new target frame (step S6).

[0073] Thereafter, the same process as above is performed to set a person frame for a new target frame. In this manner, a person frame is set for the new target frame as well, and player tracking is performed.

[0074] The CPU 30 repeats this process for all frames and tracks the players using the character frames. For example, the CPU 30 can track each player by sequentially recording the movement of the specified coordinates (such as the center of gravity coordinates or the center coordinates of the base) of the character frame in each frame together with the player ID.

[0075] 1.4 Other (1) In the above embodiment, the players on the soccer field are tracked as moving objects. The movements of players in other sports, such as rugby, tennis, and basketball, may also be tracked as moving objects.

[0076] Moreover, the movements of coaches, referees, etc., rather than just players, may also be tracked. In other words, it is possible to track all moving people.

[0077] Furthermore, instead of people, moving objects such as cars, trains, and airplanes may be tracked.

[0078] (2) In the above embodiment, a moving line is selected, and the moving direction and the moving distance are calculated based on the selected moving line. However, the moving line may not be selected, and the moving direction and the moving distance may be calculated using all the moving lines.

[0079] (3) In the above embodiment, the tracking device is configured by one PC. However, this may be configured as a server device. In this case, video data is transmitted from the terminal device to the server device, and the tracking results (movement of the figure frame of each player) are returned to the terminal device.

[0080] Also, a background difference of the video data may be calculated in the terminal device, and video data such as that shown in FIG. 5b may be transmitted to the server device.

[0081] (4) In the above embodiment, the amount of movement is determined based on the X-axis movement distance and the Y-axis movement distance. However, the amount of movement may be determined based on the movement direction and the movement distance in that movement direction.

[0082] (5) In the above embodiment, the human frame of the start frame is set by the operator. However, this may be set automatically using an algorithm such as YOLO. In addition, the automatically set human frame may be confirmed and corrected by the operator.

[0083] (6) In the above embodiment, the FAST method is used to extract feature points, but other feature point extraction methods such as OpenCV may be used. Also, each player may be asked to wear a marker, and the marker in the captured image may be recognized as a feature point.

[0084] (7) In the above embodiment, the BRISK method is used to calculate features. However, other feature calculation methods such as A-KAZE, SIFT, SURF, and HOG may also be used.

[0085] (8) In the above embodiment, the background image is generated based on the images of all frames. In other words, images showing the background other than the players occupy most of the frames. Therefore, the background image can be obtained by using the data that appears most frequently in all frames for each pixel.

[0086] However, the background image may also change, for example, due to changes in the weather, changes in the shadows of buildings due to the position of the sun, etc. Therefore, the extraction of the background image may be performed at predetermined time intervals (e.g., every hour) rather than based on all frames.

[0087] In addition, shadows of buildings and people may be removed from a captured image using a deep learning model (such as the CycleGAN algorithm), and an image of only people may be obtained by background subtraction.

[0088] (9) In the above embodiment, the person frame is rectangular, but other shapes may be used. For example, the person frame may be an oval that contains a moving object, or a shape that takes the outline of a person.

[0089] (10) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiment.

[0090] 2. Second embodiment 2.1 Functional Configuration Fig. 15 shows the functional configuration of a tracking device according to the second embodiment. Target feature point extraction means 14 extracts image feature points (target feature points) within a search frame of a target frame image FT in a moving image. This search frame is set based on a moving body frame that indicates the position of a moving body in a reference frame image prior to the target frame image FT (for example, a frame twice the size of the moving body frame).

[0091] The reference feature point acquisition means 12 acquires the recorded feature points (reference feature points) for the reference frame FS in which the moving body frame has already been set and the feature points have already been extracted. In other words, the means 12 acquires the feature points within the recorded moving body frame.

[0092] The correspondence forming means 16 judges the similarity between the surrounding pixels of each reference feature point and the surrounding pixels of each target feature point, and associates each reference feature point with each target feature point. In this way, the reference feature point of the moving body in the reference frame corresponds to the target feature point of the moving body in the target frame.

[0093] The movement line calculation means 18 calculates a movement line connecting the corresponding reference feature point and target feature point, with the reference feature point being the start point and the target feature point being the end point.

[0094] The movement calculation means 20 calculates the movement amount (X-axis direction movement amount, Y-axis direction movement amount) of the moving body based on the movement line.

[0095] The candidate moving frame setting means 25 generates multiple candidate moving body frames based on the calculated movement amount. For example, if the calculated movement amount is an X-axis direction movement amount Xm and a Y-axis direction movement amount Ym, multiple candidate moving body frames are generated by shifting the position around this position as the center.

[0096] The moving body frame selection means 27 determines, from among these candidate moving body frames, the one that contains the most feature points of the target frame as the final moving body frame.

[0097] When the above process is completed, the target frame is set as a reference frame, and the frame following the target frame is set as a new target frame, and the above process is repeated. This makes it possible to track the moving body as the movement of the moving body frame.

[0098] In this embodiment, the moving object frame of the reference frame is moved and then corrected using feature points to set the moving object frame of the target frame, so that the moving object can be captured reliably.

[0099] 2.2 Hardware Configuration The hardware configuration is the same as in the first embodiment (see FIG. 2).

[0100] 2.3 Tracking process 16 and 17 show a flowchart of the tracking program 52. Steps S1 to S13 are the same as those in the first embodiment. In the first embodiment, the person frame 62b of the target frame is set based on the X-axis movement amount Xm and the Y-axis movement amount Ym calculated in step S13.

[0101] In this embodiment, based on the X-axis movement amount Xm and the Y-axis movement amount Ym calculated in step S13, candidate person frames 621-629 as shown in Fig. 18 are set (step S141). Note that the person frame indicated by the dashed line in the figure is person frame 60b in the reference frame. A candidate person frame 621 moved by Xm, Ym, a candidate person frame 622 moved by 2Xm, Ym, a candidate person frame 623 moved by Xm, 2Ym, a candidate person frame 624 moved by 0, Ym, and a candidate person frame 629 moved by . . . , 0, 0 (i.e., not moved) are set relative to this person frame 60b.

[0102] The CPU 30 selects, from among these candidate person frames 621 to 629, the one that includes the most feature points of the target frame as the person frame (step S142). This is because the one that includes the most feature points of the player is appropriate as the person frame showing the position of the player.

[0103] In the case of FIG. 18, the candidate person frame 623 contains the most feature points, so this will be selected as the person frame.

[0104] The CPU 30 acquires the coordinate position on the entire screen of the center point PG of the bottom side of the person frame set as described above. That is, it acquires the Y coordinate position of the center point PG on the screen of Fig. 13. Furthermore, it changes the size of the person frame (while maintaining the aspect ratio) according to the acquired Y coordinate position.

[0105] In this way, the movement of the player can be tracked as a movement from the person frame 60b of the reference frame to the person frame 62b of the target frame. The above process is performed for all person frames, and the person frame of the reference frame and the person frame of the target frame for each player are set, and all players are tracked.

[0106] Next, the CPU 30 sets the target frame on which the above processing has been performed as a new reference frame (step S16). Furthermore, the CPU 30 obtains the next frame and sets it as a new target frame (step S6).

[0107] Thereafter, the same process as above is performed to set a person frame for a new target frame. In this manner, a person frame is set for the new target frame as well, and player tracking is performed.

[0108] The CPU 30 repeats this process for all frames and tracks the players using the character frames. For example, the CPU 30 can track each player by sequentially recording the movement of the specified coordinates (such as the center of gravity coordinates or the center coordinates of the base) of the character frame in each frame together with the player ID.

[0109] 2.4 Other (1) In the above embodiment, the players on the soccer field are tracked as moving objects. The movements of players in other sports, such as rugby, tennis, and basketball, may also be tracked as moving objects.

[0110] Moreover, the movements of coaches, referees, etc., rather than just the players, may also be tracked. In other words, it is possible to track all moving people.

[0111] Furthermore, instead of people, moving objects such as cars, trains, and airplanes may be tracked.

[0112] (2) In the above embodiment, a moving line is selected, and the moving direction and the moving distance are calculated based on the selected moving line. However, the moving line may not be selected, and the moving direction and the moving distance may be calculated using all the moving lines.

[0113] (3) In the above embodiment, the tracking device is configured by one PC. However, this may be configured as a server device. In this case, video data is transmitted from the terminal device to the server device, and the tracking results (movement of the figure frame of each player) are returned to the terminal device.

[0114] Also, a background difference of the video data may be calculated in the terminal device, and video data such as that shown in FIG. 5b may be transmitted to the server device.

[0115] (4) In the above embodiment, the amount of movement is determined based on the X-axis movement distance and the Y-axis movement distance. However, the amount of movement may be determined based on the movement direction and the movement distance in that movement direction.

[0116] (5) In the above embodiment, the human frame of the start frame is set by the operator. However, this may be set automatically using an algorithm such as YOLO. In addition, the automatically set human frame may be confirmed and corrected by the operator.

[0117] (6) In the above embodiment, the FAST method is used to extract feature points, but other feature point extraction methods such as OpenCV may be used. Also, each player may be asked to wear a physical marker, and the marker in the captured image may be recognized as a feature point.

[0118] (7) In the above embodiment, the BRISK method is used to calculate features. However, other feature calculation methods such as A-KAZE, SIFT, SURF, and HOG may also be used.

[0119] (8) In the above embodiment, nine candidate frames are generated, but more or fewer candidate frames may be generated. Also, the candidate frames may be set more finely, for example, by shifting them by several pixels.

[0120] (9) In the above embodiment, the background image is generated based on the images of all frames. In other words, images showing the background other than the players occupy most of the frames. Therefore, the background image can be obtained by using the data that appears most frequently in all frames for each pixel.

[0121] However, the background image may also change, for example, due to changes in the weather, changes in the shadows of buildings due to the position of the sun, etc. Therefore, the extraction of the background image may be performed at predetermined time intervals (e.g., every hour) rather than based on all frames.

[0122] In addition, shadows of buildings and people may be removed from a captured image using a deep learning model (such as the CycleGAN algorithm), and an image of only people may be obtained by background subtraction.

[0123] (10) In the above embodiment, the human frame is rectangular, but other shapes may be used. For example, the human frame may be an oval that contains a moving object, or a shape that resembles the outline of a human.

[0124] (11) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiment.

Claims

1. 1. A tracking device that tracks a movement of a moving object in a moving image captured by the moving object, comprising: a target feature point extraction means for acquiring a target frame image of a moving image capturing the moving object, and extracting feature points of the moving object as target feature points in a search frame set based on a moving object frame of a reference frame image preceding the target frame image; a reference feature point acquisition means for acquiring feature points within the moving object frame of the reference frame image as reference feature points; a correspondence forming means for forming correspondences between each of the reference feature points and each of the target feature points based on surrounding pixels of the reference feature points and surrounding pixels of the target feature points; a movement line calculation means for calculating, for each feature point, a movement line having a feature point in the reference frame image as a start point and a corresponding feature point in the target frame image as an end point; A movement calculation means for calculating a movement amount of the moving object based on the movement line; a moving object frame setting means for setting a moving object frame of the target frame image based on the calculated amount of movement, using a moving object frame of the reference frame image as a reference; In a tracking device comprising: the movement calculation means calculates a movement amount of the moving object based on a movement line whose end point position is within a predetermined range, excluding a movement line whose end point position is outside a predetermined range when the start points of the movement lines are the same point; The tracking device is characterized in that the movement calculation means calculates the amount of movement of the moving object based on a movement line whose end point falls within a predetermined radius from the center of gravity position of each end point when the starting points of each of the movement lines are set to the same point.

2. A tracking program for implementing a tracking device that tracks a motion of a moving object in a moving image captured by the moving object, the tracking program comprising: a target feature point extraction means for acquiring a target frame image of a moving image capturing the moving object, and extracting feature points of the moving object as target feature points in a search frame set based on a moving object frame of a reference frame image preceding the target frame image; a reference feature point acquisition means for acquiring feature points within the moving object frame of the reference frame image as reference feature points; a correspondence forming means for forming correspondences between each of the reference feature points and each of the target feature points based on surrounding pixels of the reference feature points and surrounding pixels of the target feature points; a movement line calculation means for calculating, for each feature point, a movement line having a feature point in the reference frame image as a start point and a corresponding feature point in the target frame image as an end point; A movement calculation means for calculating a movement amount of the moving object based on the movement line; a tracking program for causing the program to function as a moving object frame setting means for setting a moving object frame of the target frame image based on the calculated amount of movement using a moving object frame of the reference frame image as a reference, the movement calculation means calculates a movement amount of the moving object based on a movement line whose end point position is within a predetermined range, excluding a movement line whose end point position is outside a predetermined range when the start points of the movement lines are the same point; A tracking program characterized in that the movement calculation means calculates the amount of movement of a moving object based on a movement line whose end point falls within a predetermined radius centered on the center of gravity position of each end point when the starting points of each movement line are set to the same point.

3. In the device of claim 1 or the program of claim 2, The device or program, characterized in that the moving object frame setting means sets a moving object frame of the target frame image to a position where the moving object frame of the reference frame image is moved according to the calculated movement amount.

4. In any one of the devices or programs according to claims 1 to 3, The device or program, wherein the moving object frame setting means modifies a size of the moving object frame based on a position on the screen of the set moving object frame.

5. In any one of the devices or programs according to claims 1 to 4, The moving object frame setting means generates a plurality of candidate moving object frames based on the calculated amount of movement, and sets the candidate moving object frame that contains the greatest number of feature points as the moving object frame.

6. 1. A tracking device that tracks a movement of a moving object in a moving image captured by the moving object, comprising: a target feature point extraction means for acquiring a target frame image of a moving image capturing the moving object, and extracting feature points of the moving object as target feature points in a search frame set based on a moving object frame of a reference frame image preceding the target frame image; a reference feature point acquisition means for acquiring feature points within the moving object frame of the reference frame image as reference feature points; a correspondence forming means for forming correspondences between each of the reference feature points and each of the target feature points based on surrounding pixels of the reference feature points and surrounding pixels of the target feature points; a movement line calculation means for calculating, for each feature point, a movement line having a feature point in the reference frame image as a start point and a corresponding feature point in the target frame image as an end point; A movement calculation means for calculating a movement amount of the moving object based on the movement line; a moving object frame setting means for setting a moving object frame of the target frame image based on the calculated amount of movement, using a moving object frame of the reference frame image as a reference; In a tracking device comprising: The moving object frame setting means generates a plurality of candidate moving object frames based on the calculated amount of movement, and sets the candidate moving object frame that contains the greatest number of feature points as the moving object frame.

7. A tracking program for implementing a tracking device that tracks a motion of a moving object in a moving image captured by the moving object, the tracking program comprising: a target feature point extraction means for acquiring a target frame image of a moving image capturing the moving object, and extracting feature points of the moving object as target feature points in a search frame set based on a moving object frame of a reference frame image preceding the target frame image; a reference feature point acquisition means for acquiring feature points within the moving object frame of the reference frame image as reference feature points; a correspondence forming means for forming correspondences between each of the reference feature points and each of the target feature points based on surrounding pixels of the reference feature points and surrounding pixels of the target feature points; a movement line calculation means for calculating, for each feature point, a movement line having a feature point in the reference frame image as a start point and a corresponding feature point in the target frame image as an end point; A movement calculation means for calculating a movement amount of the moving object based on the movement line; a tracking program for causing the program to function as a moving object frame setting means for setting a moving object frame of the target frame image based on the calculated amount of movement using a moving object frame of the reference frame image as a reference, A tracking program characterized in that the moving object frame setting means generates a plurality of candidate moving object frames based on the calculated amount of movement, and sets the candidate moving object frame that contains the greatest number of feature points as the moving object frame.

Images