Hash-enhanced fairmot method for horse race participant tracking

Abstract

The application discloses a hash-enhanced FairMOT method for horse race player tracking, and comprises the following steps: using a horse race dataset, extracting feature vectors of players in different cameras by using an original FairMOT model; grouping according to camera positions, and training multiple groups of hash functions by using a BTSPLH hash algorithm; training the original FairMOT model as a basic framework; adding the hash functions to the end of the original FairMOT model, and inputting a test video into the model; using hash coding for player matching, and calculating hash coding weights at the latest time. The application reasonably combines the idea of the hash algorithm, improves the player matching accuracy of the original FairMOT model when dealing with frequent camera switching, mainly alleviates the camera switching problem, partially alleviates the problems of target occlusion and video blur when players sprint in the multi-target tracking task, and the additional computational cost is extremely low, and the application has good adaptability when facing scenes that need real-time multi-target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking and hash retrieval, in particular to a hash-enhanced FairMOT method for horse race participant tracking. BACKGROUND

[0002] Multi-target tracking is an important branch in the field of computer vision, and has a wide range of application scenarios and application values. At present, there are many public data sets in the direction of multi-target tracking, but in these public data sets, there are few camera switching. Compared with the current data without camera switching and the data with camera switching, the data with camera switching often brings greater challenges to existing algorithms, such as the horse race data set. When the camera switches, the pixel position, motion trend, space size occupied by the target and the specific region features of the target being tracked by the model will change greatly, thereby bringing considerable difficulty to the problem of matching accuracy of the target of the multi-target tracking task and the target of assigning the same ID to the same target at different times.

[0003] In the current research trend, the optimization of the multi-target tracking algorithm may often involve the update and parameter tuning of the neural network. This method can generally improve the effect of the algorithm, but generally accompanied by greater time and space overhead, which is not friendly to the video that needs to track multiple targets in real time.

[0004] In the field of image retrieval, the hash algorithm is a fast and light algorithm. The traditional non-deep hash algorithm itself does not involve neural network, and the generated hash code is binary code, so the traditional non-deep hash algorithm has the advantages of fast calculation and small space occupation. However, since the hash algorithm is an "approximate" algorithm, part of the data information will be lost after processing, and the precision will be worse than that of the neural network, so the idea needs to be used dialectically. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a hash-enhanced FairMOT method for horse race participant tracking, which can further process the original FairMOT model using the hash algorithm to obtain participant features that are not sensitive to camera switching, i.e. hash code, and can improve the ID matching accuracy of the participants after the camera switching, mainly alleviating the camera switching problem and partially alleviating the problem of the sprint blur phenomenon.

[0006] To achieve the above object, the technical scheme provided by the application is: the hash-enhanced FairMOT method for tracking horse racing players, which is an improved FairMOT model based on the idea of hash to realize higher tracking accuracy of horse racing players, and two parts are improved, the first part is to improve the feature extraction module of the FairMOT model: a layer of hash function mapping is added after the feature extraction module of the original FairMOT model, the hash function is trained from the player feature vectors captured by different camera positions, and the player feature extracted by the original FairMOT model at each time is mapped by the hash function to obtain the hash code of the player at each time, and the hash code has the property of being insensitive to the camera position; the second part is to improve the player matching mechanism of the FairMOT model: the original FairMOT model only uses the feature vector of the player at the last moment for matching, and is changed to use all the hash codes generated by the player in a period of time for matching, and the weighted weight of the hash code at each time is calculated according to the effective degree of the feature generated at the time, and finally the weighted hash code in a period of time is used for player matching;

[0007] The specific implementation of the hash-enhanced FairMOT method for tracking horse racing players includes the following steps:

[0008] 1) Use the horse racing data set composed of the video of different races in the horse racing field and the labeled bounding box of the player, and use the original FairMOT model to extract the feature vector of the player captured by different cameras;

[0009] 2) Group the obtained feature vectors according to the camera position, and use the BTSPLH hash algorithm to train multiple groups of hash functions;

[0010] 3) Use the entire horse racing data set to train the original FairMOT model as a basic framework;

[0011] 4) Add the trained hash function to the feature extraction module of the trained FairMOT model, obtain the FairMOT model with added hash function, and input the test video into the FairMOT model with added hash function, for each frame of the video, the FairMOT model with added hash function will detect and generate the hash code of each player in the current frame;

[0012] 5) Use the hash code group and the corresponding hash code weight of the player detected in the previous frame to match the hash code of the player detected in the current frame, and update the hash code weight and the hash code group, so as to improve the player matching accuracy of the FairMOT model when dealing with frequent camera switching.

[0013] Furthermore, in step 1), for the horse racing dataset, the video of each race consists of clips taken by multiple cameras, let... Refers to the first There are [number] cameras, and the total number of cameras is [number]. ,but For the first One camera The footage captured by this camera is called... Then the set of all camera clips is represented as :

[0014] ;

[0015] For the One camera The footage captured The original FairMOT model is randomly initialized and trained to obtain the model parameters. Independent training Once, a total of Model parameters of the original FairMOT models that are independent of each other. ;

[0016] For fragments and the model parameters obtained from training Perform the following operations: Set the parameters Substitute the fragments into the original FairMOT model. The input parameters are substituted into the original FairMOT model, and the output of the original FairMOT model yields the features of different contestants. For each segment After performing the above operations, the final feature set of all segments is obtained. for:

[0017] .

[0018] Furthermore, in step 2), for the feature set generated in step 1), The BTSPLH algorithm is used to learn the hash function for each segment's features. A set of hash functions was obtained through training. Finally obtained Group hash functions, where the input dimension of the hash function in each group is [not specified]. To maintain consistency with the output dimension of the Re-ID head in the original FairMOT model, the output dimension is set to... The set of all hash function groups is represented as :

[0019] .

[0020] Furthermore, in step 3), all fragment sets are used. Train the original FairMOT model to obtain the model parameters of the final original FairMOT model used for multi-object tracking. .

[0021] Furthermore, step 4) includes the following steps:

[0022] 4.1) Transfer the trained model parameters Substituting into the original FairMOT model, in chronological order, for each frame of the video, the original FairMOT model first derives the set of predicted contestants for that frame. and the corresponding feature vectors ,in Refers to the contestant detected in this frame. Refers to the contestant The corresponding feature vector, Refers to the total number of players predicted by the model for this frame;

[0023] 4.2) For the feature vector of each predicted player A set of hash function groups Map it to hash code :

[0024] ;

[0025] In the formula, sgn represents the sign function, which returns 1 when the value inside the function is greater than or equal to 0, and -1 when it is less than 0;

[0026] Calculate the hash codes of all players in the current frame. This yields the hash code set of all predicted contestants for the current frame. :

[0027] .

[0028] Furthermore, step 5) includes the following steps:

[0029] 5.1) Let the following be an example of something that has been detected before the current frame: The contestant, for the first detected frame before the current frame. One contestant, The set of its former bounding boxes and hash codes is called the trajectory. The set of all trajectories is ; called trajectory The included hash code group is The corresponding hash encoding weight is ; wherein is composed of hash codes, is a column vector of dimension , the meaning of is the number of frames in which the player has appeared before the current frame;

[0030] For a trajectory and the corresponding hash codes , hash code weights , the similarity between the hash code of the current frame player and the hash code of the trajectory is calculated as follows:

[0031] ;

[0032] In the formula, is the hash code of the player as a column vector, and the operator is the dot product operator, , , ;

[0033] Combine into a row column similarity matrix , and use the Hungarian algorithm to match the player to be determined and the existing trajectory ;

[0034] For a player , if it matches a trajectory , the corresponding hash code of the player is added to the hash code group corresponding to the trajectory as a new row of code, obtaining the updated hash code of the trajectory :

[0035] ;

[0036] If the player does not match any trajectory, the player is considered as a new trajectory , and is added to the trajectory set , obtaining a new trajectory set ;

[0037] 5.2) For the hash code of the current frame ​​​, the effective degree of the hash code at this time is calculated as the coding weight :

[0038] ;

[0039] In the formula, is the number of trajectories in the latest trajectory set ;

[0040] The hash code weight calculated from the current frame is added to the hash code weight of the trajectory matched by this frame ; After that, the hash code weight of the trajectory is obtained The updated hash code weight of the trajectory :

[0041] ;

[0042] Steps 4) -5) are repeated to perform continuous detection, so as to improve the player matching accuracy of the FairMOT model when dealing with frequent camera switching.

[0043] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0044] 1. The present application reasonably combines the idea of hash algorithm, improves the player matching accuracy of the original FairMOT model when dealing with frequent camera switching, mainly alleviates the camera switching problem, and partially alleviates the problem of target occlusion in multi-target tracking task and video blurring when the player sprints.

[0045] 2. While improving the algorithm effect, the optimization method based on the hash idea makes the cost after training extremely low, and has good adaptability when facing scenes that need real-time multi-target tracking. DETAILED DESCRIPTION

[0046] Figure 1 is a logical flowchart of the present application.

[0047] Figure 2 is a part of the horse racing data set example picture.

[0048] Figure 3 is a camera angle switching scene example picture, as time goes on, the image will be converted in angle, which is the difficulty that the present application focuses on.

[0049] Figure 4 is a hash function training process schematic diagram, in which represents the dimension of the feature vector, which is related to the specific network of extracting features, representative hash function, representative number of hash functions in each group of hash functions.

[0050] Figure 5 An illustration of mapping feature vectors to hash codes. DETAILED DESCRIPTION

[0051] The application will be described in further detail below with reference to embodiments and drawings, but the embodiments of the application are not limited thereto.

[0052] Referring to the drawings Figure 1 The embodiment discloses a hash-enhanced FairMOT method for horse race player tracking, which is improved based on the idea of hash to realize higher tracking accuracy of the horse race player. The method is improved in two parts. The first part is to improve the feature extraction module of the FairMOT model: a layer of hash function mapping is added after the feature extraction module of the original FairMOT model. The hash function is trained from the player feature vectors from different camera positions. The player feature extracted by the original FairMOT model at each time is mapped to the hash code of the player at each time through the hash function. The hash code has the property of being insensitive to the camera position. The second part is to improve the player matching mechanism of the FairMOT model: the original FairMOT model only uses the feature vector of the player at the last moment for matching, which is changed to use all the hash codes generated by the player in a period of time for matching. At the same time, the weighted weight of the hash code at each time is calculated according to the effective degree of the feature generated at the time, and finally the weighted hash code in a period of time is used for player matching.

[0053] Description of the data used:

[0054] The horse race dataset is used, which is composed of a total of 30 race videos and player labeled bounding boxes. Among them, 18 are used as the training set, and the remaining 12 are used as the test set. The video resolution is 1280x720, and the number of frames is about 2000 to 3500. The number of players in each race is 7 to 15. It is observed that the total number of races in these videos is composed of 9 camera shots. Each race video is composed of 3 to 5 camera shots from 9 cameras. Some data examples are shown in the accompanying Figure 2 The method mainly aims at the example of camera switching, as shown in the accompanying Figure 3 .

[0055] Step one: generate player feature vectors;

[0056] In the horse race dataset, the video of each race is composed of multiple camera shots, which means Refers to the first There are 9 cameras in total. For the first One camera The footage captured by this camera is called... Then the set of all camera clips can be represented as :

[0057] ;

[0058] 1.1) For the first One camera The footage captured The original FairMOT model is randomly initialized and trained to obtain the model parameters. The model parameters of nine independent FairMOT models were obtained through nine separate training runs. .

[0059] 1.2) For fragments and the parameters of the original FairMOT model obtained from training Perform the following operations: Set the parameters Substitute the fragments into the original FairMOT model. The input parameters are substituted into the original FairMOT model, and the output of the original FairMOT model yields the features of different contestants. For each segment After performing the above operations, the final feature set of all segments is obtained. for:

[0060] ;

[0061] Step 2: Train the hash function;

[0062] A diagram illustrating the training of the hash function is attached. Figure 4 As shown, for the feature set generated in the previous step The BTSPLH algorithm is used to learn the hash function for each segment's features. A set of hash functions was obtained through training. Ultimately, nine hash functions were obtained. The input dimension of each hash function was made consistent with the output dimension of the original FairMOT model's Re-ID header, which is 512 dimensions. The output dimension was set to... Specifically, we take 256, and the set of all hash function groups is represented as follows: :

[0063] ;

[0064] Step three: train the original FairMOT model as a base framework;

[0065] Using all the fragment sets , the original FairMOT model is trained to obtain the model parameters of the original FairMOT model for multi-target tracking ; Note that the difference between this step and step 1.1) is that step 1.1) only uses the fragment to train the model parameters of the original FairMOT model for this fragment , while step three uses all the fragments to train the model parameters of an original FairMOT model .

[0066] Step four: After adding the hash function to the feature generation module of the original FairMOT model, input the test video into the model to generate the feature vector of the player, and map it to the hash code through the hash function, as shown in the attached Figure 5 ;

[0067] 4.1) Substitute the trained model parameters into the original FairMOT model, and in chronological order, for each frame of the video, first obtain the predicted player set and the corresponding feature vector of this frame from the original FairMOT model, where denotes the player detected in this frame, denotes the corresponding feature vector of the player , that is, the output of the object feature prediction module in the figure, denotes the total number of players contained in this frame predicted by the model, note that is not fixed, but given by the original FairMOT model at any time during running;

[0068] 4.2) Map the feature vector of the player to the hash code using the set of hash function groups :

[0069] ;

[0070] In the formula, sgn represents the sign function, which is 1 when the value inside the function is greater than or equal to 0, and -1 when the value is less than 0;

[0071] Calculate the hash code of all players in the current frame , and obtain the hash code set of all predicted players in the current frame :

[0072] ;

[0073] Step five: Perform player matching, calculate the latest time hash code weight;

[0074] 5.1) Let the player has been detected before the current frame bit player, here is not a fixed value, but is updated in real time with the running of the model, for the first player detected before the current frame, the set of information such as the once bounding box, hash code of the player is called trajectory , the set of all trajectories is ; call the hash code group contained in the trajectory , the corresponding hash code weight is ; wherein is composed of row hash code, is a column vector of dimension, the meaning of is the number of frames that the trajectory corresponding player has appeared before the current frame, when the algorithm detects the first frame, the trajectory set is empty set at this time;

[0075] For trajectory and the corresponding hash code , hash code weight , the similarity calculation formula of hash code of the current frame player :

[0076] ;

[0077] In the formula, is the hash code of the player as a column vector, the operator is the dot product operator, , ;

[0078] Combine into row column similarity matrix , use the Hungarian algorithm to match the player to be determined and the existing trajectory ;

[0079] For player , if it matches a trajectory , the player corresponding to the hash code is added to the trajectory as a new row of codes , and the trajectory updated hash code is obtained:

[0080] ;

[0081] If the player does not match any trajectory or is an empty set, the player is considered as a new trajectory , and is added to the trajectory set to obtain a new trajectory set ;

[0082] 5.2) For the hash code of the current frame , the effectiveness of the hash code at this time is calculated as the code weight:

[0083] ;

[0084] In the formula, is the number of trajectories in the latest trajectory set ;

[0085] The hash code weight calculated from the current frame is added to the hash code weight of the trajectory matched by the frame to obtain the updated hash code weight of the trajectory :

[0086] ;

[0087] Steps 4) - 5) are repeated to continuously detect and improve the player matching accuracy of the FairMOT model when dealing with frequent camera switching.

[0088] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods, and are all included in the protection scope of the present application.​

Claims

1. A hash enhanced FairMOT method for tracking racehorse runners, characterized in that, The method is improved based on the idea of hash to realize higher tracking accuracy of the horse race player, which improves two parts, the first part is to improve the feature extraction module of the FairMOT model: a layer of hash function mapping is added after the original FairMOT model feature extraction module, the hash function is trained from the player feature vectors from different camera positions, the player feature extracted by the original FairMOT model at each time is mapped to get the hash code of the player at each time, and the hash code has the property of being insensitive to the camera position; The second part is to improve the player matching mechanism of the FairMOT model: the original FairMOT model only uses the player's feature vector at the last moment for matching, which is changed to use all the hash codes generated by the player in a period of time for matching, and the weighted weight of the hash code at each moment is calculated according to the effective degree of the feature generated at that time, and finally the weighted hash code in a period of time is used for player matching; The specific implementation of the hash-enhanced FairMOT method for horse race player tracking includes the following steps: 1) Using the horse race dataset composed of race videos and player labeled bounding boxes from different races in the horse race field, using the original FairMOT model to extract the player feature vectors from different cameras; 2) Group the obtained feature vectors according to the camera position, and use the BTSPLH hash algorithm to train multiple hash functions; 3) Use the entire horse race dataset to train the original FairMOT model as the basic framework; 4) Add the trained hash function to the feature extraction module of the trained FairMOT model, get the FairMOT model with added hash function, input the test video into the FairMOT model with added hash function, for each frame of the video, the FairMOT model with added hash function will detect and generate hash codes for each player in the current frame, including the following steps: 4.1) Put the trained model parameters into the original FairMOT model, in chronological order, for each frame of the video, first by the original FairMOT model to get the predicted player set of this frame and the corresponding feature vector , where refers to the player detected in this frame, refers to the player corresponding feature vector, refers to the total number of players contained in this frame predicted by the model; 4.2) For each predicted player's feature vector using a set of hash function groups mapping it into a hash code : ； Where sgn represents the sign function, when the value in the function is greater than or equal to 0, the result is 1, and less than 0, the result is -1; calculating hash codes of all players in the current frame , obtaining a set of hash codes of all predicted players in the current frame : ； 5) Match the hash code group and the corresponding hash code weight of the player detected in the previous frame with the hash code detected in the current frame, and update the hash code weight and the hash code group to improve the player matching accuracy of the FairMOT model when dealing with frequent camera switching.

2. The hash enhanced FairMOT method for racehorse player tracking according to claim 1, wherein, In step 1), for the horse race dataset, the video of each race consists of segments taken by multiple cameras, let denote the th camera, and the total number of cameras be , then , for the th camera , the segment taken by this camera is denoted as , and the set of segments of all cameras is denoted as : ； The first camera The second camera The third camera The fourth camera The fifth camera The sixth camera The seventh camera The eighth camera For the segment and the trained model parameters , the following operations are performed: substituting the parameters into the original FairMOT model, inputting the segment into the original FairMOT model after substituting the parameters, and the original FairMOT model outputs the features of different players ; the above operations are performed for each segment , and finally the feature set obtained by all segments is: 。 3. The hash-enhanced FairMOT method for racehorse player tracking according to claim 2, wherein, In step 2), for the feature set generated in step 1) , respectively use the BTSPLH algorithm to learn the hash function, for the feature of each segment , a set of hash functions are trained , finally get hash functions, let the input dimension of the hash function in each group consistent with the output dimension of the Re-ID head of the original FairMOT model, the output dimension is set to , the set of all hash function groups is represented as : 。 4. The hash enhanced FairMOT method for racehorse player tracking according to claim 3, wherein, In step 3), all fragment sets are used to train the original FairMOT model to obtain model parameters of the original FairMOT model finally used for multi-target tracking . .

5. The hash enhanced FairMOT method for racehorse player tracking according to claim 4, wherein, The step 5) includes the following steps: 5.1) Let the following be an example of something that has been detected before the current frame: The contestant, for the first detected frame before the current frame. One contestant, The set of its former bounding boxes and hash codes is called the trajectory. The set of all trajectories is ; called trajectory The included hash code group is The corresponding hash encoding weight is ;in Depend on Row hash encoding composition, for A column vector of dimension, The meaning is trajectory The number of frames the player appeared in before the current frame; For trajectory and the corresponding hash code , hash code weight , similarity with the hash code of the current frame player The calculation formula of the hash code of the current frame player is: ​ ； In the formula, is the hash code of the player As a column vector, the operator is the dot product operator, , ;​ combining into row column similarity matrix , using hungarian algorithm to match the to-be-judged player and the existing trajectory ​ For the player , if it matches the track , the corresponding hash code of the player is added to the track as a new row of code , and the updated hash code of the track is obtained: ； If the player is not matched to any trajectory, the player is considered a new trajectory , and is added to the set of trajectories , resulting in a new set of trajectories ; 5.2) Hash coding for the current frame , the effective degree of the hash coding at this time is calculated as the coding weight : ； wherein is the latest trajectory set number of trajectories within hash code weight computed for the current frame added as a new row to the track that this frame is matched to hash code weight track updated hash code weight : ； Repeat steps 4) -5) to continuously detect and improve the player matching accuracy of the FairMOT model when dealing with frequent camera switching. The step 5) includes the following steps:

Images