A player re-identification method and system

Abstract

The present invention discloses a player re-identification method, comprising the following steps: positioning the players in the game video through a target detection algorithm, outputting the player's image and position; creating a corresponding track for the result detected in the first frame, initializing the position of the track, and setting the state to an uncertain state; inputting the player image into the player re-identification network to extract the player ReID features, and saving them in the corresponding track; transmitting the player image of the next frame to the player re-identification network to extract the player ReID features; judging whether there is a determined track, if so, proceeding to the next step, if not, returning to the previous step; calculating the cosine distance between the player ReID features of the current frame and the player ReID features of the historical frames saved in the track; ReID feature matching and IOU feature matching. The player re-identification method and system of the present invention have good discrimination and robustness, can realize accurate player re-identification in basketball game scenarios, and have higher efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular, mainly to a player re-identification method and system. Background Art

[0002] In basketball games, accurately identifying and tracking specific basketball players from different surveillance cameras is an important and challenging task. Tracking the player's position plays a very important role in the subsequent analysis of the team's offensive and defensive tactics, statistical game data, and analysis of the player's overall level.

[0003] The existing basketball game player tracking and positioning mainly has the following implementation methods:

[0004] 1. RFID tags: By carrying radio frequency identification (RFID) tags or sensors on players, players’ location information can be obtained in real time. Receivers are arranged around the court, and when a player with a tag enters the receiving range, the receiver can read the tag information and determine the location.

[0005] 2. GPS positioning: Some advanced sports venues use the Global Positioning System (GPS) to track the location of players. Players are equipped with devices with GPS receivers, and the location information is transmitted via wireless signals to a central system for processing and analysis.

[0006] The positioning and tracking based on these two methods have many problems, 1. The accuracy is not high enough, the error is large, and it is inconvenient to carry. When the network environment is unstable, there will be data loss and positioning delay. 2. The efficiency of manual video annotation is very low and it takes a lot of time.

[0007] In view of this, there is a need to improve existing player tracking methods. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a player re-identification method and system to solve the problems of low accuracy and large errors in the existing player positioning system, as well as low efficiency in the way of manual video annotation.

[0009] The present invention discloses a player re-identification method, comprising the following steps:

[0010] S100, positioning the players in the game video by using a target detection algorithm, and outputting the images and positions of the players;

[0011] S200, creating a corresponding track from the result detected in the first frame, initializing the position of the track, and setting the state to an uncertain state; inputting the player image into the player re-identification network to extract the player ReID features, and saving them in the corresponding track;

[0012] S300, passing the next frame of the player image to the player re-identification network to extract the player ReID features;

[0013] S400, determine whether there is a trajectory of a certain state, if yes, proceed to the next step, if not, return to step S300;

[0014] S500, calculating the cosine distance between the player ReID feature of the current frame and the player ReID feature of the historical frame saved in the trajectory;

[0015] S600, ReID feature matching and IOU feature matching;

[0016] S700, for the detection results in which both ReID feature matching and IOU matching are unsuccessful, a corresponding track is created, the position of the track is initialized according to the position of the player in the detection result, and the state is set to an uncertain state;

[0017] S800, traverse all uncertain trajectories, and if the trajectory successfully matches three consecutive frames, change the trajectory state to a certain state;

[0018] S900, repeat steps S300 to S800 until all frames of the video are traversed.

[0019] The present invention also discloses a player re-identification system, which uses the above-mentioned player re-identification method to re-identify players.

[0020] The beneficial effects of the present invention are:

[0021] By using a deep learning algorithm to fuse uniform color features, number plate digital features and global features, and by training an end-to-end neural network model, the uniform color features and number plate digital features of the players can be effectively combined to form a comprehensive re-identification feature representation. This feature representation has good discrimination and robustness, and can achieve accurate player re-identification in basketball game scenarios.

[0022] Compared with traditional portable electronic devices, the present invention is more convenient and quick. There is no need to carry any equipment. The players of both teams can just wear their respective uniforms. Compared with manual labeling, the accuracy rate is higher and the efficiency is also higher, thus solving the technical problem that the existing player tracking relies on manual labeling, and the efficiency of manual labeling is very low under the complex and intertwined situation of the game process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 is a flow chart of a player re-identification method of the present invention;

[0025] Figure 2 It is a structural schematic diagram of a number plate detector;

[0026] Figure 3 This is a schematic diagram of the structure of the branch network for extracting number plate features;

[0027] Figure 4 Flowchart for uniform color feature extraction and global feature extraction. DETAILED DESCRIPTION

[0028] In order to further understand the content, features and effects of the present invention, the following embodiments are given as examples and described in detail with reference to the accompanying drawings:

[0029] Reference Figure 1 , is a player re-identification method disclosed in the present invention, the method comprising the following steps:

[0030] S100, in a computer, positioning players on the court in a basketball game video by using a target detection algorithm YOLOv7, and outputting images and positions of the players, where the size of the player images is H×W;

[0031] Among them, the YOLO algorithm is a real-time target detection algorithm, which can achieve target detection by dividing each frame image in the video into grids and predicting bounding boxes and categories on each grid. The YOLOv7 algorithm used here is the 7th version of the YOLO algorithm. The YOLOv7 algorithm has been trained with a preset data set and can recognize a variety of objects in the image including people, and can output images containing specified objects.

[0032] S200, create a corresponding track for the result detected in the first frame, initialize the position of the track, and set the state to an uncertain state. Input the player image into the player re-identification network to extract the player ReID features and save them in the corresponding track.

[0033] Specifically, the player re-identification network includes a backbone network and three branch networks. The backbone network uses the deep learning neural network model ResNet50, that is, a residual network model with a depth of 50 layers. The backbone network is used to identify the players in each image and output the feature map of the players in each image.

[0034] The three branch networks are: number plate feature extraction branch network, jersey color feature extraction branch network and global feature extraction branch network.

[0035] After adjustment, the ResNet50 model used in the present invention has four residual blocks, namely residual block 1, residual block 2, residual block 3 and residual block 4, and the sizes of the feature maps output by each residual block are H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, and H / 16×W / 16, respectively; when the stride of the convolution is set to 1, an image of size H×W is input, and the final residual block outputs a feature map of H / 16×W / 16.

[0036] In the backbone network, the feature map output by each residual block after the player image is processed by ResNet50 is represented as F, F∈R h×w×c , where h, w, and c represent the height, width, and number of channels of the feature map extracted by the ResNet50 network, respectively.

[0037] Reference Figure 3 The number plate feature extraction branch network includes a number plate detector, which is used to find the center point, area width and height, and center point offset of the number plate according to the feature map F output by the backbone network; after finding the center point, area width and height, and center point offset of the number plate, for each player's image, a Gaussian distribution heat map is generated according to the actual position of its center point, area width and height, and center point offset; the obtained heat map is element-by-element multiplied with the feature map output by the backbone network to obtain the mask feature, and the mask feature can be obtained after average pooling.

[0038] Reference Figure 2 , the feature map F output by the backbone network is input into the number plate detector, and the following steps are performed:

[0039] The output feature map of residual block 4 in the backbone network is concatenated with the output of residual block 3, and then a 1×1 convolution is performed to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 16×W / 16.

[0040] The obtained feature map is then upsampled by interpolation, and then concatenated with the feature map output by residual block 2, followed by a 1×1 convolution to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 8×W / 8.

[0041] The obtained feature map is then upsampled by interpolation, and then concatenated with the feature map output by residual block 1, followed by a 1×1 convolution to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 4×W / 4.

[0042] The center point position, area width and height, and center point offset of the head part after the above feature fusion are predicted to obtain the center point position, area width and height, and center point offset of the number plate.

[0043] The uniform color feature extraction branch network and the global feature extraction branch network are used to extract uniform color features and global features respectively.

[0044] Therefore, more specifically, the method of inputting a player image into a player re-identification network to extract player ReID features includes the following steps:

[0045] S210, inputting the player image into the backbone network and extracting the feature map F;

[0046] S220, number plate feature extraction, including:

[0047] S221, input the feature map F extracted from the backbone network into the number plate detector to find the center point position, area width and height, and center point offset of the number plate, wherein the feature map F includes the feature map output by each residual block;

[0048] S222, generating a Gaussian distribution heat map according to the center point position, area width and height, and center point offset of the number plate;

[0049] S223, then multiply the obtained heat map by the feature map F output by the backbone network element by element to obtain a mask feature map;

[0050] S224, after average pooling the mask feature map, obtain the number plate feature.

[0051] S230, uniform color feature extraction and global feature extraction, refer to Figure 4 ,include:

[0052] S231, after average pooling, the feature map F extracted by the backbone network is passed to the uniform color feature extraction branch network and the global feature branch network respectively, and their respective fully connected layers are trained respectively;

[0053] S232, connecting the uniform color classification task to the end of the uniform color feature extraction branch network and connecting the ID classification task to the end of the global feature extraction branch network for training;

[0054] S233. After the training is completed, the classification task is removed, and the feature vector output by the last fully connected layer is obtained as the uniform color feature vector and the global feature vector.

[0055] S240, feature fusion and weighting, including:

[0056] S241, concatenating the features obtained from the three branches in the dimension of the number of channels to obtain a concatenated feature map Fconcat;

[0057] S242. Considering the different functions of different features and the fact that the number plate feature will be affected by occlusion, a feature weighted subnetwork is introduced here to weight the concatenated feature map Fconcat. The weighted feature is expressed as:

[0058] Fweight = tanh(Fconcat⊙W+B)

[0059] S243, the weighted features are connected to the fully connected layer, and finally a 512-dimensional player re-identification feature is output to obtain the player ReID feature of the player image of the frame.

[0060] S300: Pass the next frame of player image to the player re-identification network to extract player ReID features.

[0061] S400, determine whether there is a trajectory of a certain state, if yes, proceed to the next step, if not, return to step S300.

[0062] S500, calculating the cosine distance between the player ReID feature of the current frame and the player ReID feature of the historical frame saved in the trajectory, the formula is: in is the player ReID feature of the current frame, is the player ReID feature saved in the i-th track.

[0063] S600, ReID feature matching and IOU feature matching:

[0064] S610, ReID feature matching: Use the calculated cosine distance as the cost value to construct a cost matrix, set a matching threshold, and then use a matching algorithm to perform ReID feature matching on the player detection result of the current frame and the existing trajectory, and output the matched trajectory, the unmatched trajectory, and the unmatched detection result;

[0065] S620, the unmatched track output by the ReID feature matching is judged whether it has been unmatched for 80 consecutive frames, if so, the track is deleted, otherwise it is retained;

[0066] S630, setting a corresponding threshold, performing IOU matching on the unmatched trajectories and unmatched detection results retained in the ReID feature matching and the uncertain trajectories, and outputting the matched trajectories, unmatched trajectories and unmatched detection results;

[0067] S640, merge the tracks that are successfully matched in ReID feature matching and IOU matching, update the position of the track according to the detection result; and delete the tracks that are not successfully matched in IOU matching.

[0068] S700: For detection results in which both ReID feature matching and IOU matching are unsuccessful, a corresponding track is created, the position of the track is initialized according to the position of the player in the detection result, and the state is set to an uncertain state.

[0069] S800, traverse all uncertain trajectories, and if the trajectory successfully matches three consecutive frames, change the trajectory state to a certain state.

[0070] S900, repeat steps S300 to S800 until all frames of the video are traversed.

[0071] The present invention also discloses a player re-identification system, which uses the above-mentioned player re-identification method to re-identify players.

[0072] The beneficial effect of the present invention is that by adopting a deep learning algorithm to fuse the uniform color features, the number plate digital features and the global features, and by training an end-to-end neural network model, the uniform color features and the number plate digital features of the players can be effectively combined to form a comprehensive re-identification feature representation. This feature representation has good discrimination and robustness, and can achieve accurate player re-identification in basketball game scenarios.

[0073] Compared with traditional portable electronic devices, the present invention is more convenient and quick. There is no need to carry any equipment. The players of both teams can just wear their respective uniforms. Compared with manual labeling, the accuracy rate is higher and the efficiency is also higher, thus solving the technical problem that the existing player tracking relies on manual labeling, and the efficiency of manual labeling is very low under the complex and intertwined situation of the game process.

[0074] The above is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A player re-identification method, characterized in that: The steps include: S100, positioning the players in the game video by using a target detection algorithm, and outputting the images and positions of the players; S200, creating a corresponding track from the result detected in the first frame, initializing the position of the track, and setting the state to an uncertain state; inputting the player image into the player re-identification network to extract the player ReID features, and saving them in the corresponding track; S300, passing the next frame of the player image to the player re-identification network to extract the player ReID features; S400, determine whether there is a trajectory of a certain state, if yes, proceed to the next step, if not, return to step S300; S500, calculating the cosine distance between the player ReID feature of the current frame and the player ReID feature of the historical frame saved in the trajectory; S600, ReID feature matching and IOU feature matching; S700, for the detection results in which both ReID feature matching and IOU matching are unsuccessful, a corresponding track is created, the position of the track is initialized according to the position of the player in the detection result, and the state is set to an uncertain state; S800, traverse all uncertain trajectories, and if the trajectory successfully matches three consecutive frames, change the trajectory state to a certain state; S900, repeat steps S300 to S800 until all frames of the video are traversed.

2. The player re-identification method according to claim 1, characterized in that: The player re-identification network includes a backbone network, a number plate feature extraction branch network, a uniform color feature extraction branch network, and a global feature extraction branch network; The backbone network adopts the ResNet50 model, which has four residual blocks, and the width and height of the feature map are W and H respectively; The number plate feature extraction branch network includes a number plate detector, which is used to find the center point position, area width and height, and center point offset of the number plate according to the feature map output by the backbone network; The uniform color feature extraction branch network and the global feature extraction branch network are used to extract uniform color features and global features respectively.

3. The player re-identification method according to claim 2, characterized in that: The method of inputting a player image into a player re-identification network to extract player ReID features comprises the following steps: S210, inputting the player image into the backbone network to extract the feature map; S220, number plate feature extraction; S230, uniform color feature extraction and global feature extraction; S240, feature fusion and weighting.

4. The player re-identification method according to claim 3, characterized in that: The method for extracting number plate features comprises the following steps: S221, input the feature map extracted from the backbone network into the number plate detector, find the center point position, area width and height, and center point offset of the number plate, wherein the feature map F includes the feature map output by each residual block; S222, generating a Gaussian distribution heat map according to the center point position, area width and height, and center point offset of the number plate; S223, then multiply the obtained heat map by the feature map F output by the backbone network element by element to obtain a mask feature map; S224, after average pooling the mask feature map, obtain the number plate feature.

5. The player re-identification method according to claim 4, characterized in that: The method by which the number plate detector finds the center point position, area width and height, and center point offset of the number plate according to the feature map of the backbone network is: The output feature map of residual block 4 in the backbone network is concatenated with the output of residual block 3, and then a 1×1 convolution is performed to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 16×W / 16. The obtained feature map is then upsampled by interpolation, and then concatenated with the feature map output by residual block 2, followed by a 1×1 convolution to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 8×W / 8. The obtained feature map is then upsampled by interpolation, and then concatenated with the feature map output by residual block 1, followed by a 1×1 convolution to reduce the number of channels, followed by a 3×3 convolution to obtain a feature map of H / 4×W / 4. The center point position, area width and height, and center point offset of the head part after the above feature fusion are predicted to obtain the center point position, area width and height, and center point offset of the number plate.

6. The player re-identification method according to any one of claims 3 to 5, characterized in that: The feature fusion and weighting include: S241, concatenating the features obtained from the three branches in the dimension of the number of channels to obtain a concatenated feature map Fconcat; S242, weighting the concatenated feature map Fconcat, and the weighted feature is expressed as: Fweight=tanh(Fconcat⊙W+B); S243, the weighted features are connected to the fully connected layer, and finally a 512-dimensional player re-identification feature is output to obtain the player ReID feature of the player image of the frame.

7. The player re-identification method according to any one of claims 1 to 5, characterized in that: The calculation formula of the cosine distance is: where r j T is the player ReID feature of the current frame, is the player ReID feature saved in the i-th track.

8. The player re-identification method according to any one of claims 1 to 5, characterized in that: The steps of ReID feature matching and IOU feature matching include: S610, ReID feature matching: Use the calculated cosine distance as the cost value to construct a cost matrix, set a matching threshold, and then use a matching algorithm to perform ReID feature matching on the player detection result of the current frame and the existing trajectory, and output the matched trajectory, the unmatched trajectory, and the unmatched detection result; S620, the unmatched track output by the ReID feature matching is judged whether it has been unmatched for 80 consecutive frames, if so, the track is deleted, otherwise it is retained; S630, setting a corresponding threshold, performing IOU matching on the unmatched trajectories and unmatched detection results retained in the ReID feature matching and the uncertain trajectories, and outputting the matched trajectories, unmatched trajectories and unmatched detection results; S640, merge the tracks that are successfully matched in ReID feature matching and IOU matching, update the position of the track according to the detection result; and delete the tracks that are not successfully matched in IOU matching.

9. A player re-identification system, characterized in that: The player re-identification method according to any one of claims 1 to 8 is used to re-identify the player.

Images