A multi-modal vision-based event athlete identity recognition method and system

Abstract

The application belongs to the technical field of image recognition, and discloses a sports event athlete identity recognition method and system based on multi-modal vision; the method comprises the following steps: collecting a sports event video stream and performing adaptive preprocessing to obtain a multi-source synchronous video frame sequence; performing athlete target detection on the multi-source synchronous video frame sequence, performing image segmentation according to the detection result to obtain an athlete foreground region, performing multi-target tracking on the athlete foreground region to obtain a multi-target tracking result; performing dynamic biological feature extraction on the multi-target tracking result to obtain a multi-modal feature set; performing feature fusion on the multi-modal feature set based on an attention mechanism to obtain a fused feature, comparing the fused feature with an athlete feature database to obtain an athlete identity; the application realizes non-interference, high-precision and high-stability identity recognition of athletes, and meets the reliability and practicality requirements of identity verification in a sports event scene.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to a method and system for identifying athletes in sports events based on multimodal vision. Background Technology

[0002] In scenarios such as competition training monitoring, on-site refereeing assistance, and security management verification, the accuracy and non-intrusiveness of athlete identification are core requirements.

[0003] Chinese patent application CN106339738A discloses an intelligent sensing system and method for running: Several motion detection posts are set at predetermined locations; the GIS 3D geographic information of each motion detection post is pre-stored on a backend server; when a runner passes a motion detection post, an RFID reader senses and receives the identification information sent by the RFID electronic tag; the main control card records the identification information of the RFID electronic tag, the time point of the RFID reader's sensing of the RFID electronic tag, and the location information of the motion detection post at that moment; simultaneously, the recorded information is uploaded to the backend server via a wireless communication module; the backend server sends voice and light commands to the main control card, and the speaker and LED light on the motion detection post provide sound and light prompts to the user; the backend server processes the collected information and the pre-stored GIS 3D geographic information, and the resulting running data is displayed online in real time on the interface window, and the required running data is wirelessly transmitted to a mobile phone and a display screen.

[0004] While the above methods can meet the needs of most scenarios, research and practical application of these methods and existing technologies have revealed at least the following shortcomings:

[0005] The above methods rely on athletes wearing physical sensors, which can interfere with athletes' work. Furthermore, the sensors are usually tied to the user, and identification relies on the device ID rather than the user's own biometrics. Once the devices are interchanged or shared, the identification will fail.

[0006] In view of this, the present invention proposes a method and system for identifying athletes in sports events based on multimodal vision to solve the above problems. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a method for identifying athletes in sports events based on multimodal vision, comprising:

[0008] The process involves acquiring event video streams and performing adaptive preprocessing to obtain a multi-source synchronized video frame sequence. Athlete target detection is then performed on the multi-source synchronized video frame sequence. Based on the detection results, image segmentation is performed to obtain the athlete foreground region. Multi-target tracking is then performed on the athlete foreground region to obtain the multi-target tracking results. The methods for obtaining the athlete foreground region include:

[0009] The process involves acquiring detection results and obtaining cue feature vectors for the detection bounding boxes. It then samples *a* foreground points within the detection bounding boxes and *b* background points outside the bounding boxes, normalizes the point coordinates, and encodes them to obtain cue feature vectors for individual points. The cue feature vectors of the detection bounding boxes and individual points are concatenated and normalized using a normalization layer to obtain cue features. Video frames in the multi-source synchronized video sequence are divided into S×S non-overlapping blocks. Each non-overlapping block is transformed into a feature vector through a linear embedding layer to form an initial feature map. This initial feature map is then processed using attention weights that incorporate edge sharpness weights to obtain a global image feature map. Finally, the cue features are linearly transformed to match the dimension of the global image feature map to obtain upgraded cue features. These upgraded cue features are then further enhanced using a cross-attention layer. The system fuses the target feature map with the global image feature map to obtain a fused feature map. It generates masks of c resolutions through c parallel convolutional branches, and obtains a binary mask based on these masks. It iterates through each pixel in the binary mask, collecting the coordinates of all pixels with a value of 1 to form the athlete's foreground region. Dynamic biometric extraction is performed on the multi-target tracking results to obtain a multimodal feature set. The methods for obtaining the multimodal feature set include: acquiring the foreground region sequence corresponding to each ID; obtaining micro-expression features, gait dynamics features, and body posture features based on the foreground region sequence; concatenating the micro-expression features, gait dynamics features, and body posture features to obtain multimodal features; fusing the multimodal feature set based on an attention mechanism to obtain fused features; and comparing the fused features with the athlete feature database to obtain the athlete's identity.

[0010] Furthermore, methods for obtaining multi-source synchronized video frame sequences include:

[0011] Based on the camera's built-in GPS or network clock protocol, the frames of all video streams are time-synchronized, and a unified timestamp is assigned to each frame to obtain a synchronized video stream. Adaptive white balance correction is performed on the synchronized video stream to obtain white balance-corrected video image frames. Gamma correction is performed on the white balance-corrected video image frames to obtain gamma-corrected video image frames. Non-local mean denoising is performed on the gamma-corrected video image frames to obtain a multi-source synchronized video frame sequence.

[0012] Furthermore, methods for obtaining white-balance corrected video image frames include:

[0013] Calculate the average values ​​of the R, G, and B channel pixels in the image frames of the synchronized video stream, respectively. Combine the average values ​​of the G channel pixels and the average values ​​of the R channel pixels to correct the R channel pixels of the image frames of the synchronized video stream, and obtain the corrected R channel pixels. Combine the average values ​​of the G channel pixels and the average values ​​of the B channel pixels to correct the B channel pixels of the image frames of the synchronized video stream, and obtain the corrected B channel pixels. Obtain the white balance corrected video image frames.

[0014] Furthermore, methods for obtaining gamma-corrected video image frames include:

[0015] The grayscale values ​​of the white-balance-corrected video image frames are calculated based on the corrected R-channel pixels, corrected B-channel pixels, and original G-channel pixels. The maximum and minimum pixel values ​​are obtained by statistical analysis of the grayscale values ​​of the white-balance-corrected video image frames. The contrast ratio is calculated based on the maximum and minimum pixel values. The gamma coefficient is calculated based on the contrast ratio. The grayscale values ​​of the white-balance-corrected video image frames are then adjusted using the gamma coefficient to obtain the gamma-corrected video image frames.

[0016] Furthermore, methods for obtaining multi-source synchronized video frame sequences include:

[0017] For gamma-corrected video image frames, Centered on a Y×Y-based search window, with raised edges In the search window For a given U×U neighborhood block, calculate the squared L2 norm of the two neighboring blocks, and combine the squared L2 norms of the two neighboring blocks with a smoothing parameter to obtain the pixel. Compared to The weights of all pixels in the search window are summed to obtain the normalization factor. Based on the normalization factor, the gamma-corrected gray value of each pixel in the search window, and the corresponding weight, the gray value after non-local mean denoising is calculated. The gray value of each pixel in the gamma-corrected video image frame is calculated to obtain the multi-source synchronized video frame sequence.

[0018] Furthermore, methods for obtaining test results include:

[0019] The multi-source synchronized video sequence is used as input to the object detection model to obtain the detection results. The detection results include the top-left corner coordinates, width, height, confidence score, and class label of M bounding boxes; M is the number of athletes detected in the current frame; the class label is athlete, and the class label is 1 when an athlete is detected.

[0020] Furthermore, methods for obtaining cue feature vectors include:

[0021] The detection results are obtained by concatenating the coordinates of the top-left corner, width, and height of the detection box to obtain the detection bounding box. Then, the video frame resolution from the multi-source synchronized video sequence is converted to normalized coordinates to obtain a normalized detection box. The normalized detection box is then fused with linear transformation and positional encoding to obtain the cue feature vector of the detection bounding box. In the global image feature map, for non-overlapping blocks with edge gradients not lower than the edge threshold, the corresponding edge sharpness weight is the first weight; otherwise, it is the second weight. The method for obtaining the binary mask includes:

[0022] C parallel convolutional branches are used to generate masks of c different resolutions. Then, bilinear interpolation is used to upsample the low-resolution masks below the resolution threshold to the original frame size. The intersection-union ratio (IoU) between each low-resolution mask and the detection bounding box is calculated. Low-resolution masks with IoU not lower than the IoU threshold and foreground area not lower than the area threshold are selected. Finally, threshold binarization is used to obtain a binary mask.

[0023] Furthermore, methods for obtaining multi-target tracking results include:

[0024] Each foreground region is standardized to obtain the input image. This input image is then used as input to a pre-trained ResNet-50 network, and an appearance feature vector is obtained through a global average pooling layer. The athlete's horizontal and vertical velocities at time t are acquired and combined with the athlete's position at time t to obtain the athlete's motion state vector. A Kalman filter algorithm is used for state prediction to obtain an updated motion state vector. The appearance matching degree is calculated based on the appearance feature vector and cosine similarity. Motion similarity is calculated based on the updated motion state vector and the observation matrix. The association cost is calculated based on the appearance matching degree and motion similarity. For all current bounding boxes and all historical tracking trajectories, the association cost between each pair of bounding boxes and historical tracking trajectories is calculated. A cost matrix is ​​formed; the Hungarian algorithm is used to operate on the cost matrix to minimize the sum of all matching pairs; for successfully matched detection boxes k and trajectories j, the original ID of trajectory j is assigned to detection box k to continue historical tracking; for unmatched detection boxes k without historical trajectories to match, a new unique ID is assigned to them as the starting point of the new tracking trajectory; when the target is occluded, the trajectory is continuously predicted through Kalman filtering; when the target reappears, the appearance feature vector of the detection box corresponding to the target is extracted, and the cosine similarity is calculated with the last saved appearance feature vector of all predicted trajectories during the occlusion period to obtain the appearance matching degree; the predicted trajectory with the highest appearance matching degree and exceeding the matching threshold is found, and the ID of the detection box corresponding to the target is re-associated with the ID of the predicted trajectory to obtain the multi-target tracking result.

[0025] Furthermore, methods for obtaining multimodal feature sets include:

[0026] Foreground region sequences corresponding to each ID are obtained. These sequences are used as input to a facial neural network to obtain A facial key points. Subframe sequences containing these key facial regions are extracted. The subframe sequences are divided into temporal segments by time windows. These temporal segments are used as input to a feature extraction model to obtain micro-expression features. The foreground region sequences are used as input to a foreground extraction model to obtain B key joints. Using the B key joints at each time step as nodes, the Euclidean distance between any two different nodes is calculated. The maximum value of all joint distances at time t is calculated, and the Euclidean distance is normalized and weighted based on the maximum value to obtain the edge weights between nodes. A graph sequence is constructed and input into a graph convolutional network to obtain spatial features. These spatial features are then used as input to a bidirectional LSTM to obtain gait dynamics features. The pose parameters of each frame in the foreground region sequence, including trunk tilt angle, limb opening and closing angle, and center of gravity swing amplitude, are calculated to obtain a pose parameter sequence. This pose parameter sequence is used as input to a convolutional neural network to obtain body pose features. The micro-expression features, gait dynamics features, and body pose features are concatenated to obtain multimodal features.

[0027] Furthermore, methods for obtaining fusion features include:

[0028] The micro-expression features, gait dynamics features, and body posture features in the multimodal features are vertically concatenated and then input into the attention network. The attention network assigns attention weights to each feature, and the fused features are obtained by weighting based on the attention weights.

[0029] Furthermore, methods for obtaining athlete status include:

[0030] The cosine similarity between the fused feature and the fused feature of each standard multimodal feature in the database is calculated to obtain the identity similarity. The fused features of the standard multimodal features are collected and extracted by the athlete in a preset standard environment before the competition. The identity similarity is judged based on the preset similarity threshold. If there is a unique fused feature of the standard multimodal features that makes the identity similarity greater than the preset similarity threshold, then the ID corresponding to the fused feature is determined to be the ID corresponding to the fused feature of the standard multimodal features. Otherwise, it is marked as unidentified and awaits manual review.

[0031] A sports event athlete identification system based on multimodal vision, comprising the following steps:

[0032] Video acquisition module: Acquires event video streams and performs adaptive preprocessing to obtain multi-source synchronized video frame sequences.

[0033] The detection and tracking module performs athlete target detection on a multi-source synchronized video frame sequence, segments the image based on the detection results to obtain the athlete foreground region, and performs multi-target tracking on the athlete foreground region to obtain the multi-target tracking result. The methods for obtaining the athlete foreground region include:

[0034] The detection process involves acquiring detection results and obtaining cue feature vectors for the detection bounding boxes. A few foreground points within the detection bounding boxes and b background points outside the bounding boxes are sampled, and their coordinates are normalized and encoded to obtain cue feature vectors for individual points. The cue feature vectors of the detection bounding boxes and individual points are concatenated and normalized using a normalization layer to obtain cue features. Video frames in the multi-source synchronized video sequence are divided into S×S non-overlapping blocks. Each non-overlapping block is transformed into a feature vector through a linear embedding layer to form an initial feature map. This initial feature map is then processed using attention weights that incorporate edge sharpness weights to obtain a global image feature map. The cue features are then linearly transformed to be dimensionality-upgraded to match the global image feature map, resulting in upgraded cue features. These upgraded cue features and the global image feature map are then fused using a cross-attention layer to obtain a fused feature map. C parallel convolutional branches generate masks of c different resolutions, and a binary mask is obtained based on these masks. Each pixel in the binary mask is traversed, and the coordinates of all pixels with a value of 1 are collected to form the athlete foreground region.

[0035] Feature extraction module: Performs dynamic biometric feature extraction on multi-target tracking results to obtain a multimodal feature set; the method for obtaining the multimodal feature set includes: obtaining the foreground region sequence corresponding to each ID, obtaining micro-expression features, gait dynamics features and body posture features based on the foreground region sequence; and concatenating the micro-expression features, gait dynamics features and body posture features to obtain the multimodal features.

[0036] Identity recognition module: Based on the attention mechanism, feature fusion is performed on the multimodal feature set to obtain fused features. The fused features are then compared with the athlete feature database to obtain the athlete's identity.

[0037] The technical effects and advantages of the multimodal vision-based athlete identification method and system proposed in this invention are as follows:

[0038] This invention employs a purely visual technology approach throughout. First, it adaptively preprocesses multi-source sports video streams, using built-in GPS or network clock protocols in the cameras to synchronize the time of each video stream frame and assign a unified timestamp, ensuring consistent data sequence across multiple perspectives. Next, adaptive white balance correction eliminates light source color temperature interference, gamma correction optimizes image contrast, and non-local mean denoising reduces motion noise and graininess, improving video frame quality and laying the foundation for accurate recognition. Then, the preprocessed multi-source synchronized video frame sequence is input into an improved YOLOv5 model for athlete target detection, accurately outputting detection results including bounding boxes, confidence scores, and category labels. Based on these results, normalized detection boxes are constructed and cue features are encoded. Combined with global image features fused with edge sharpness weights, pixel-level athlete foreground regions are generated through cross-attention fusion and multiple low-resolution mask filtering, eliminating background interference and focusing on the target. Finally, after standardizing the foreground region, a pre-trained ResNet-50 network is used... The system extracts athlete appearance features using a network, predicts motion states using Kalman filtering, and calculates the association cost by combining appearance matching degree and motion similarity. Then, the Hungarian algorithm is used to achieve optimal matching between the detection box and historical trajectory. Each athlete is assigned and maintained with a unique ID. When the target is occluded, the trajectory is continuously predicted using Kalman filtering. Once the target reappears, the ID is re-associated based on appearance features to ensure tracking continuity. Next, based on the foreground region sequence with unique IDs, facial key points are extracted using a facial neural network, and sub-frame sequences are extracted to obtain micro-expression features. Key joints are obtained through a foreground extraction model, and a graph sequence is constructed to extract gait dynamics features. Body posture features are obtained by calculating parameters such as trunk tilt angle and limb opening and closing angle, forming a multimodal feature set. The multimodal features are then vertically concatenated and input into an attention network, where adaptive weights are assigned to each feature to achieve deep fusion. Finally, the cosine similarity between the fused features and the corresponding features in the pre-competition standard feature library is calculated, and the athlete's identity is determined based on a preset threshold. This invention completely eliminates the reliance on physical sensors, avoiding interference from sensor wear on athletes' movement. It also replaces device IDs with athletes' inherent multimodal biometrics as the basis for identification, fundamentally solving the identification failure problem caused by device interchangeability or sharing. At the same time, it is adapted to complex conditions such as athletes' rapid movement, limb overlap, and brief occlusion during competitions. Through multi-source synchronization to ensure data consistency, high-precision detection and segmentation to lock onto targets, stable tracking to maintain ID continuity, and multimodal feature fusion to improve individual differentiation, it ultimately achieves interference-free, highly accurate, and highly stable identification of athletes, meeting the reliability and practicality requirements of identity verification in competition scenarios. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a method for identifying athletes in sports events based on multimodal vision, according to the present invention.

[0040] Figure 2 This is a schematic diagram of the data flow in this invention.

[0041] Figure 3 This is a schematic diagram illustrating the data flow for obtaining multi-target tracking results according to the present invention.

[0042] Figure 4 This is a schematic diagram of the structure of a sports athlete identification system based on multimodal vision according to the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] Please see Figure 1 , Figure 2 As shown in this embodiment, a method for identifying athletes in sports events based on multimodal vision includes:

[0046] The event video stream is acquired and adaptive preprocessed to obtain a multi-source synchronized video frame sequence.

[0047] Methods for obtaining multi-source synchronized video frame sequences include:

[0048] Based on the camera's built-in GPS or network clock protocol, such as NTP, the frames of all video streams are synchronized in time, and a unified timestamp is assigned to each frame to obtain the synchronized video stream.

[0049] Adaptive white balance correction is performed on the synchronized video stream to obtain white balance corrected video image frames.

[0050] Methods for obtaining white balance corrected video image frames include:

[0051] Calculate the average pixel values ​​of the R, G, and B channels in the image frames of the synchronized video stream. Combine the average values ​​of the G and R channels to correct the R channel pixels of the synchronized video stream image frames, obtaining the corrected R channel pixels. Similarly, combine the average values ​​of the G and B channels to correct the B channel pixels of the synchronized video stream image frames, obtaining the corrected B channel pixels. This yields the white balance corrected video image frames. ,in, These are the corrected R channel pixels; These are the corrected G channel pixels; These are the corrected B-channel pixels; The R channel pixels of the image frame in the synchronized video stream; For the G-channel pixels of the image frame in the synchronized video stream; For the B-channel pixels of the image frame in the synchronized video stream; This represents the average pixel value of the G channel. This represents the average pixel value of the R channel. This represents the average pixel value of the B channel.

[0052] Perform gamma correction on the white balance corrected video image frames to obtain gamma-corrected video image frames.

[0053] Methods for obtaining gamma-corrected video image frames include:

[0054] The grayscale values ​​of the white-balance corrected video image frames are calculated based on the corrected R-channel pixels, corrected B-channel pixels, and original G-channel pixels; for example... ,in, This represents the grayscale value of the video image frame after white balance correction.

[0055] The maximum and minimum pixel values ​​are obtained by statistically analyzing the grayscale values ​​of the white-balance corrected video image frames. The contrast ratio is then calculated based on these values. The gamma coefficient is calculated from the contrast ratio, and the grayscale values ​​of the white-balance corrected video image frames are adjusted using this gamma coefficient to obtain the gamma-corrected video image frames. (The image frame's grayscale value is shown below.) ,in, The grayscale value of the video image frame after white balance correction; Gamma coefficient, Contrast ,in, The maximum pixel value. The minimum pixel value, The target contrast can be set through historical data statistics.

[0056] Nonlocal mean denoising is performed on the gamma-corrected video image frames to obtain a multi-source synchronized video frame sequence.

[0057] Methods for obtaining multi-source synchronized video frame sequences include:

[0058] For gamma-corrected video image frames, Centered on a Y×Y-based search window, with raised edges In the search window For a given U×U neighborhood block, calculate the squared L2 norm of the two neighboring blocks, and combine the squared L2 norms of the two neighboring blocks with a smoothing parameter to obtain the pixel. Compared to The weights of all pixels within the search window are summed to obtain a normalization factor. Based on the normalization factor, the gamma-corrected grayscale value of each pixel within the search window, and the corresponding weight, the grayscale value after non-local mean denoising is calculated. The grayscale value of each pixel in the gamma-corrected video image frame is then calculated to obtain a multi-source synchronized video frame sequence. For example, after non-local mean denoising... grayscale value ,in, for The corresponding search window; Normalization factor; For pixels Compared to The weight, , It is an exponential function. For The neighborhood block centered on, For The center's neighborhood block; It is an L2 norm; This is a smoothing parameter used to adjust the sensitivity of neighboring block similarity. It is typically set to 10 and can be manually adjusted.

[0059] The above method first achieves time synchronization of multi-source video streams and assigns a unified timestamp through the camera's built-in GPS or NTP protocol to ensure the temporal consistency of multi-view frames. Then, through adaptive white balance correction, gamma correction, and non-local mean denoising, the quality and consistency of video frames are gradually improved. The resulting multi-source synchronized video frame sequence can provide a clear and stable visual data foundation for the subsequent extraction of athletes' own dynamic biometric features such as micro-expressions, gait dynamics, and body posture. It eliminates the need for athletes to wear any physical sensors, fundamentally avoiding sensor interference with athletes' movements, and ensures that subsequent identity recognition relies on the athlete's unique and irreplaceable biometric features, rather than an ID bound to a device. This completely solves the technical problems of physical sensors interfering with athletes and identity recognition failure caused by device interchangeability or sharing.

[0060] Reference Figure 3 Athlete target detection is performed on a multi-source synchronized video frame sequence. Based on the detection results, image segmentation is performed to obtain the athlete foreground region. Multi-target tracking is then performed on the athlete foreground region to obtain the multi-target tracking results.

[0061] Methods for obtaining test results include:

[0062] The multi-source synchronized video sequence is used as input to the object detection model to obtain the detection results. The detection results include the top-left corner coordinates, width, height, confidence score, and class label of M bounding boxes; M is the number of athletes detected in the current frame; the class label is athlete, and the class label is 1 when an athlete is detected.

[0063] The target detection model adopts an improved YOLOv5 model, which improves the Backbone and Neck layers based on the Backbone, Neck and Head layers of the YOLOv5 model.

[0064] In the improved YOLOv5 model, a new feature extraction branch is added to the Backbone layer, such as in the 11th layer, and dimensionality reduction is performed through convolution. The 2×2 max pooling of downsampling in the 3rd, 5th and 7th layers of the Backbone layer is modified to a 3×3 convolution with a stride of 2.

[0065] For the feature map at each scale of the Neck layer, the spatial dimension is first compressed by global average pooling, and then channel weights are learned through two fully connected layers. Channel attention weights are applied to the feature map at each scale according to the channel weights to obtain channel-weighted feature maps. The channel-weighted feature maps are then captured by 3×3 convolution to capture spatial correlations, and then activated by Sigmoid to obtain spatial weights. Attention weights are applied to the channel-weighted feature maps according to the spatial weights to calculate the double attention-weighted feature maps. Small-scale features, such as 11, in adjacent scales are upsampled and concatenated with large-scale features, such as 9, and then the number of channels is unified by 1×1 convolution.

[0066] Training methods for object detection models include:

[0067] SL group detection training data is collected in advance. The detection training data includes multi-source synchronized video sequences and detection results. The detection results include the top-left corner coordinates, width, height, confidence score, and class label of M bounding boxes.

[0068] The detection training data is used as input to the object detection model, and the detection results are used as the output. The goal is to minimize the error between the output detection result and the actual detection result. A natural heuristic optimization algorithm is used to optimize the network parameters of the object detection model, obtaining the network parameters that minimize the error between the output control parameters and the actual control parameters. The object detection model constructed using these network parameters is then used as the trained object detection model. The training methods for subsequent models are similar to those for the object detection model, and will not be repeated in later steps.

[0069] The aforementioned method inputs multi-source synchronous video sequences into an improved YOLOv5 model. Optimization of the Backbone layer enhances the completeness of low-level feature extraction, and the dual attention mechanism of the Neck layer significantly improves robustness in complex situations such as rapid athlete movement and limb overlap during competitions. The final output includes accurate bounding boxes and confidence scores. This high-precision detection accurately locates the athlete target in each frame, providing a reliable spatial localization foundation for subsequent image segmentation to extract the athlete's foreground region, multi-target tracking to assign stable IDs, and dynamic biometric extraction. This allows the entire identity recognition process to be based entirely on the athlete's own visual biometrics, without relying on physical sensors that could interfere with movement. It also fundamentally avoids the device interchangeability or shared device failure issues caused by reliance on device IDs, achieving interference-free identity recognition based on the athlete's inherent biometrics.

[0070] Methods for obtaining the athlete's foreground area include:

[0071] To obtain the detection results, the top-left corner coordinates, width, and height of the detection box are concatenated to obtain the detection bounding box. Then, the video frame resolution from the multi-source synchronized video sequence is converted to normalized coordinates to obtain a normalized detection box. This normalized detection box is then fused with linear transformation and positional encoding to obtain the cue feature vector of the detection bounding box. For example, the first... Detection bounding boxes Perform normalization to obtain the first A normalized detection frame ,in, The coordinates of the top-left corner of the detection box; The width of the detection frame; The height of the detection frame; The resolution of video frames in a multi-source synchronized video sequence. For the width of video frames in a multi-source synchronized video sequence, For the high frame of a multi-source synchronized video sequence; the first cue feature vectors of bounding boxes ,in, For the position encoding module, sinusoidal position encoding is used to generate position features; It is a linear transformation layer that maps the bounding box coordinate features into vectors of the same dimension as the image features.

[0072] Sample and detect a foreground points within the bounding box and b background points outside the bounding box, normalize the point coordinates, and encode them to obtain the cue feature vector of a single point; as shown in the example. Points Hint feature vector ,in, For the first There are 10 sampling points, including foreground and background points; These are normalized sampling points, meaning the coordinates are the normalized coordinates; Point coordinate embedding layer; Embedded by type; For the first sampling points Type tags, Time can represent the foreground point. Time can represent a background point.

[0073] The cue feature vectors of the detected bounding boxes are concatenated with the cue feature vectors of individual points, and then standardized through a standardization layer to obtain cue features.

[0074] The video frames in the multi-source synchronized video sequence are divided into S×S non-overlapping blocks. Each non-overlapping block is transformed into a feature vector through a linear embedding layer to form an initial feature map. The initial feature map is then processed by an attention weight that incorporates edge sharpness weights to obtain a global image feature map. Specifically, for non-overlapping blocks with edge gradients not lower than an edge threshold, the corresponding edge sharpness weight is assigned as the first weight; otherwise, it is assigned as the second weight. (The attention weights that incorporate edge sharpness weights are then used.) ,in, , and For querying the matrix, key matrix, and value matrix; For activation functions; As edge sharpness weights, for edge gradients For non-overlapping blocks, the corresponding edge sharpness weight is 1.2, otherwise it is 0.8. The edge sharpness weight can enhance the features of sharp areas and suppress the interference of blurry areas. It is the transpose of the vector; This represents the vector dimension output by the linear embedding layer.

[0075] The cue features are updimensionalized to match the dimension of the global image feature map through a linear transformation to obtain updimensional cue features. The updimensional cue features and the global image feature map are then fused through a cross-attention layer to obtain a fused feature map.

[0076] C parallel convolutional branches are used to generate c types of masks with different resolutions. Then, bilinear interpolation is used to upsample the low-resolution masks below the resolution threshold to the original frame size. The intersection-union ratio (IoU) between each low-resolution mask and the detection bounding box is calculated. Low-resolution masks with an IoU not lower than the IoU threshold and a foreground area occupying a box area not lower than the area threshold are selected. Then, threshold binarization is used to obtain a binary mask. The binarization threshold is generally set to 0.5. The resolution threshold is obtained by statistical analysis of historical data.

[0077] Iterate through each pixel in the binary mask, collect the coordinates of all pixels with a value of 1, and form the athlete's foreground region.

[0078] The method described above first constructs a normalized detection box based on the detection results and encodes it as a cue feature. Then, it segments the video frames and embeds the features, fuses edge sharpness weights to optimize global image feature extraction, and then fuses the cue features and global image features through cross-attention to generate multiple low-resolution masks. Binary masks with high matching degree with the detection box and reasonable foreground proportions are then selected, ultimately extracting pixel-level athlete foreground regions. This purely vision-driven, precise segmentation method eliminates the need for athletes to wear any physical sensors that would interfere with their movement. It also provides a clean target region free from background interference for subsequent multi-target tracking, micro-expression / gait dynamics, and other dynamic biometric features extraction. This ensures that the identification process always relies on the athlete's inherent and irreplaceable biometric features, rather than an ID bound to a device. This fundamentally solves the technical problem of physical sensors interfering with movement and identification failure caused by device interchangeability or sharing, laying a crucial regional foundation for contactless, highly reliable athlete identification.

[0079] Methods for obtaining multi-target tracking results include:

[0080] The input image is obtained by standardizing each foreground region.

[0081] The input image is used as input to a pre-trained ResNet-50 network, and the appearance feature vector is obtained through a global average pooling layer.

[0082] The athlete's horizontal and vertical velocities at time t are obtained, and combined with the athlete's position at time t, the athlete's motion state vector is obtained. The state is then predicted based on the Kalman filter algorithm to obtain the updated motion state vector.

[0083] The appearance matching degree is calculated based on the appearance feature vector and cosine similarity; the motion similarity is calculated based on the updated motion state vector and observation matrix; and the association cost is calculated based on the appearance matching degree and motion similarity. (Example: Appearance matching degree...) Motion similarity Related costs ,in, For the first Taiwanese camera Time detection box The appearance feature vector; For the first Taiwanese camera Time and historical trajectory The appearance feature vector; Let be the magnitude of the vector; For detection box The center coordinates; The observation matrix; For the historical trajectory exist The Kalman filter predicts the state vector at time t. The variance of motion noise; These are the weighting coefficients.

[0084] For all current bounding boxes and all historical tracking trajectories, calculate the association cost between each pair of bounding boxes and historical tracking trajectories to form a cost matrix.

[0085] The Hungarian algorithm is used to operate on the cost matrix to minimize the sum of all matching pairs. For a successfully matched detection box k and trajectory j, the original ID of trajectory j is assigned to the detection box k to continue historical tracking. For an unmatched detection box k that has no historical trajectory to match, a new unique ID is assigned to it as the starting point of the new tracking trajectory. The original ID is also the ID initially assigned by the system when the corresponding detection box first appears and is not matched with a historical trajectory.

[0086] When the target is occluded, the trajectory is continuously predicted using Kalman filtering. When the target reappears, the appearance feature vector of the detection box corresponding to the target is extracted. The cosine similarity is calculated with the last saved appearance feature vector of all predicted trajectories during the occlusion period to obtain the appearance matching degree. The predicted trajectory with the highest appearance matching degree and exceeding the matching threshold is found. The ID of the detection box corresponding to the target is re-associated with the ID of the predicted trajectory to restore the tracking continuity and obtain the multi-target tracking result.

[0087] The method described above first standardizes the athlete's foreground region, then extracts appearance feature vectors using a ResNet-50 network. Combined with motion state vectors predicted by Kalman filtering, the association cost is calculated based on appearance matching and motion similarity. Finally, the Hungarian algorithm is used to achieve optimal matching between the detection box and historical trajectories, assigning and maintaining a unique ID for each athlete. Even if the target is occluded, the trajectory can be continuously predicted using Kalman filtering, and the ID can be re-associated based on appearance features upon recurrence, ensuring tracking continuity. This tracking method, which relies entirely on the athlete's appearance and biometric features, eliminates the need for athletes to wear any physical sensors that might interfere with their movement. Furthermore, it ensures that ID assignment and maintenance are entirely tied to the athlete's own characteristics rather than the device, fundamentally solving the technical problems of physical sensors interfering with movement and device interchangeability or sharing leading to identification failure. This provides a stable and reliable ID association foundation for subsequent biometric-based identification.

[0088] Dynamic biometrics are extracted from the multi-target tracking results to obtain a multimodal feature set.

[0089] Methods for obtaining multimodal feature sets include:

[0090] Obtain the foreground region sequence corresponding to each ID, use the foreground region sequence as input to the facial neural network, obtain A facial key points, and extract the subframe sequence containing the facial key regions; A is generally 68, where key points 1-17 are the jaw, key points 18-27 are the eyebrows, key points 28-36 are the nose, key points 37-48 are the eyes, and key points 49-68 are the mouth.

[0091] The subframe sequence is divided into temporal segments according to a time window, and the temporal segments are used as input to the feature extraction model to obtain micro-expression features.

[0092] By using the foreground region sequence as input to the foreground extraction model, B key joints are obtained.

[0093] Using the B key joints at each time step as nodes, calculate the Euclidean distance between any two different nodes. Calculate the maximum value of all joint distances at time t. Combine this maximum value with the normalized weighted Euclidean distances to obtain the edge weights between nodes, and construct the resulting graph sequence. ,in, For nodes and nodes In time Edge weights; For nodes and nodes Euclidean distance between them; For all joint distances at time 10:00 The maximum value is obtained by inputting the graph sequence into a graph convolutional network to obtain spatial features, and then using the spatial features as input to a bidirectional LSTM to obtain gait dynamics features.

[0094] The pose parameters of each frame in the foreground region sequence are calculated, including the torso tilt angle, limb opening and closing angle, and center of gravity swing amplitude, to obtain the pose parameter sequence.

[0095] The body pose features are obtained by using the sequence of pose parameters as input to a convolutional neural network.

[0096] Multimodal features are obtained by splicing together micro-expression features, gait dynamics features, and body posture features.

[0097] The above method first extracts 68 facial key points from the foreground region sequence corresponding to each athlete's unique ID using a facial neural network and extracts key facial subframe sequences. Micro-expression features are then obtained through temporal segmentation and feature extraction models. Next, key joint points are obtained through a foreground extraction model. A graph sequence is constructed using joint points as nodes and normalized Euclidean distance as edge weights. Gait dynamics features are extracted using a graph convolutional network and a bidirectional LSTM. Simultaneously, posture parameters such as torso tilt angle are calculated, and body posture features are extracted through a convolutional neural network. Finally, these are concatenated to form a multimodal feature set. The entire feature extraction process is based on video visual signals, requiring no physical sensors worn by athletes, completely avoiding sensor interference with the movement state. Furthermore, the extracted micro-expression, gait, and posture features are all inherent biological characteristics of the athletes, possessing uniqueness and stability, and are unrelated to device IDs. Even in cases of device interchangeability or sharing, accurate identity matching can be achieved based on these inherent biological characteristics, completely solving the recognition failure problem caused by reliance on device IDs and providing core feature support for subsequent biometric-based identity recognition.

[0098] The multimodal feature set is fused based on the attention mechanism to obtain fused features. The fused features are then compared with the athlete feature database to obtain the athlete's identity.

[0099] Methods for obtaining fusion features include:

[0100] The micro-expression features, gait dynamics features, and body posture features in the multimodal features are vertically concatenated and then input into the attention network. The attention network assigns attention weights to each feature, and the fused features are obtained by weighting based on the attention weights.

[0101] The aforementioned method vertically concatenates micro-expression features, gait dynamics features, and body posture features, then inputs them into an attention network, assigning adaptive weights to each feature. The resulting fused features comprehensively preserve the athlete's inherent, multi-dimensional biometric information while enhancing the representation of individual uniqueness through weight adjustment. The entire process requires no physical sensors, completely avoiding sensor interference with the athlete. Furthermore, the core of the fused features lies in the athlete's irreplaceable biometric characteristics, unrelated to device IDs. Even in cases of device interchangeability or sharing, accurate matching can be achieved based on these stable intrinsic features. This approach fundamentally solves the interference problems caused by reliance on physical sensors and the recognition failure problems caused by reliance on device IDs at the feature fusion level, providing a highly reliable comprehensive feature basis for final identity recognition.

[0102] Methods to obtain athlete status include:

[0103] The cosine similarity between the fused feature and the fused feature of each standard multimodal feature in the database is calculated to obtain the identity similarity. The fused features of the standard multimodal features are collected and extracted by the athlete in a preset standard environment before the competition. The identity similarity is judged based on the preset similarity threshold. If there is a unique fused feature of the standard multimodal features that makes the identity similarity greater than the preset similarity threshold, then the ID corresponding to the fused feature is determined to be the ID corresponding to the fused feature of the standard multimodal features. Otherwise, it is marked as unidentified and awaits manual review.

[0104] The method described above first calculates the cosine similarity between the athlete's fused features extracted from the competition and the standard fused features in the database. Then, it determines the athlete's identity based on a preset similarity threshold: only when the similarity between a unique standard feature and the competition's fused features exceeds the threshold is the ID corresponding to the standard feature assigned to the athlete; otherwise, it is marked for verification. The entire identity determination process is entirely based on the athlete's inherent and irreplaceable biometrics. It requires no physical sensors that might interfere with movement and completely eliminates dependence on device IDs. Even in cases of missing sensors, interchangeable or shared devices, as long as the athlete's biometrics match the standard features in the database, the identity can be accurately identified. This fundamentally solves the problems of physical sensors interfering with athletes and device-associated ID failures leading to identification errors, achieving reliable identity verification based on the athlete's own attributes.

[0105] Example 2

[0106] Please see Figure 4 As shown in this embodiment, a sports event athlete identification system based on multimodal vision includes:

[0107] Video acquisition module: Acquires event video streams and performs adaptive preprocessing to obtain multi-source synchronized video frame sequences.

[0108] Detection and tracking module: Performs athlete target detection on multi-source synchronous video frame sequences, performs image segmentation based on the detection results to obtain the athlete foreground region, performs multi-target tracking on the athlete foreground region, and obtains multi-target tracking results.

[0109] Feature extraction module: Performs dynamic biometric extraction on multi-target tracking results to obtain a multimodal feature set.

[0110] Identity recognition module: Based on the attention mechanism, feature fusion is performed on the multimodal feature set to obtain fused features. The fused features are then compared with the athlete feature database to obtain the athlete's identity.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0112] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying athletes in sports events based on multimodal vision, characterized in that, include: The event video stream was acquired and adaptive preprocessed to obtain a multi-source synchronized video frame sequence; Athlete target detection is performed on a multi-source synchronized video frame sequence. Based on the detection results, image segmentation is performed to obtain the athlete foreground region. Multi-target tracking is then performed on the athlete foreground region to obtain the multi-target tracking results. Methods for obtaining the athlete's foreground area include: Obtain the detection results and then obtain the cue feature vector of the detection bounding box based on the detection results; Sample and detect a foreground points within the bounding box and b background points outside the bounding box, normalize the point coordinates and encode them to obtain the cue feature vector of a single point; The cue feature vector of the detected bounding box is concatenated with the cue feature vector of a single point, and then normalized through a normalization layer to obtain the cue features; The video frames in the multi-source synchronized video sequence are divided into S×S non-overlapping blocks. Each non-overlapping block is transformed into a feature vector through a linear embedding layer to form an initial feature map. The initial feature map is then processed by the attention weights that fuse edge sharpness weights to obtain a global image feature map. The cue features are up-dimensionalized to match the dimension of the global image feature map through a linear transformation to obtain up-dimensional cue features. The up-dimensional cue features and the global image feature map are then fused through a cross-attention layer to obtain a fused feature map. By using c parallel convolutional branches, c types of masks with different resolutions are generated. A binary mask is obtained based on the mask. Each pixel in the binary mask is traversed, and the coordinates of all pixels with a value of 1 are collected to form the athlete's foreground region. Dynamic biometrics are extracted from the multi-target tracking results to obtain a multimodal feature set. The method for obtaining the multimodal feature set includes: obtaining the foreground region sequence corresponding to each ID, obtaining micro-expression features, gait dynamics features and body posture features based on the foreground region sequence, and concatenating the micro-expression features, gait dynamics features and body posture features to obtain the multimodal features. The multimodal feature set is fused based on the attention mechanism to obtain fused features. The fused features are then compared with the athlete feature database to obtain the athlete's identity.

2. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining multi-source synchronized video frame sequences include: Based on the camera's built-in GPS or network clock protocol, the frames of all video streams are synchronized in time, and a unified timestamp is assigned to each frame to obtain the synchronized video stream. Perform adaptive white balance correction on the synchronized video stream to obtain white balance corrected video image frames. Perform gamma correction on the white balance corrected video image frames to obtain gamma-corrected video image frames; Nonlocal mean denoising is performed on the gamma-corrected video image frames to obtain a multi-source synchronized video frame sequence.

3. The method for identifying athletes in sports events based on multimodal vision according to claim 2, characterized in that, Methods for obtaining white balance corrected video image frames include: Calculate the average values ​​of the R, G, and B channel pixels in the image frames of the synchronized video stream, respectively. Combine the average values ​​of the G channel pixels and the average values ​​of the R channel pixels to correct the R channel pixels of the image frames of the synchronized video stream, and obtain the corrected R channel pixels. Combine the average values ​​of the G channel pixels and the average values ​​of the B channel pixels to correct the B channel pixels of the image frames of the synchronized video stream, and obtain the corrected B channel pixels. Obtain the white balance corrected video image frames.

4. The method for identifying athletes in sports events based on multimodal vision according to claim 2, characterized in that, Methods for obtaining gamma-corrected video image frames include: The grayscale value of the white balance corrected video image frame is calculated based on the corrected R channel pixels, the corrected B channel pixels, and the original G channel pixels. The maximum and minimum pixel values ​​are obtained by statistically analyzing the grayscale values ​​of the white-balance corrected video image frames. The contrast ratio is then calculated based on the maximum and minimum pixel values. The gamma coefficient is calculated based on the contrast ratio. The grayscale values ​​of the white-balance corrected video image frames are then adjusted using the gamma coefficient to obtain the gamma-corrected video image frames.

5. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining multi-source synchronized video frame sequences include: For gamma-corrected video image frames, Centered on a Y×Y-based search window, with raised edges In the search window For a given U×U neighborhood block, calculate the squared L2 norm of the two neighboring blocks, and combine the squared L2 norms of the two neighboring blocks with a smoothing parameter to obtain the pixel. Compared to The weights of all pixels in the search window are summed to obtain the normalization factor. Based on the normalization factor, the gamma-corrected gray value of each pixel in the search window, and the corresponding weight, the gray value after non-local mean denoising is calculated. The gray value of each pixel in the gamma-corrected video image frame is calculated to obtain the multi-source synchronized video frame sequence.

6. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining test results include: The multi-source synchronized video sequence is used as input to the object detection model to obtain the detection results. The detection results include the top-left corner coordinates, width, height, confidence score, and class label of M bounding boxes; M is the number of athletes detected in the current frame; the class label is athlete, and the class label is 1 when an athlete is detected.

7. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining cue feature vectors include: The detection results are obtained by concatenating the coordinates of the top left corner, width, and height of the detection box to obtain the detection bounding box. Then, the video frame resolution in the multi-source synchronous video sequence is converted into normalized coordinates to obtain the normalized detection box. The normalized detection box is then fused with linear transformation and position encoding to obtain the cue feature vector of the detection bounding box. In the global image feature map, for non-overlapping blocks whose edge gradient is not lower than the edge threshold, the corresponding edge sharpness weight is the first weight; otherwise, it is the second weight. Methods for obtaining binary masks include: C parallel convolutional branches are used to generate masks of c different resolutions. Then, bilinear interpolation is used to upsample the low-resolution masks below the resolution threshold to the original frame size. The intersection-union ratio (IoU) between each low-resolution mask and the detection bounding box is calculated. Low-resolution masks with IoU not lower than the IoU threshold and foreground area not lower than the area threshold are selected. Finally, threshold binarization is used to obtain a binary mask.

8. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining multi-target tracking results include: The input image is obtained by standardizing each foreground region. The input image is used as input to a pre-trained ResNet-50 network, and the appearance feature vector is obtained through a global average pooling layer. The athlete's horizontal and vertical velocities at time t are obtained, and combined with the athlete's position at time t, the athlete's motion state vector is obtained by concatenation. The state is then predicted based on the Kalman filter algorithm to obtain the updated motion state vector. The appearance matching degree is calculated based on the appearance feature vector and cosine similarity; the motion similarity is calculated based on the updated motion state vector and observation matrix; and the association cost is calculated based on the appearance matching degree and motion similarity. For all current bounding boxes and all historical tracking trajectories, calculate the association cost between each pair of bounding boxes and historical tracking trajectories to form a cost matrix; use the Hungarian algorithm to operate on the cost matrix to minimize the sum of all matching pairs; for a successfully matched bounding box k and trajectory j, assign the original ID of trajectory j to bounding box k to continue historical tracking; for unmatched bounding boxes k without historical trajectories to match, assign a new unique ID as the starting point of the new tracking trajectory. When the target is occluded, the trajectory is continuously predicted using Kalman filtering. When the target reappears, the appearance feature vector of the detection box corresponding to the target is extracted. The cosine similarity is calculated with the last saved appearance feature vector of all predicted trajectories during the occlusion period to obtain the appearance matching degree. The predicted trajectory with the highest appearance matching degree and exceeding the matching threshold is found. The ID of the detection box corresponding to the target is re-associated with the ID of the predicted trajectory to obtain the multi-target tracking result.

9. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining multimodal feature sets include: Obtain the foreground region sequence corresponding to each ID, use the foreground region sequence as input to the facial neural network to obtain A facial key points, and extract the subframe sequence containing the facial key regions. The subframe sequence is divided into temporal segments according to a time window, and the temporal segments are used as input to the feature extraction model to obtain micro-expression features; Using the foreground region sequence as input to the foreground extraction model, B key joints are obtained; Using B key joints at each time point as nodes, calculate the Euclidean distance between any two different nodes, calculate the maximum value of all joint distances at time t, combine the maximum value to normalize and weight the Euclidean distance to obtain the edge weights between nodes, construct a graph sequence, input the graph sequence into a graph convolutional network to obtain spatial features, and then use the spatial features as the input of a bidirectional LSTM to obtain gait dynamics features. Calculate the pose parameters for each frame in the foreground region sequence, including the torso tilt angle, limb opening and closing angle, and center of gravity swing amplitude, to obtain the pose parameter sequence; The body pose features are obtained by using the pose parameter sequence as input to a convolutional neural network. Multimodal features are obtained by splicing together micro-expression features, gait dynamics features, and body posture features.

10. A method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods for obtaining fusion features include: The micro-expression features, gait dynamics features, and body posture features in the multimodal features are vertically concatenated and then input into the attention network. The attention network assigns attention weights to each feature, and the fused features are obtained by weighting based on the attention weights.

11. The method for identifying athletes in sports events based on multimodal vision according to claim 1, characterized in that, Methods to obtain athlete status include: The cosine similarity between the fused feature and the fused feature of each standard multimodal feature in the database is calculated to obtain the identity similarity. The fused features of the standard multimodal features are collected and extracted by the athlete in a preset standard environment before the competition. The identity similarity is judged based on the preset similarity threshold. If there is a unique fused feature of the standard multimodal features that makes the identity similarity greater than the preset similarity threshold, then the ID corresponding to the fused feature is determined to be the ID corresponding to the fused feature of the standard multimodal features. Otherwise, it is marked as unidentified and awaits manual review.

12. A sports event athlete identification system based on multimodal vision, implementing the sports event athlete identification method based on multimodal vision as described in any one of claims 1-11, characterized in that, include: Video acquisition module: Acquires event video streams and performs adaptive preprocessing to obtain multi-source synchronized video frame sequences; Detection and tracking module: Performs athlete target detection on multi-source synchronous video frame sequences, performs image segmentation based on the detection results to obtain the athlete foreground region, performs multi-target tracking on the athlete foreground region, and obtains multi-target tracking results; Methods for obtaining the athlete's foreground area include: Obtain the detection results and then obtain the cue feature vector of the detection bounding box based on the detection results; Sample and detect a foreground points within the bounding box and b background points outside the bounding box, normalize the point coordinates and encode them to obtain the cue feature vector of a single point; The cue feature vector of the detected bounding box is concatenated with the cue feature vector of a single point, and then normalized through a normalization layer to obtain the cue features; The video frames in the multi-source synchronized video sequence are divided into S×S non-overlapping blocks. Each non-overlapping block is transformed into a feature vector through a linear embedding layer to form an initial feature map. The initial feature map is then processed by the attention weights that fuse edge sharpness weights to obtain a global image feature map. The cue features are up-dimensionalized to match the dimension of the global image feature map through a linear transformation to obtain up-dimensional cue features. The up-dimensional cue features and the global image feature map are then fused through a cross-attention layer to obtain a fused feature map. By using c parallel convolutional branches, c types of masks with different resolutions are generated. A binary mask is obtained based on the mask. Each pixel in the binary mask is traversed, and the coordinates of all pixels with a value of 1 are collected to form the athlete's foreground region. Feature extraction module: Performs dynamic biometric feature extraction on multi-target tracking results to obtain a multimodal feature set; the method for obtaining the multimodal feature set includes: obtaining the foreground region sequence corresponding to each ID, obtaining micro-expression features, gait dynamics features and body posture features based on the foreground region sequence; concatenating the micro-expression features, gait dynamics features and body posture features to obtain the multimodal features; Identity recognition module: Based on the attention mechanism, feature fusion is performed on the multimodal feature set to obtain fused features. The fused features are then compared with the athlete feature database to obtain the athlete's identity.

Images