A method and system for multi-modal feature fusion and clustering for ski videos

Abstract

This invention relates to the fields of computer vision and video analysis technology, and discloses a multimodal feature fusion and clustering method and system for ski videos. It aims to solve the problems of low accuracy and poor efficiency in existing ski resort video retrieval schemes under complex scenarios with strong occlusion, high clothing similarity, and motion blur. This invention first extracts the skier's appearance features, motion features, and equipment semantic features from captured ski video clips. After projecting these three types of features onto a unified dimension, the fusion weights of each feature are adaptively adjusted based on the skier's movement speed to obtain a fused feature vector. Then, a time decay weight is introduced to adjust feature similarity, and a density clustering algorithm is used to aggregate video clips of the same skier. Simultaneously, model parameters are optimized through user feedback. This invention significantly improves the accuracy and efficiency of ski video retrieval and is suitable for ski resort video retrieval and operation service scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video analysis technology, specifically to a multimodal feature fusion method and system for accurately clustering and retrieving video clips of skiers in a ski resort setting. Background Technology

[0002] At ski resorts, skiers cannot use their phones to record videos of themselves skiing. Therefore, resort operators typically deploy cameras around the slopes to automatically capture skiers' videos, which skiers can retrieve later. Currently, there are three main solutions for skier video retrieval, but all have significant shortcomings and cannot meet the needs for high-precision and high-efficiency retrieval.

[0003] The existing technology has the following drawbacks:

[0004] (1) Manual selection method: The operator displays all the videos taken that day on the screen, allowing tourists to manually select their own videos. This method has high labor costs, extremely low search efficiency, and is prone to selection errors, and is currently being phased out.

[0005] (2) General ReID Algorithm Retrieval Scheme: A general pedestrian re-identification algorithm is adopted to extract the visual features of skiers for comparison. For example, the existing technology (CN202310865814.0) uses pose estimation for appearance modeling and matching; another scheme (CN111563404B) uses DTP convolution and TSA model to capture spatiotemporal cues. However, these schemes are mainly designed for ordinary pedestrian scenes and do not consider the special environment of ski resorts (such as faces being covered by ski goggles, helmets, high clothing similarity, distance from the camera, and varied postures), resulting in extremely low recognition accuracy based on single or simple weighted appearance features in skiing scenarios.

[0006] RFID signal tag retrieval scheme: This scheme triggers a camera to capture images using an RFID reader, binding the video to the RFID tag ID worn by the skier. The skier then enters the ID to retrieve the video. For example, existing RFID-triggered video capture schemes (CN103593975B, CN106815618A) trigger the camera or identify the user's location using RFID signals. However, these schemes have significant drawbacks: a single RFID signal cannot distinguish between multiple skiers entering the sensing area simultaneously, and RFID triggering is random, making it impossible to ensure capture at the optimal position or that the skier is within the camera's line of sight. This results in retrieved videos containing numerous other skiers' videos and invalid content, making them impractical.

[0007] In summary, existing technologies lack consideration for the specific characteristics of skiing scenarios, fail to utilize ski equipment as a key discriminative feature, lack adaptive processing capabilities for motion fuzziness, and fail to effectively integrate physical time information into feature matching and clustering processes. As a result, high-precision and high-efficiency video retrieval cannot be achieved in the complex environment of ski resorts, and a targeted solution is urgently needed. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of the prior art and provide a multimodal feature fusion and clustering method and system for ski videos. It focuses on solving the special technical problem of "how to significantly improve the accuracy and efficiency of tourist video retrieval through accurate feature matching and clustering in complex environments such as ski resorts with long distances, strong obstructions, strong reflections, and similar clothing". The invention achieves the technical effects of significantly improving the accuracy and speed of video retrieval, reducing retrieval errors, improving user experience, and enhancing the operational efficiency of scenic spots. A Multimodal Feature Fusion and Clustering Method for Skiing Videos

[0009] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for multimodal feature fusion and clustering of skiing videos, comprising:

[0010] S01: Multimodal feature extraction, extracting the skier's appearance features, motion features, and equipment semantic features from video clips captured by the camera;

[0011] S02: Adaptive multimodal feature fusion. After projecting the three features onto a unified dimension, the fusion weights of each feature are adaptively adjusted based on the skier's movement speed to obtain the final fused feature vector.

[0012] S03: Time decay weighted clustering introduces time decay weights to adjust the similarity of fusion features of different video clips, and uses density clustering algorithm to aggregate all video clips of the same skier.

[0013] S04: Interactive training feedback, receiving user feedback on clustering results, optimizing feature fusion weights and model parameters based on feedback information, and improving subsequent retrieval accuracy.

[0014] In a preferred embodiment of this application, step S01, multimodal feature extraction, specifically includes:

[0015] 1. Input source: A collection of videos captured by all cameras. Each video carries original timestamp information, including the start time of video capture, total video length, time prediction parameters (physical viewing angle width w of the row where the center point of the ROI area in the camera frame is located, physical distance D between the current camera and the next camera), and video ID number (videos bound to RFID tags must include the tag ID number).

[0016] 2. Appearance feature extraction: The YOLOv5+ResNet100 network is used to detect and extract human targets in video frames, resulting in a 1024-dimensional appearance feature vector, which is then optimized using the IBN structure and the ArcFace loss function.

[0017] 3. Motion feature extraction: The Temporal Segment Network (TSN) is used to perform temporal modeling on continuous video frames and extract 1024-dimensional motion feature vectors to characterize the skier's gliding posture and trajectory.

[0018] 4. Equipment Feature Extraction: YOLOv5 is used to detect 8 predefined categories of ski equipment (skis, helmets, goggles, gloves, ski suits, knee pads, backpacks, and ski poles). For each detected equipment, its category label is mapped to a 256-dimensional feature vector through a learnable embedding layer. An improved Sigmoid function is used to weight the equipment detection confidence. After summing the weighted feature vectors of all equipment, the dimensionality is increased to 1024-dimensional equipment semantic feature vector through a fully connected layer.

[0019] In a preferred embodiment of this application, in step S01, the improved Sigmoid function formula for equipment feature extraction is: σ(x) = 1 / (1+e⁻ᵏ⁽ˣ⁻ 0·5 ⁾), where k is the steepness coefficient, k>5, and the default value is 8, which is used to enhance the detection results of high-confidence equipment and suppress low-confidence noise; the calculation process of equipment semantic feature vector includes: first, obtaining a 256-dimensional basic equipment feature vector by weighted summation, and then projecting it to 1024 dimensions through a fully connected layer to obtain the final equipment semantic feature vector.

[0020] As a preferred embodiment of this application, the specific implementation of step S02, adaptive multimodal feature fusion, is as follows:

[0021] 1. Dimensional Consistency: Ensure consistency in feature dimensions by integrating 1024-dimensional appearance features, 1024-dimensional motion features, and 1024-dimensional equipment features;

[0022] 2. Weighted fusion: Learnable weight coefficients α, β, and γ (satisfying α+β+γ=1) are used for weighted fusion to obtain the final fused feature vector. The initial weights are set to α=0.5, β=0.3, and γ=0.2.

[0023] 3. Adaptive Adjustment: The weight coefficients are adaptively adjusted based on the skier's movement speed (motion_blur). The faster the movement speed, the more severe the motion blur. The apparent feature weight α and equipment feature weight γ decrease, while the motion feature weight β increases. The adjustment rules are as follows: if motion_blur ≤ threshold (1000-1500 pixels / second), the weights remain unchanged; if motion_blur > threshold, then γ = min(γ × (threshold / motion_blur), 0.1), β = β × (motion_blur / threshold), α = 1 - β - γ.

[0024] As a preferred embodiment of this application, the specific implementation of step S03, which is based on time decay weighted clustering, is as follows:

[0025] 1. Time decay weight calculation: Calculate the time difference between the video segment feature extraction timestamp T_k and the predicted arrival time T_arrival, and calculate the time decay weight using the exponential decay function w = exp(-λ|T_k - T_arrival|), where λ is the decay coefficient (default 0.2). The larger the time difference, the lower the weight.

[0026] 2. Weighted distance calculation: Calculate the cosine distance between the fused feature vectors of different video segments, and combine it with their respective time decay weights to obtain the final similarity distance;

[0027] 3. Clustering Implementation: The OPTICS density clustering algorithm is used to cluster all video segments based on weighted similarity distance, and the top 3 video sets belonging to the same skier are output.

[0028] 4. Time prediction correction: Every 10 days of system operation, the correction coefficient tt is calculated based on the actual capture time of the video at each camera location, and the predicted arrival time T_arrival is calibrated. When λ deviates significantly from 1 (tt>1.5 or tt<0.6), the system alarm is triggered.

[0029] In a preferred embodiment of this application, the calculation process of the predicted arrival time T_arrival in step S03 includes:

[0030] 1. Speed ​​conversion preprocessing: Based on the actual width w of the camera screen coverage, calculate the actual distance r = w / 1920 (meters) represented by each pixel, and convert the pixel speed of the target in the video into physical speed v_phy = motion_blur × r;

[0031] 2. Arrival Time Prediction: Based on the trigger time T_trigger when the target passes through the current camera ROI region, the physical velocity v_phy, and the physical distance D between the current camera and the next camera, the kinematic model is used to predict the time T_arrival when the target arrives at the next camera ROI region.

[0032] 3. Correction and optimization: Based on historical actual arrival time data, a correction coefficient is calculated to dynamically calibrate T_arrival and reduce prediction error.

[0033] As a preferred embodiment of this application, the specific implementation of step S04, interactive training feedback, is as follows:

[0034] 1. Video Preview and Selection: Skiers can obtain and preview the top 3 clustered videos by entering their RFID tag ID number or uploading a full-body photo of themselves;

[0035] 2. User feedback: Users mark the clustering results on the APP interface to confirm "This is me" or "This is not me", and the system records the user feedback data;

[0036] 3. Model Optimization: The system collects user feedback data as new training samples and fine-tunes the fully connected layer and weight coefficients α, β, and γ of the feature fusion network through incremental learning to continuously improve clustering accuracy. A multimodal feature fusion and clustering system for skiing videos.

[0037] Secondly, embodiments of the present invention also provide a multimodal feature fusion and clustering system for skiing videos, used to implement the above method, the system comprising:

[0038] Feature extraction module: This module is used to execute step S01, which performs human target detection and analysis on the input captured video slices, extracts human appearance features, motion features, and equipment semantic features, and completes the preliminary acquisition and processing of multimodal features. This module uses YOLOv5, ResNet100, and TSN temporal network as the core network, supports video input of 1080P and above resolution, can automatically identify 8 types of ski equipment, and output 1024-dimensional appearance features, 1024-dimensional motion features, and 1024-dimensional equipment semantic features.

[0039] Feature fusion module: used to execute step S02, adopts an adaptive weighted fusion strategy based on motion speed, dynamically adjusts the weight coefficients α, β, and γ, and outputs the final fused feature vector; this module supports multi-task training, taking into account the main task of feature fusion and auxiliary tasks of equipment classification and feature space optimization, and optimizes the model through ArcFace loss, Triplet loss, and equipment classification loss.

[0040] Clustering analysis module: used to execute step S03, including time decay module, weighted distance calculation module and clustering execution module; the time decay module calculates time decay weight and calibrates the predicted arrival time T_arrival; the weighted distance calculation module calculates the time decay weighted cosine distance between fused features; the clustering execution module uses the OPTICS density clustering algorithm to complete the clustering of video segments and outputs a video set of the same skier.

[0041] Interactive training feedback system: used to execute step S04, including video preview module, user feedback module and model optimization module; the video preview module allows users to obtain and preview clustering videos through RFID tag ID or their own photos; the user feedback module collects user labeling information on the clustering results; the model optimization module uses incremental learning to fine-tune model parameters and feature fusion weights using feedback data to achieve continuous optimization of system performance.

[0042] By implementing the embodiments of this invention, and by fusing skier appearance features, motion features, and equipment semantic features, combined with adaptive adjustment of fusion weights based on motion speed, and introducing time decay weights to optimize clustering results, this invention effectively solves the video retrieval challenges in skiing scenarios, such as strong occlusion, similar clothing, motion blur, and target feature accuracy decaying with distance. Furthermore, an interactive feedback mechanism enables model self-optimization, further improving retrieval accuracy. Simultaneously, the automated clustering process replaces manual screening, significantly improving scenic area operational efficiency and skier retrieval experience, providing an efficient and accurate solution for ski resort video services. Attached Figure Description

[0043] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0044] Figure 1 This is a system architecture diagram of a skiing video multimodal feature fusion and clustering system provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of the multimodal feature fusion and clustering method provided in the embodiments of the present invention;

[0046] Figure 3 This is a network architecture diagram of the feature extraction module provided in an embodiment of the present invention;

[0047] Figure 4 This is a flowchart of equipment feature extraction provided in an embodiment of the present invention;

[0048] Figure 5This is a diagram of the multi-task training architecture provided in an embodiment of the present invention;

[0049] Figure 6 This is a flowchart of cluster analysis provided in an embodiment of the present invention.

[0050] Explanation of reference numerals in the attached figures:

[0051] Figure 1 In Chinese: S01-Video Input Module, S02-Feature Extraction Module, S03-Feature Fusion Module, S04-Cluster Analysis Module, S05-Interactive Training Feedback System, S06-Video Output Module;

[0052] Figure 2 In Chinese: S01 - Multimodal feature extraction, S02 - Adaptive multimodal feature fusion, S03 - Time decay weighted clustering, S04 - Interactive training feedback;

[0053] Figure 3 In the diagram: 1-Input image, 2-Appearance branch (YOLOv5+ResNet100), 3-Motion branch (TSN temporal network), 4-Armament branch (YOLOv5), 5-Feature fusion unit;

[0054] Figure 4 In the diagram: S01 - Input human image, S02 - Equipment detection (YOLOv5), S03 - Output detection results, S04 - Feature normalization to 256 dimensions, S05 - Calculation of confidence weights, S06 - Weighted processing, S07 - Summation and aggregation, S08 - Output equipment features.

[0055] Figure 5 In the middle: S01 - Shared convolutional layer, S02 - Appearance branch (ArcFace Loss), S03 - Equipment branch (equipment classification Loss), S04 - Motion branch (Triplet Loss), S05 - Weighted fusion, S06 - Backpropagation;

[0056] Figure 6 In the process: S01 - Input fused feature vector, S02 - Calculate time decay weight, S03 - Construct weighted distance matrix, S04 - OPTICS clustering, S05 - Sort by cluster size, S06 - Output TOP3 clustering results. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0058] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0059] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by those skilled in the art to which this invention pertains.

[0060] Application scenarios of this invention:

[0061] 1. Skiers can reach speeds of 5-30 m / s during skiing, which can easily cause motion blur and affect the accuracy of feature extraction.

[0062] 2. The ski resort environment is complex, with interference factors such as strong glare, obstructions from external objects, similar clothing among skiers, and faces being obscured by goggles / helmets;

[0063] 3. The cameras in the scenic area are deployed at key locations on the ski slope, and the shooting distance is relatively far, making it difficult to identify the facial features of skiers. It is necessary to rely on multi-dimensional features to distinguish individuals.

[0064] 4. Skiers' skiing tracks have temporal continuity, and clustering effects can be optimized through time information to improve retrieval accuracy.

[0065] Please refer to Figure 1 This is a system architecture diagram of multimodal feature fusion and clustering for ski videos provided in the first embodiment of the present invention. The system mainly includes a video input module, a feature extraction module, a feature fusion module, a clustering analysis module, an interactive training feedback system, and a video output module. These modules work together to achieve accurate clustering and retrieval of ski videos.

[0066] S01, Video Input Module: Receives video streams captured by cameras in the scenic area, segments the video, extracts additional information such as timestamps and physical parameters of each video segment, and synchronously transmits the video segments and additional information to the feature extraction module.

[0067] S02, Feature Extraction Module: Receives video clips and additional information, extracts appearance features, motion features and equipment semantic features respectively, and transmits the three features to the feature fusion module;

[0068] S03, Feature Fusion Module: Performs dimensional unification and adaptive weighted fusion on the three features, outputs the final fused feature vector, and transmits it to the clustering analysis module;

[0069] S04, Clustering Analysis Module: Calculates feature similarity based on time decay weight, performs clustering using the OPTICS algorithm, outputs a set of videos of the same skier, and transmits them to the interactive training feedback system and video output module;

[0070] S05. Interactive Training Feedback System: Receives user feedback information, optimizes model parameters and fusion weights through incremental learning, and feeds it back to the feature fusion module and feature extraction module;

[0071] S06. Video Output Module: Displays the clustered video collection to the user for preview and selection.

[0072] Please refer to Figure 2 This is a flowchart of the multimodal feature fusion and clustering method provided in this embodiment of the invention. The specific process is as follows:

[0073] S01, Multimodal Feature Extraction: Input video clips captured by the camera and additional information, and extract three types of features respectively;

[0074] Example: Input a 3-second video clip with 1080P resolution, captured at 10:00:00 on [Date], 202X. The camera's field of view is w=192 meters wide, and the distance to the next camera is D=500 meters. Extract 1024-dimensional appearance features using YOLOv5+ResNet100, extract 1024-dimensional motion features using TSN network, and detect three types of equipment (snowboards, helmets, and goggles) using YOLOv5. After weighted fusion and dimensionality upscaling using a fully connected layer, obtain 1024-dimensional equipment semantic features.

[0075] S02, Adaptive Multimodal Feature Fusion: Projects three features onto 1024 dimensions, adjusts weights based on motion speed, and then fuses them;

[0076] Example: Calculate the speed of a skier in a video: motion_blur = 1200 pixels / second, set the threshold = 1000, and the initial weights α = 0.5, β = 0.3, γ = 0.2; since motion_blur > threshold, after adjustment γ = min(0.2 × (1000 / 1200), 0.1) ≈ 0.167, β = 0.3 × (1200 / 1000) = 0.36, α = 1 - 0.36 - 0.167 ≈ 0.473; obtain a final 1024-dimensional fused feature vector through weighted fusion.

[0077] S03, Time-decay weighted clustering: Calculate the time decay weights, construct the weighted distance matrix, and cluster using the OPTICS algorithm;

[0078] Example: Calculate the video feature extraction time T_k=10:00:01, the predicted arrival time T_arrival=10:00:03, λ=0.2, and the time decay weight w=exp(-0.2×|10:00:01-10:00:03|)=exp(-0.4)≈0.67; calculate the cosine distance between the fused features of this video and other videos, combine the weights to obtain the weighted similarity distance, and cluster using the OPTICS algorithm to output the TOP3 video set.

[0079] S04, Interactive Training Feedback: Users preview the video set, mark incorrect clustering results, and the system optimizes the model through incremental learning;

[0080] Example: When a user tags a video as "not me", the system uses the video features, user features, and tag information as training samples, fine-tunes the feature fusion weights α, β, and γ, and improves the accuracy of subsequent clustering.

[0081] Please refer to Figure 3 This is a network architecture diagram of the feature extraction module provided in an embodiment of the present invention. The module contains three branches, which extract features from three different modalities:

[0082] 1. Appearance Branch: Input a single frame of high-definition image, perform human detection through YOLOv5, extract features through ResNet100, optimize through IBN structure and ArcFace loss, and output 1024-dimensional appearance features.

[0083] 2. Motion Branch: Input 5 consecutive frames of images (extracted every 5 frames), extract spatiotemporal features through 3D convolution, segment and aggregate through TSN temporal network and trajectory pooling, and output 1024-dimensional motion features;

[0084] 3. Equipment Branch: Input a single frame of high-definition image, detect 8 types of equipment using YOLOv5, output equipment category and confidence score, and output 1024-dimensional equipment semantic features through embedding layer mapping, confidence score weighting, summation and aggregation.

[0085] Please refer to Figure 4 This is a flowchart of equipment feature extraction provided in an embodiment of the present invention. The specific process is as follows:

[0086] S01, Input Human Image: Receives a single frame of video image and focuses on the human body region;

[0087] S02, Gear Detection (YOLOv5): Detects skiing equipment in an image and outputs the equipment category and detection confidence.

[0088] S03, output the detection results: for example, snowboard (confidence 0.92), helmet (confidence 0.88), and snow goggles (confidence 0.85) were detected.

[0089] S04, Feature normalization to 256 dimensions: Map the category label of each piece of equipment to a 256-dimensional feature vector;

[0090] S05, Confidence weight calculation: The weights are calculated using a modified Sigmoid function (k=8), with the snowboard weight σ(0.92)=1 / (1+e⁻). 8 ⁽ 0·9 ²⁻ 0·5 (⁾)≈0.96, helmet weight≈0.94, snow goggles weight≈0.93;

[0091] S06, Weighted processing: Multiply the feature vector of each piece of equipment by its corresponding weight;

[0092] S07, Summation and Aggregation: Summing the weighted feature vectors to obtain 256-dimensional basic equipment features;

[0093] S08, Output Equipment Features: The 256-dimensional features are upgraded to 1024-dimensional features through a fully connected layer to obtain the final equipment semantic features.

[0094] Please refer to Figure 5 This is a diagram of the multi-task training architecture provided in an embodiment of the present invention. The system adopts a multi-task training method, sharing underlying convolutional features, with dedicated task heads corresponding to different tasks:

[0095] 1. Main task: Multimodal feature fusion, outputting the final fused feature;

[0096] 2. Auxiliary tasks: 3 major classification features (output prediction results for 3 categories), feature space optimization (improve feature discriminative power);

[0097] 3. Loss function: Total loss L = λ1×L_arc + λ2×L_triplet + L_eqp, where λ1=0.5, λ2=0.3, L_arc is ArcFace loss, L_triplet is Triplet loss, and L_eqp is equipment classification loss;

[0098] 4. Collaborative Mechanism: Through backpropagation, the gradients of the three tasks are fused, and the parameters of the shared convolutional layer and the head of each task are optimized simultaneously to improve the accuracy of feature extraction and fusion.

[0099] Please refer to Figure 6 This is a flowchart of clustering analysis provided in an embodiment of the present invention. The specific process is as follows:

[0100] S01, Input fusion feature vector: Receive the fusion features of all video segments output by the feature fusion module;

[0101] S02, Calculate the time decay weight: Calculate the time decay weight w based on T_k and T_arrival for each video;

[0102] S03, Construct a weighted distance matrix: Calculate the cosine distance between any two video fusion features, combine it with the time decay weight to obtain the weighted similarity distance, and construct the distance matrix;

[0103] S04, OPTICS Clustering: The OPTICS density clustering algorithm is used to perform clustering based on the distance matrix, forming multiple clusters;

[0104] S05, Sort by cluster size: Sort the clusters by the number of videos they contain, and select the top 3 clusters with the most videos in each cluster.

[0105] S06, Output TOP3 clustering results: Output the video sets corresponding to the TOP3 clusters as candidate videos for the same skier.

[0106] By implementing the embodiments of the present invention, the video retrieval problem in the complex environment of ski resorts is effectively solved by fusing three modal features and combining adaptive weight adjustment and time decay clustering, thereby improving retrieval accuracy and efficiency. The interactive feedback mechanism enables continuous optimization of the model, further adapting to the diversity of skiing scenarios. The system has a high degree of automation, replacing manual screening, reducing the operating costs of scenic spots, and improving the user experience for skiers.

[0107] This invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the aforementioned multimodal feature fusion and clustering method for skiing videos.

[0108] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. Those skilled in the art can implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

Claims

1. A method for multimodal feature fusion and clustering of skiing videos, characterized in that, Includes the following steps: S01: Multimodal feature extraction, extracting the skier's appearance features, motion features, and equipment semantic features from video clips captured by the camera; S02: Adaptive multimodal feature fusion, which projects the equipment semantic features, appearance features, and motion features to a unified dimension, and then performs adaptive weighted fusion based on the target motion speed to obtain the final feature representation; S03: Based on time decay weighted clustering, calculate the similarity between the final feature representations of different video segments. The similarity calculation introduces a weight based on time decay for adjustment. Based on the similarity after time decay weighting, the video segments are clustered to output a video set of the same skier. S04: Interactive training feedback, receiving user feedback on clustering results, and optimizing the weight parameters or model parameters of feature fusion based on the feedback.

2. The method as described in claim 1, characterized in that, In step S01, the equipment semantic features are obtained by detecting a variety of predefined ski equipment and weighting and fusing them based on their detection confidence; the appearance features and motion features are extracted by human detection network and temporal network, respectively.

3. The method as described in claim 1, characterized in that, In step S02, the adaptive weighted fusion dynamically adjusts the weight coefficients based on the skier's movement speed. The faster the movement speed, the higher the weight of the movement features, and the lower the weight of the equipment features and appearance features.

4. The method as described in claim 1, characterized in that, In step S03, the time decay weight is determined based on the difference between the feature extraction time of the video segment and the predicted arrival time, and the predicted arrival time is predicted by the physical movement speed and distance of the target between different cameras.

5. The method as described in claim 1, characterized in that, In step S04, the interactive training feedback specifically includes: skiers obtain and preview the clustered video set by entering the RFID tag ID number or uploading a full-body photo of themselves; users mark and confirm the clustering results on the interface; and the system optimizes the model through incremental learning based on the feedback data.

6. A multimodal feature fusion and clustering system for skiing videos, used to implement the method described in any one of claims 1-5, characterized in that, The system includes: The feature extraction module is used to perform the multimodal feature extraction; The feature fusion module is used to perform the adaptive multimodal feature fusion; The clustering analysis module is used to perform the time-decay weighted clustering. An interactive training feedback system is used to perform the interactive training feedback.

Images