Event dynamic behavior recognition and real-time analysis method based on multi-modal deep learning

Abstract

The application discloses a multi-modal deep learning-based event dynamic behavior recognition and real-time analysis method, which comprises the following steps: acquiring multi-modal data and performing pretreatment; performing feature extraction on the pretreated multi-modal data, fusing different modal features extracted through a deep interaction and semantic alignment technology into global multi-modal representation; based on an edge model, performing initial recognition on the global multi-modal representation to obtain an initial prediction result; acquiring a prediction confidence corresponding to the initial prediction result; if the prediction confidence is not less than a preset prediction confidence, directly taking the initial prediction result as an edge end output; if the prediction confidence is less than the preset prediction confidence, based on a cloud end model, performing recognition on the global multi-modal representation to obtain a final recognition result and sending the edge end output. The application realizes real-time and intelligent processing of personnel action recognition, event detection, data statistics and visual analysis in sports events.

Description

Technical Field

[0001] This invention relates to the field of sports event data analysis technology, and in particular to a method for identifying and analyzing dynamic behaviors in sports events in real time based on multimodal deep learning. Background Technology

[0002] With the rapid development of the sports industry and intelligent media technology, event video analysis, athlete behavior recognition, and key event detection have become important research directions in the intelligentization of sports data. Currently, event analysis systems mainly rely on single-modal video data for moving target detection and action recognition, using convolutional neural networks (CNNs) or spatiotemporal convolutional networks (3D-CNNs) to classify athlete movements. However, single visual modalities have the following significant problems in complex event environments:

[0003] Limited recognition accuracy: When there are changes in lighting, occlusion, camera angle shift, or multi-person interaction scenarios, the feature representation ability of the visual model decreases significantly, resulting in insufficient recognition accuracy.

[0004] Insufficient real-time performance: The computational overhead of complex visual reasoning is large, making it difficult to meet the millisecond-level response requirements of live streaming or referee assistance systems;

[0005] Semantic information deficiency: Traditional video analysis systems cannot utilize audio signals (such as whistles, audience reactions, and narration) and text information (such as subtitles, comments, and technical statistics), thus lacking a deep understanding of the semantics of events;

[0006] Multi-source data fragmentation: The asynchronous timestamps and semantic misalignment between different modal data make it difficult to integrate cross-modal information and construct a unified semantic representation of events.

[0007] In recent years, multimodal learning and deep neural networks have achieved remarkable results in image, speech, and natural language processing. However, in sports event scenarios, there is still a lack of mature engineering solutions for achieving efficient fusion, temporal synchronization, and real-time event detection of multimodal data. Most existing systems only remain at the offline analysis stage and cannot achieve real-time data updates, event triggering, and visualization.

[0008] Therefore, there is an urgent need for an intelligent system that can integrate multimodal information from video, audio, and text, and achieve action recognition, event detection, and visualization output on a millisecond timescale, in order to support the needs of real-time judgment assistance, tactical analysis, and data presentation in live sports broadcasts. Summary of the Invention

[0009] To address the aforementioned technical challenges, this invention proposes a method for dynamic behavior recognition and real-time analysis in sports events based on multimodal deep learning. This method comprehensively utilizes heterogeneous data from multiple sources, including video, audio, and text, and leverages the cross-modal feature fusion and temporal modeling capabilities of deep neural networks. It enables real-time and intelligent processing of personnel action recognition, event detection, data statistics, and visualization analysis in sports events. This method is applicable to various scenarios such as live sports event data presentation, referee assistance in decision-making, tactical analysis, and intelligent event management.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows:

[0011] A method for identifying and analyzing dynamic behaviors in sporting events based on multimodal deep learning, comprising at least one edge device and one cloud device, wherein an edge model is deployed on the edge device and a cloud model is deployed on the cloud device, and includes the following steps:

[0012] Acquire and preprocess multimodal data, including images, point clouds, audio, and text;

[0013] Feature extraction is performed on the preprocessed multimodal data, and the extracted features from different modalities are fused into a global multimodal representation through deep interaction and semantic alignment techniques.

[0014] Based on the edge model, the global multimodal representation is initially identified to obtain initial prediction results;

[0015] Obtain the prediction confidence corresponding to the initial prediction result; if the prediction confidence is not less than the prediction confidence, directly use the initial prediction result as the edge output; if the prediction confidence is less than the prediction confidence, identify the global multimodal representation based on the cloud model to obtain the final identification result and send it to the edge output.

[0016] Preferably, the preprocessing includes the following steps:

[0017] Perform resolution scaling, color normalization, and geometric enhancement operations on the image data;

[0018] Perform downsampling, filtering, coordinate transformation, voxelization, or neighborhood construction on point cloud data;

[0019] Convert the time-domain signal in the audio data into a spectrum or mel spectrum, and then normalize it;

[0020] Perform word segmentation, sub-word encoding, and removal of invalid symbols on text data.

[0021] Preferably, the feature extraction includes the following steps:

[0022] Visual Transformer or convolutional neural network is used to map image data into image features;

[0023] Point cloud data is encoded into point cloud features using PointNet, Point Transformer, or voxel convolutional networks;

[0024] Acoustic coding networks based on one-dimensional convolution or Transformer are used to encode audio data into audio features;

[0025] The Transformer language model is used to map text data into text features.

[0026] Preferably, the step of fusing extracted modal features into a global multimodal representation through deep interaction and semantic alignment techniques includes the following steps:

[0027] For each layer of cross-modal attention, point cloud features, audio features, and text features are modally concatenated to obtain an auxiliary sequence. The image features are used as the projection of the main modality to obtain the query matrix, and the auxiliary sequence is projected to obtain the key matrix and value matrix respectively. The attention weights between the main modality and the auxiliary sequence are calculated through a single-head attention mechanism and weighted aggregation is performed to obtain the single-head attention result. The single-head attention results calculated in parallel are concatenated and then subjected to linear transformation to obtain the multi-head attention output.

[0028] Stack several layers of cross-modal Transformers to output a fused full token sequence; pool the fused full token sequence to obtain a global multimodal representation.

[0029] Preferably, it also includes open word recognition, comprising the following steps:

[0030] Unknown class samples are labeled with categories, and corresponding natural language descriptions are generated based on the category labels. The natural language descriptions are then input into a text encoder to obtain the category codes.

[0031] The global multimodal representation and category encoding are L2 normalized and the cosine similarity between the normalized vectors is calculated. The cosine similarity is converted into a probability distribution using the softmax function. The category with the highest probability is selected as the prediction result. If the maximum similarity of all categories is less than the preset category determination threshold, it is determined to be an unknown class.

[0032] Preferably, the total loss function during the training process of the cloud model is... The formula is as follows:

[0033]

[0034] In the formula, For multi-task joint losses, For self-monitoring losses, This is the supervised loss corresponding to the pseudo-label. It is a hyperparameter used to balance different loss weights;

[0035]

[0036]

[0037] in, These are the predicted features at time t+1. It is the true characteristic at time t+1. For cross-entropy loss in classification tasks, This is the sum of the bounding box regression loss, the object existence loss, and the detection classification loss in the detection task. For image segmentation tasks, this refers to pixel-level cross-entropy or Dice loss. For open vocabulary recognition tasks, cross-entropy loss, , , , These are the weights for classification tasks, detection tasks, image segmentation tasks, and open vocabulary recognition tasks, respectively.

[0038] Based on the above technical solution, the beneficial effects of the present invention are:

[0039] 1) Multimodal fusion enhances recognition stability and robustness in complex scenarios. By fusing multimodal data such as images, point clouds, audio, and text, the system can complement each other across modalities, achieving scene understanding capabilities far superior to those of a single sensor. Point clouds supplement 3D spatial information, solving the problem of missing depth and supporting key judgments such as offside and ball trajectory; audio captures event signals such as whistles and ball-hitting sounds, which can assist in event judgment when visual occlusion occurs; text provides rules and semantic constraints, helping to understand the meaning of the event behind the actions, significantly improving the system's reliability in extreme event environments. Automatically learning semantic mappings between different modalities, it associates visual actions, audio signals, and text descriptions to form a complete understanding of event events; the generated global representation can support parallel processing of multiple tasks such as tactical analysis, technical statistics, and fine-grained action recognition; focusing on key features and suppressing irrelevant noise improves the accuracy and recall rate of core event detection such as goals and fouls, providing reliable support for real-time broadcasting and auxiliary judgment.

[0040] 2) The cross-modal attention mechanism can automatically learn the semantic mapping relationships between different modalities, achieving high-quality semantic alignment and unified representation of event data: improving modality matching accuracy, strengthening the semantic association between visual and textual, and visual and point cloud data, forming a global multimodal representation, and providing a unified feature foundation for event analysis; this representation can be shared by visual tasks such as event scene search, semantic localization, and text guidance, significantly improving the overall expressive power of the model, such as supporting complex event analysis needs like "finding all player foul scenes," "locating key frames of tactical positioning," and "filtering specific offensive actions based on text descriptions," achieving more efficient and accurate event data mining and event understanding.

[0041] 3) The cloud-based model learns the evolution patterns of the game scenario through a temporal Transformer to predict future events: It can predict key game dynamics such as player running trajectories, ball landing points, and tactical development trends, providing forward-looking data support for real-time tactical prompts, referee-assisted judgments, game reviews, and tactical optimization; Based on temporal understanding capabilities, it can identify potential foul risks, key scoring opportunities, and other events in advance, improving the predictability and timeliness of game analysis, helping coaching teams to quickly adjust tactics, refereeing teams to intervene in judgment scenarios in advance, and optimizing game operations and competitive performance.

[0042] 4) This invention adopts a semantic similarity matching strategy of global multimodal representation and text description, supports open vocabulary recognition, and can instantly recognize new categories without retraining: it can quickly adapt to new action or event types under different projects and different competition rules, such as new foul gestures in basketball and special tactical actions in football. Only the category description needs to be input to expand the recognition range, meeting the needs of rapid changes brought about by the update of competition rules and project innovation; it supports "0-shot recognition", which greatly improves the versatility and portability of the competition system. There is no need to repeatedly develop models for different sports projects. It can be quickly migrated to various competition scenarios such as basketball, football, and badminton, reducing deployment costs and cycle.

[0043] 5) A unified multi-task recognition architecture reduces redundancy and improves overall efficiency. The system is based on a unified multimodal representation and supports tasks such as classification, detection, segmentation, tracking, and image / text generation. Different tasks share feature bases, improving model efficiency and generalization ability, reducing training costs and model storage redundancy, and making it easier to scale to new tasks, achieving "train once, use for multiple tasks".

[0044] 6) Employing a collaborative mechanism of edge fast response + cloud deep inference, perfectly adapting to the needs of the event scenario: The edge achieves millisecond-level response, and can output key data such as player position, ball trajectory, and basic foul judgment in real time, meeting the needs of real-time data presentation and referee assistance in live broadcasts; when encountering complex scenarios (such as multiple people obscuring the view, high-speed motion blur, and controversial calls), the data is automatically uploaded to the cloud for fine inference, outputting high-precision event analysis results, effectively balancing latency and inference accuracy; this architecture reduces the computing power pressure on the equipment at the event site, enabling the system to run stably on edge devices with limited resources, while the deep inference in the cloud can provide more comprehensive event review data (such as tactical replays and detailed analysis of player movements), taking into account both real-time performance and analytical depth.

[0045] 7) Self-supervised learning and automatic labeling significantly reduce labeling costs. Through a cloud-based model prediction error-driven self-supervised training and automatic label generation mechanism, this patent reduces reliance on manually labeled large-scale data. It continuously improves model performance using unlabeled data, reducing data labeling costs by over 70%, automatically generating high-quality labels (requiring only manual review), increasing productivity, and enabling the model to learn continuously and adapt to long-term changing real-world environments.

[0046] 8) The system possesses strong generalization capabilities, adapting to multiple industries and scenarios. Its multimodal architecture combined with a unified semantic representation structure enables the system to be independent of specific scenarios or tasks, allowing for rapid adaptation to various sports events: There's no need to repeatedly develop models for different sports (such as basketball, football, badminton, and track and field); only minor adjustments are required for application migration, significantly reducing deployment costs and timelines across multiple event scenarios. It exhibits strong robustness when migrating across scenarios (indoor / outdoor venues, day / night, different lighting conditions) and devices (high-definition cameras / mobile devices, different sensor configurations), reducing the decrease in recognition accuracy caused by scene changes and providing a stable foundation for the standardized collection and analysis of event data.

[0047] 9) Possesses future scalability, forming the foundation for general artificial intelligence vision. This patent can serve as the underlying visual foundation for higher-level intelligent systems in the sports field: it can be naturally integrated into sports AI Agent systems to achieve "vision + language + action" linkage, supporting automated operations based on sports scene understanding, such as automatically generating match reports, tactical review reports, and key event clips, or triggering automated processes such as tactical prompts and referee reminders based on real-time match data; it provides key visual capabilities for building a general sports artificial intelligence platform, supporting subsequent expansion to advanced application scenarios such as intelligent commentary, virtual referee assistance, and player training analysis, and has the potential for continuous iteration and upgrades. Attached Figure Description

[0048] Figure 1This is a schematic diagram of a method for identifying and analyzing dynamic behaviors in sports events based on multimodal deep learning, as shown in one embodiment. Detailed Implementation

[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0050] like Figure 1 As shown, this embodiment provides a method for identifying and analyzing dynamic behaviors in sports events in real time based on multimodal deep learning, including the following steps:

[0051] S100:

[0052] Time Index:

[0053] Multimodal data is acquired in real time via edge devices (cameras, LiDAR, microphones, caption capture devices). This multimodal data includes... ,as follows:

[0054]

[0055] Examples of event scene data: Images: 25 frames per second of player motion frames in a football match, close-up of a basketball dunk, and referee gestures; Point clouds: 3D point clouds of player positions and basketball flight trajectory collected by LiDAR; Audio: Referee whistles, player shouts, audience cheers, and real-time commentary; Text: Live broadcast subtitles, event technical statistics text, and referee decision records text.

[0056] The time synchronization unit is based on the timestamp. Using this as an anchor point, select items whose time deviation does not exceed a threshold. The samples satisfy: This constitutes a unified time-aligned multimodal set. :

[0057]

[0058] S110:

[0059] Preprocessing of the multimodal data is performed at the edge to obtain preprocessed image data. Point cloud data Audio data Text data ,as follows:

[0060]

[0061] Among them: image preprocessing Combined operators such as resolution scaling, normalization, and data augmentation; point cloud preprocessing. Downsampling, filtering, coordinate transformation / calibration, etc. Audio time-frequency diagrams are obtained using STFT (Short Time Fourier Transform) or Mel (Mel Correlation Transform). : ; Word segmentation, sub-word encoding, etc.

[0062] Spatial calibration (taking the camera coordinate system as an example):

[0063]

[0064] in As an external reference, This is a camera projection model.

[0065] Form a standardized input after preprocessing at the edge (to avoid bandwidth pressure on the original data transmission and ensure real-time performance):

[0066]

[0067] S120:

[0068] S120-1 Image Encoding

[0069] Preprocessed image After linear mapping and visual Transformer (or CNN) encoding, the output dimension is... Image features :

[0070]

[0071] S120-2 Point Cloud Encoding

[0072] Aligned point cloud set Feature encoding of an unordered point set is performed using the PointNet point cloud network, with an output dimension of [dimensionality missing]. Point cloud features :

[0073]

[0074] S120-3 audio encoding

[0075] Time-frequency graph obtained by converting audio signals Treated as image / sequence data, input to a dedicated acoustic encoder, with an output dimension of... audio features :

[0076]

[0077] S120-4 Text Encoding

[0078] The text is tokenized using a tokenizer to obtain a token sequence. The sequence is fed into the Transformer language model to complete semantic encoding, and the output dimension is... Text features :

[0079]

[0080] Output of this step:

[0081]

[0082] S130:

[0083] Each sensor in time Collect raw observations

[0084] S130-1 Cross-modal attention computation

[0085] Image tokens are used as the primary modality, and feature sequences from other modalities are concatenated to form auxiliary sequences. :

[0086]

[0087] For each layer of cross-modal attention, the main modality (image) features Projection yields the query matrix The key matrix is ​​obtained by projecting the auxiliary sequences respectively. Sum matrix :

[0088]

[0089] Single-head attention. The standard Transformer scaled dot product attention formula is used to calculate the attention weights of the main modality on the auxiliary sequence and perform weighted aggregation:

[0090]

[0091] in yes The dimension of the vector is used to scale the dot product result to avoid numerical overflow.

[0092] Multiple attention. The outputs of the individual attention heads are concatenated and then output through the projection matrix. Interaction results integrating multiple attention groups:

[0093]

[0094] S130-2 Fusion Global Multimodal Representation

[0095] Stack several layers of cross-modal Transformers (denoted as...) This allows multimodal features to interact deeply and outputs the fused full token sequence:

[0096]

[0097] in It is the total number of all tokens after merging. It is the feature dimension.

[0098] Then, the global multimodal representation is obtained through pooling:

[0099]

[0100] For example, average pooling can be used:

[0101]

[0102] S140:

[0103] Each sensor in time Collect raw observations

[0104] S140-1 Timing State Modeling

[0105] Global multimodal feature sequences within the historical time window As a temporal state, it is input into a cloud-based temporal model (such as Transformer or GRU), and outputs the hidden state representation at the current time step. :

[0106]

[0107] S140-2 Future Feature Prediction

[0108] Cloud-based models utilize time-series modules with learnable parameters. Predict the hidden state or features in the next time step. :

[0109]

[0110] By restoring the feature space through linear mapping, the predicted features at time t+1 are obtained. :

[0111]

[0112] S140-3 Cloud Model Training Loss

[0113] Mean squared error loss can be used to sum the squared 2-norm of the predicted features and the true features for all historical time points, thus measuring the global prediction bias.

[0114]

[0115] KL divergence or contrastive learning loss can also be introduced.

[0116] S150:

[0117] Each sensor in time Collect raw observations and extract global multimodal representations. .

[0118] S150-1 Category Text Encoding

[0119] For each category Natural Language Description Input text encoder:

[0120]

[0121] To ensure numerical stability, L2 normalization is performed:

[0122]

[0123] S150-2 Semantic Similarity and Open Lexical Classification

[0124] Cosine similarity:

[0125]

[0126] The probability distribution of open words can be obtained using softmax:

[0127]

[0128] in Temperature coefficient. Prediction category:

[0129]

[0130] At the same time, new category determination thresholds can be set. ,when It is determined to be "unknown class" in a timely manner, which effectively avoids misclassification in open scenarios.

[0131] S160:

[0132] Each sensor in time Collect raw observations

[0133] S160-1 Classification Header

[0134] The fully connected layer performs multi-class classification, and the global multimodal representation extracted at time t outputs the classification result:

[0135]

[0136] Cross-entropy loss in classification tasks:

[0137]

[0138] S160-2 Detection Head

[0139] Will As a spatial feature map, the decoder obtains anchor box candidate features, and then the bounding box and category are regressed:

[0140]

[0141] in For the first Candidate box parameters The category logits for this candidate bounding box. The detection task loss is the bounding box regression loss. Target Existence Loss Detection and classification loss The sum can be written as:

[0142]

[0143] S160-3 Segmentation Head

[0144] based on The upsampling decoder generates a size of [size]. The number of channels is the number of categories. Pixel-level mask:

[0145]

[0146] Pixel-level cross-entropy or Dice loss can be used:

[0147]

[0148] S160-4 Multi-task Joint Loss

[0149] Perform multi-task joint optimization on a unified feature space:

[0150]

[0151] in For open vocabulary recognition, cross-entropy loss, This represents the task weight.

[0152] S170:

[0153] Edge model and cloud model are respectively .

[0154] Edge-to-edge models for global multimodal representation First, perform local fast inference:

[0155]

[0156] Marginal reasoning confidence for:

[0157]

[0158] Collaborative decision-making rules: setting confidence thresholds This serves as the criterion for determining the splitting of reasoning tasks; when the confidence level of marginal reasoning... When: This indicates that the edge model is sufficiently certain about the result, eliminating the need to upload data and allowing for direct local output of the decision, ensuring millisecond-level low latency. Or, if a high-risk scenario is detected: this indicates that the inference results of the edge model have high uncertainty, or that the scenario has extremely high requirements for accuracy / security. In this case, the features or even the raw data are uploaded to the cloud, and a high-precision cloud model is called to complete the inference.

[0159]

[0160] Final decision output :

[0161]

[0162] S180:

[0163] S180-1 Predictive Self-Supervision

[0164] The self-supervised training signal is directly obtained by minimizing the L2 squared error between the future features predicted by the cloud model and the actual future features:

[0165]

[0166] It can also be extended to the form of contrastive loss (InfoNCE):

[0167]

[0168] S180-2 Automatic Labeling and Incremental Learning

[0169] For unlabeled samples (corresponding features) The pseudo-label distribution is obtained using the current model:

[0170]

[0171] If the maximum confidence level of the sample satisfies The highest-confidence category predicted will then be used as the pseudo-label.

[0172]

[0173] The filtered high-confidence pseudo-labeled samples Add an expanded training set and minimize the result through incremental training:

[0174]

[0175] In the formula, For multi-task joint losses, For self-monitoring losses, This is the supervised loss corresponding to the pseudo-label. It is a hyperparameter used to balance different loss weights.

[0176] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0177] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A method for dynamic behavior recognition and real-time analysis in sports events based on multimodal deep learning, comprising at least one edge terminal and a cloud terminal, wherein an edge model is deployed at the edge terminal and a cloud model is deployed in the cloud terminal, characterized in that, Includes the following steps: Acquire and preprocess multimodal data, including images, point clouds, audio, and text; Feature extraction is performed on the preprocessed multimodal data, and the extracted features from different modalities are fused into a global multimodal representation through deep interaction and semantic alignment techniques. Based on the edge model, the global multimodal representation is initially identified to obtain initial prediction results; Obtain the prediction confidence corresponding to the initial prediction result; if the prediction confidence is not less than the prediction confidence, directly use the initial prediction result as the edge output; if the prediction confidence is less than the prediction confidence, identify the global multimodal representation based on the cloud model to obtain the final identification result and send it to the edge output.

2. The method for identifying and analyzing dynamic sports behavior in real time based on multimodal deep learning according to claim 1, characterized in that, The preprocessing includes the following steps: Perform resolution scaling, color normalization, and geometric enhancement operations on the image data; Perform downsampling, filtering, coordinate transformation, voxelization, or neighborhood construction on point cloud data; Convert the time-domain signal in the audio data into a spectrum or mel spectrum, and then normalize it; Perform word segmentation, sub-word encoding, and removal of invalid symbols on text data.

3. The method for identifying and analyzing dynamic sports behavior based on multimodal deep learning according to claim 2, characterized in that, The feature extraction includes the following steps: Visual Transformer or convolutional neural network is used to map image data into image features; Point cloud data is encoded into point cloud features using PointNet, Point Transformer, or voxel convolutional networks; Acoustic coding networks based on one-dimensional convolution or Transformer are used to encode audio data into audio features; The Transformer language model is used to map text data into text features.

4. The method for identifying and analyzing dynamic sports behavior in real time based on multimodal deep learning according to claim 3, characterized in that, The process of fusing extracted modal features into a global multimodal representation through deep interaction and semantic alignment techniques includes the following steps: For each layer of cross-modal attention, point cloud features, audio features, and text features are modally concatenated to obtain an auxiliary sequence; the image features are used as the projection of the main modality to obtain the query matrix, and the auxiliary sequence is projected to obtain the key matrix and value matrix respectively. The attention weight between the main modality and the auxiliary sequence is calculated through a single-head attention mechanism and weighted aggregation is completed to obtain the single-head attention result. The multi-head attention output is obtained by concatenating the results of multiple parallel single-head attention calculations and then performing a linear transformation. Stack several layers of cross-modal Transformers to output a fused full token sequence; pool the fused full token sequence to obtain a global multimodal representation.

5. The method for identifying and analyzing dynamic sports behavior in real time based on multimodal deep learning according to claim 1, characterized in that, It also includes open word recognition, which includes the following steps: Unknown class samples are labeled with categories, and corresponding natural language descriptions are generated based on the category labels. The natural language descriptions are then input into a text encoder to obtain the category codes. The global multimodal representation and category encoding are L2 normalized and the cosine similarity between the normalized vectors is calculated. The cosine similarity is converted into a probability distribution using the softmax function. The category with the highest probability is selected as the prediction result. If the maximum similarity of all categories is less than the preset category determination threshold, it is determined to be an unknown class.

6. The method for identifying and analyzing dynamic sports behavior in real time based on multimodal deep learning according to claim 5, characterized in that, The total loss function during the training of the cloud model The formula is as follows: In the formula, For multi-task joint losses, For self-monitoring losses, This is the supervised loss corresponding to the pseudo-label. It is a hyperparameter used to balance different loss weights; , ,in, These are the predicted features at time t+1. It is the true characteristic at time t+1. For cross-entropy loss in classification tasks, This is the sum of the bounding box regression loss, the object existence loss, and the detection classification loss in the detection task. For image segmentation tasks, this refers to pixel-level cross-entropy or Dice loss. For open vocabulary recognition tasks, cross-entropy loss, , , , These are the weights for classification tasks, detection tasks, image segmentation tasks, and open vocabulary recognition tasks, respectively.

Images