Sports event multi-modal data fusion analysis method based on AI real-time feedback

Abstract

The invention relates to the technical field of sports event data analysis, and discloses a sports event multi-modal data fusion analysis method based on AI real-time feedback. The method comprises the following steps: acquiring real-time multi-modal data from multiple data sources such as video acquisition equipment and an audio sensor, preprocessing, inputting into a fusion analysis model to generate a fusion feature vector, obtaining a target feature vector through a dynamic weight distribution algorithm, inputting into a real-time feedback model to generate a real-time analysis result, and finally performing label mapping by using a classification model. Meanwhile, the fusion analysis model is trained and optimized, the real-time feedback model can adjust a strategy according to a match state, and the classification model can update a label mapping rule online. According to the method, efficient fusion analysis and real-time feedback of the multi-modal data of the sports event are realized, the accuracy and practicability of event analysis are improved, and powerful support is provided for decision making, appreciation and the like of the sports event.

Description

Technical Field

[0001] The present invention relates to the technical field of sports event data analysis, and specifically to a multi-modal data fusion analysis method for sports events based on AI real-time feedback. Background Art

[0002] In the field of sports events, with the rapid development of technology, data-driven event analysis has become increasingly important. Traditional sports event analysis methods mainly rely on manual observation and simple data statistics. This method is not only inefficient but also difficult to obtain comprehensive and in-depth event information, with many limitations. In terms of data acquisition, only limited event data such as game scores and player playing time could be collected in the past, and it was impossible to cover various aspects of information such as the subtle movements of athletes during the game and the on-site atmosphere. However, modern sports events have higher and higher requirements for the comprehensiveness and accuracy of data. For example, in professional football games, the coaching team needs to understand detailed data such as the physical energy consumption and the quality of technical movements of players during the game in order to make targeted tactical adjustments and player training arrangements, but the traditional method is difficult to meet this requirement.

[0003] At the level of data processing and analysis, early analysis methods could not effectively process and integrate various types of data. Nowadays, the data generated in sports events presents multi-modal characteristics, including video data, audio data, athlete movement trajectory data, etc. Taking basketball games as an example, video data can show the running positions and passing actions of players; audio data can reflect the emotions of on-site audiences and the tactical guidance of coaches; movement trajectory data can accurately present the movement paths and speed changes of players on the court. However, traditional analysis methods cannot organically combine these multi-modal data for comprehensive and in-depth analysis, resulting in the inability to fully explore the value behind the data.

[0004] In addition, traditional sports event analysis lacks real-time performance. During intense games, relevant personnel such as coaches and referees need to obtain accurate event analysis results in a timely manner in order to make reasonable decisions. For example, in tennis games, referees need to quickly judge whether a player's serve is valid and whether there are fouls; coaches need to adjust tactics according to the real-time situation of the game. However, due to the slow processing speed of traditional analysis methods, effective analysis results cannot be provided in a timely manner, affecting the decision-making efficiency and viewing experience of the event.

[0005] With the continuous progress of artificial intelligence technology and sensor technology, new possibilities are provided to solve the above problems. However, the current research and application of multi-modal data fusion analysis for sports events are still in the development stage, and a mature and perfect technical system has not yet been formed. Therefore, developing a method that can efficiently acquire, process, and analyze multi-modal data of sports events and achieve real-time feedback has become an urgent problem to be solved in the current field of sports event data analysis. Summary of the Invention

[0006] The object of the present invention is to provide a multi-modal data fusion analysis method for sports events based on AI real-time feedback to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: A multi-modal data fusion analysis method for sports events based on AI real-time feedback, the method comprising: Obtain real-time multi-modal data from at least one data source, wherein the data source includes a video acquisition device, an audio sensor, a motion sensor and a user terminal, and the multi-modal data includes video stream data, audio stream data, motion trajectory data and user interaction data; Preprocess the multi-modal data to eliminate noise and extract initial features; Input the initial features into a preset fusion analysis model to generate a fusion feature vector, wherein the fusion analysis model includes parallel convolutional neural network branches and recurrent neural network branches, the convolutional neural network branches are used to extract spatial features, and the recurrent neural network branches are used to extract temporal features; Based on a preset dynamic weight allocation algorithm, perform weighted integration on the fusion feature vector to obtain a target feature vector, wherein the dynamic weight allocation algorithm adjusts the weight coefficients of the features of each branch according to the data quality and real-time update frequency of the multi-modal data; Input the target feature vector into a preset real-time feedback model to generate a real-time analysis result, wherein the real-time feedback model is constructed based on a streaming data processing architecture and is used to dynamically adjust the feature extraction strategy according to the current event state; Use a preset classification model to perform label mapping on the real-time analysis result.

[0008] Preferably, the method further includes training the fusion analysis model in the following manner: Obtain a historical multi-modal data set, wherein the sample data in the historical multi-modal data set is labeled with corresponding fusion feature labels; divide the historical multi-modal data set into multiple training subsets based on a sliding time window and input them into an initial fusion model in chronological order for iterative training, and adjust the model parameters according to the difference rate between the predicted features and the fusion feature labels after each iteration until the average value of the difference rate is less than or equal to a preset threshold; During the training process, the scale of the training subset for each iteration is dynamically reduced according to the difference rate of the previous iteration, specifically including: calculating the difference rate of each sample in the previous iteration, and retaining the samples with a difference rate lower than the median value after arranging them in ascending order of the difference rate to form a new training subset.

[0009] Preferably, obtaining real-time multi-modal data from at least one data source includes: Capturing panoramic video stream data and partial close-up video stream data of the event venue through a video acquisition device at a first sampling rate; Collecting audio data related to the motion state through an audio sensor deployed on the athlete's equipment, and collecting acceleration, angular velocity, and position coordinate data through a motion sensor at a second sampling rate; Receiving real-time comment data and interaction request data input by the audience through a user terminal, and converting the comment data into a text feature vector.

[0010] Preferably, preprocessing the multi-modal data further includes: Extracting key frames from the panoramic video stream data and partial close-up video stream data, and calculating the motion vector between adjacent frames based on the optical flow method; Performing frame windowing processing on the audio stream data, and extracting acoustic features through Mel-frequency cepstral coefficients; Performing Kalman filtering on the motion trajectory data to eliminate measurement noise and complete missing coordinate points.

[0011] Preferably, the training process of the fusion analysis model further includes: In each iteration, calculating the branch contribution degree according to the output features of the convolutional neural network branch and the recurrent neural network branch; Dynamically adjusting the number of network layers and the number of neurons of each branch in the initial fusion model based on the branch contribution degree.

[0012] Preferably, the calculation method of the branch contribution degree is: Calculating the first cosine similarity between the spatial features output by the convolutional neural network branch and the fusion feature label, and the second cosine similarity between the temporal features output by the recurrent neural network branch and the fusion feature label respectively; Taking the ratio of the first cosine similarity to the second cosine similarity as the branch contribution degree.

[0013] Preferably, the implementation of the dynamic weight allocation algorithm includes: Generating a quality evaluation coefficient according to the image clarity of the video stream data, the signal-to-noise ratio of the audio stream data, and the sampling integrity of the motion trajectory data; Determining the initial weight of each branch feature based on the product of the quality evaluation coefficient and the preset weight reference value; Exponentially decaying and adjusting the initial weight according to the real-time update frequency to obtain the final weight coefficient.

[0014] Preferably, the dynamic adjustment strategy of the real-time feedback model includes: Monitor the key event trigger signals in the current event stage. If a goal, foul, or substitution event is detected, increase the feature extraction dimension of the corresponding modal data; Dynamically expand or compress the number of computing nodes in the streaming data processing architecture according to the request frequency of the user terminal.

[0015] Preferably, the method for updating the label mapping rule of the classification model is as follows: Extract the corrected label of the user terminal for the real-time analysis result and the associated context features; Form an incremental training set with the corrected label and the context features, and update the decision boundary of the classification model through an online learning algorithm; The implementation of the online learning algorithm includes: Calculate the distance between the feature vectors of each sample in the incremental training set and the hyperplane of the current classification model; If the distance is less than the preset fault tolerance threshold, retain the original decision boundary; otherwise, adjust the hyperplane parameters through the support vector machine incremental training algorithm.

[0016] Preferably, after generating the real-time analysis result, the method further includes: pushing the real-time analysis result to the user terminal through a preset feedback interface, and updating the label mapping rule of the classification model according to the interaction data of the user terminal.

[0017] Compared with the prior art, the beneficial effects of the present invention are: In the data acquisition and preprocessing stage, by collecting data from multiple data sources such as video capture devices, audio sensors, motion sensors, and user terminals, rich and comprehensive multi-modal data can be obtained, covering video streams, audio streams, motion trajectories, and user interaction data, etc. This makes the analysis of the event no longer limited to a single dimension. For example, in a track and field event, not only can the motion trajectory data such as the running speed and displacement of the athlete be obtained, but also the audio data such as the cheers of the on-site audience, the breathing sounds of the athlete, and the video data of the athlete's action postures can be collected, comprehensively reflecting the event situation. At the same time, preprocessing the multi-modal data, such as extracting key frames and calculating motion vectors for video data, frame windowing and acoustic feature extraction for audio data, and Kalman filtering processing for motion trajectory data, effectively eliminates noise and extracts initial features, improves the data quality, and lays a solid foundation for subsequent accurate analysis.

[0018] In terms of the fusion analysis model, parallel convolutional neural network branches and recurrent neural network branches are adopted. The convolutional neural network branches are good at extracting spatial features, and the recurrent neural network branches focus on extracting temporal features. The two work together to generate fusion feature vectors, which can capture complex feature information in the data more accurately. Taking football matches as an example, the convolutional neural network can analyze spatial relationships such as the positions of players on the field and passing lines, while the recurrent neural network can process time series information such as the sequence of players' actions and changes in the game rhythm. Combining the advantages of both greatly improves the ability to extract event features.

[0019] The dynamic weight allocation algorithm adjusts the feature weight coefficients of each branch according to data quality and real-time update frequency, fully considering the importance differences of different data at different stages of the event. For example, in the critical moments of a basketball game, such as the decisive ball stage, the image clarity of video stream data and the sampling integrity of motion trajectory data are more critical for analysis and decision-making. The algorithm will correspondingly increase the weights of these data branches to ensure that the analysis results are more in line with the actual event situation, enhancing the reliability and pertinence of the analysis.

[0020] The real-time feedback model is built based on a streaming data processing architecture and can dynamically adjust the feature extraction strategy according to the event state. When key events such as goals, fouls, or substitutions are detected, the feature extraction dimensions of the corresponding modality data are increased; the number of computing nodes is dynamically expanded or compressed according to the request frequency of the user terminal. This enables the analysis system to quickly respond to event changes, timely adjust the analysis focus, and at the same time optimize the resource utilization efficiency. In a volleyball game, when a key scoring ball appears, the system quickly increases the feature extraction dimensions of the video, audio, and motion trajectory data at that moment, providing more detailed and accurate analysis results for the audience and coaches; when the audience's attention is high and the request frequency is large, the computing nodes are expanded to speed up the processing speed to ensure real-time performance.

[0021] The label mapping rules of the classification model can be updated online through the corrected labels and associated context features of the user terminal. This means that the model can continuously adapt to new event situations and user feedback, and continuously optimize the decision boundary. For example, in a badminton game, if the audience or professionals correct the label of a real-time analysis result, the system incorporates this correction information into the incremental training set and adjusts the classification model through an online learning algorithm, making the label mapping of subsequent analysis results more accurate and improving the practicality and credibility of the analysis results.

[0022] In addition, pushing the real-time analysis results to the user terminal and updating the label mapping rules according to the user interaction data realizes the two-way interaction between the system and the user. On the one hand, users can obtain accurate event analysis information in a timely manner, enhancing the viewing experience; on the other hand, the system continuously optimizes with the help of user feedback, forming a virtuous cycle and further improving the application value of the entire analysis method in sports events. Brief Description of the Drawings

[0023] Figure 1 It is the working principle diagram of the multi-modal data fusion analysis method for sports events described in the present invention; Figure 2 It is the flow chart of the training of the fusion analysis model; Figure 3 It is the flow chart of calculating the branch contribution degree of the fusion analysis model; Figure 4 It is the flow chart of updating the label mapping rule of the classification model. Detailed Embodiment

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] Please refer to Figures 1-4 , the present invention provides a multi-modal data fusion analysis method for sports events based on AI real-time feedback. The method includes: Obtain real-time multi-modal data: Collect data from multiple data sources. The data sources include video capture devices, audio sensors, motion sensors, and user terminals. The video capture device can capture video stream data of the event venue, the audio sensor captures audio stream data related to the motion state, the motion sensor obtains the motion trajectory data of the athlete, and the user terminal receives user interaction data input by the audience. These multi-modal data include video stream data, audio stream data, motion trajectory data, and user interaction data, providing a basis for subsequent analysis.

[0026] Preprocess the multi-modal data: Perform preprocessing operations on the collected multi-modal data. The purpose is to eliminate the noise in the data and extract the initial features. The preprocessed data can improve the accuracy and efficiency of subsequent analysis.

[0027] Generate a fusion feature vector: Input the initial features obtained by preprocessing into a preset fusion analysis model. The fusion analysis model includes parallel convolutional neural network branches and recurrent neural network branches. The convolutional neural network branch is responsible for extracting the spatial features of the data, and the recurrent neural network branch is used to extract the temporal features. The two work together to generate a fusion feature vector.

[0028] Obtain the target feature vector: Use a preset dynamic weight allocation algorithm to perform weighted integration on the fused feature vector. This algorithm adjusts the weight coefficients of each branch feature according to the data quality and real-time update frequency of the multimodal data, and finally obtains the target feature vector, making the analysis result more targeted and reliable.

[0029] Generate real-time analysis results: Input the target feature vector into a preset real-time feedback model. This model is constructed based on a streaming data processing architecture and can dynamically adjust the feature extraction strategy according to the current event status, thereby generating real-time analysis results to support the real-time evaluation and decision-making of the event.

[0030] Label mapping: Use a preset classification model to perform label mapping on the real-time analysis results, converting the analysis results into a label form that is easy to understand and apply, facilitating subsequent use and processing.

[0031] The present invention will be further described below in conjunction with Embodiments 1 to 5: Embodiment 1:

[0032] In this embodiment, the specific processes of obtaining real-time multimodal data and multimodal data preprocessing are further elaborated. Specifically, when obtaining real-time multimodal data, a panoramic video stream data and local close-up video stream data of the event venue are captured by a video acquisition device at a first sampling rate. The video acquisition device here can be multiple high-definition cameras distributed at different positions of the event venue to ensure that the entire event scene can be comprehensively captured. The panoramic video stream data is used to show the situation of the entire event venue, enabling the audience and analysts to understand the overall situation of the event; the local close-up video stream data focuses on the key actions, expressions, etc. of the athletes, providing materials for in-depth analysis of the athletes' performances.

[0033] Audio data related to the motion state is collected through audio sensors deployed on the athletes' equipment, such as the breathing sounds, footsteps, shouts, etc. of the athletes. These audio data can reflect information such as the athletes' exercise intensity and emotional state. At the same time, motion sensors collect acceleration, angular velocity, and position coordinate data at a second sampling rate to accurately record the athletes' movement trajectories on the field. The motion sensors can be inertial measurement units (IMUs), which are small, lightweight, and can accurately obtain various motion parameters in real time.

[0034] Real-time comment data and interaction request data input by the audience are received through the user terminal, and the comment data is converted into a text feature vector. The user terminal can be devices such as the audience's mobile phones, tablets, etc. The audience can comment at any time while watching the event, and these comments are transformed into text feature vectors that can be understood and processed by a computer through natural language processing technology.

[0035] When preprocessing multi-modal data, key frames are extracted from panoramic video stream data and local close-up video stream data, and motion vectors between adjacent frames are calculated based on the optical flow method. Key frame extraction can reduce the data volume and improve the processing efficiency, and representative frames are selected for subsequent analysis. The calculation formula of the optical flow method is: ; Among them, represents the optical flow energy function, which is used to measure the error of optical flow estimation; is the image pixel coordinate; is the pixel intensity at the time coordinate ; is the optical flow vector, indicating the displacement of the pixel point from time to time; is the regularization parameter, which is used to balance the weights of the optical flow smoothing term and the data term, and prevent the optical flow estimation from being overly smoothed and losing detailed information. By calculating the motion vector, the motion direction and speed of athletes and objects in the video can be analyzed.

[0036] The audio stream data is subjected to frame-by-frame windowing processing, and acoustic features are extracted through Mel Frequency Cepstral Coefficients (MFCC). Frame-by-frame windowing processing is to divide the continuous audio signal into short frames and apply a window function to reduce spectral leakage. During the MFCC feature extraction process, the audio signal is first converted to the Mel frequency scale, and then the discrete cosine transform is performed to obtain the MFCC coefficients. The Mel frequency calculation formula is: ; Among them, represents the Mel frequency, represents the actual frequency. These MFCC coefficients can effectively characterize the features of the audio and are used for subsequent analysis of the vocal features of athletes, the on-site atmosphere, etc.

[0037] The motion trajectory data is processed by Kalman filtering to eliminate measurement noise and complete missing coordinate points. Kalman filtering is an optimal linear recursive filtering algorithm, and through two steps of prediction and update, the estimation of the motion trajectory is continuously optimized.

[0038] Example 2: The training process of the fusion analysis model includes: Obtain a historical multi-modal data set, and the sample data in this data set are all labeled with corresponding fusion feature labels. These historical data are obtained by collecting and organizing the multi-modal data of previous sports events, and the fusion feature labels are determined based on professional analysis and judgment, representing the characteristic results that each sample data should have.

[0039] The historical multimodal dataset is divided into multiple training subsets based on a sliding time window and sequentially input into the initial fusion model for iterative training in chronological order. The size of the sliding time window can be adjusted according to the characteristics of the event and the properties of the data. For example, for events with a faster pace, the window can be set smaller to more accurately capture data changes; for events with a slower pace, the window can be appropriately increased. After each iteration, the model parameters are adjusted according to the difference rate between the predicted features and the fusion feature labels until the average value of the difference rate is less than or equal to a preset threshold. Multiple methods can be used to calculate the difference rate, such as the mean squared error (MSE), etc. The formula for the mean squared error is: ; where, is the number of samples, is the true label (i.e., the fusion feature label) of the -th sample, is the -th sample's predicted features. By continuously adjusting the model parameters, the prediction results of the model are made closer to the true labels.

[0040] During the training process, the scale of the training subset for each iteration is dynamically reduced according to the difference rate of the previous iteration. Specifically, it includes calculating the difference rate of each sample in the previous iteration, sorting them in ascending order of the difference rate, and retaining the samples with a difference rate lower than the median to form a new training subset. The purpose of this is to gradually focus on the samples with relatively poor model prediction effects during the training process, improving the training efficiency and the accuracy of the model. For example, in a certain iteration, there are 100 samples. After calculating the difference rate of each sample and sorting them, the median (the difference rate of the 50 - th sample) is taken as the threshold, and 50 samples with a difference rate lower than this threshold are retained to form a new training subset for the next iteration of training.

[0041] In each iteration, the branch contribution degree is calculated based on the output features of the convolutional neural network branch and the recurrent neural network branch. The calculation method of the branch contribution degree is: calculate the first cosine similarity between the spatial features output by the convolutional neural network branch and the fusion feature labels, and the second cosine similarity between the temporal features output by the recurrent neural network branch and the fusion feature labels respectively; take the ratio of the first cosine similarity to the second cosine similarity as the branch contribution degree. The formula for the cosine similarity is: ; where, and respectively represent two vectors. In this scenario, can be the spatial feature vector output by the convolutional neural network branch, is the fusion feature label vector (when calculating the first cosine similarity); or is the temporal feature vector output by the recurrent neural network branch, is the fused feature label vector (when calculating the second cosine similarity). and are the norms of vectors and respectively, is the cosine similarity between the two vectors. By calculating the branch contribution degree, we can understand the importance of each branch in the model, and then dynamically adjust the number of network layers and the number of neurons of each branch in the initial fusion model based on the branch contribution degree. If the contribution degree of the convolutional neural network branch is high, it means that the spatial features have a greater impact on the model, and the number of network layers or the number of neurons of the convolutional neural network branch can be appropriately increased to better extract the spatial features; on the contrary, if the contribution degree of the recurrent neural network branch is high, the corresponding adjustment is made to the recurrent neural network branch.

[0042] Example 3: The dynamic weight allocation algorithm adjusts the weight coefficients of the features of each branch according to the data quality and real-time update frequency of the multimodal data to obtain a more reasonable target feature vector.

[0043] Generate a quality evaluation coefficient according to the image clarity of the video stream data, the signal-to-noise ratio of the audio stream data, and the sampling integrity of the motion trajectory data. For the image clarity of the video stream data, it can be measured by calculating the gradient magnitude of the image. Assume the image , and its gradient magnitude calculation formula is: ; where is the gradient magnitude at the coordinate , and are the partial derivatives of the image in the and directions respectively. By statistically analyzing the gradient magnitude of the entire image, a value representing the image clarity is obtained.

[0044] For the signal-to-noise ratio of the audio stream data, its calculation formula is: ; where is the original pure audio signal, is the audio signal containing noise, represents the serial number of the sampling point. The higher the signal-to-noise ratio, the greater the intensity of the useful signal relative to the noise in the audio signal, and the better the audio quality.

[0045] For the sampling integrity of motion trajectory data, it can be measured by calculating the ratio of the actual number of sampled points to the theoretical number of sampled points. The image clarity value of the video stream data, the signal-to-noise ratio value of the audio stream data, and the sampling integrity ratio of the motion trajectory data are comprehensively calculated to obtain a quality evaluation coefficient. For example, the weighted summation method can be used, and corresponding weights are set according to the importance of different data types. Assume the weight of the video stream data is , the weight of the audio stream data is , the weight of the motion trajectory data is , and . The formula for calculating the quality evaluation coefficient is: ; Determine the initial weights of each branch feature based on the product of the quality evaluation coefficient and the preset weight reference value. The preset weight reference value is a basic weight value determined based on experience and previous experiments, and different branches can have different reference values. For example, the preset weight reference value of the convolutional neural network branch is , the preset weight reference value of the recurrent neural network branch is , then the initial weight of the convolutional neural network branch , the initial weight of the recurrent neural network branch .

[0046] Exponentially decay and adjust the initial weights according to the real-time update frequency to obtain the final weight coefficients. Assume the real-time update frequency of the video stream data is , the real-time update frequency of the audio stream data is , the real-time update frequency of the motion trajectory data is , and the preset decay factor is . Taking the convolutional neural network branch as an example, the formula for calculating its final weight coefficient is: ; The formula for calculating the final weight coefficient of the recurrent neural network branch is: ; In this way, data with a high update frequency will get relatively higher weights in weight allocation, more timely reflecting the real-time changes of the event.

[0047] Example 4: The real-time feedback model is constructed based on the streaming data processing architecture and can dynamically adjust the feature extraction strategy according to the current event status.

[0048] Monitor the key event trigger signals of the current event stage. If a goal, foul, or substitution event is detected, increase the feature extraction dimensions of the corresponding modal data. For example, when a goal event is detected, for video stream data, in addition to the regular extraction of spatial and motion features, the feature extraction dimensions of the expressions and action details of players at the moment of scoring can also be increased; for audio stream data, the extraction dimensions of the intensity, frequency, etc. of the cheers of the on-site audience can be increased; for motion trajectory data, the extraction dimensions of the changes in the motion trajectories of the scoring player and related players before and after the goal can be increased. By increasing these feature extraction dimensions, key events can be analyzed more comprehensively and deeply, providing richer information for the real-time evaluation of the event.

[0049] Dynamically expand or compress the number of computing nodes in the streaming data processing architecture according to the request frequency of the user terminal. When the request frequency of the user terminal is high, it indicates that the user has a high degree of attention to the event and has a more urgent need for real-time analysis results. At this time, increase the number of computing nodes to improve the data processing speed and generate real-time analysis results faster; when the request frequency is low, appropriately reduce the number of computing nodes to save computing resources. Assume that the current number of computing nodes is , the request frequency of the user terminal is , the preset expansion threshold is , and the compression threshold is . When , the number of computing nodes is adjusted to ( is the number of added nodes, which can be set according to the actual situation); when , the number of computing nodes is adjusted to ( is the number of reduced nodes, which can also be set according to the actual situation).

[0050] The label mapping rule update method of the classification model is as follows: Extract the corrected labels and associated context features of the user terminal for the real-time analysis results. When the user views the real-time analysis results and finds that the results do not match the actual situation, the user can correct the labels. At the same time, the system records these corrected labels and the associated context features of the corrections, such as the event time, score situation, and on-field player lineup at that time.

[0051] Form an incremental training set with the corrected labels and context features, and update the decision boundary of the classification model through an online learning algorithm. The implementation of the online learning algorithm includes: calculating the distance between the feature vectors of each sample in the incremental training set and the hyperplane of the current classification model. Assume that the hyperplane equation of the classification model is , where is the normal vector of the hyperplane, is the feature vector of the sample, is the bias term. The sample feature vector The distance calculation formula from the hyperplane is: ; where, is the distance from the feature vector of sample to the hyperplane, is the normal vector modulus. If the distance is less than the preset fault tolerance threshold, the original decision boundary is retained; otherwise, the hyperplane parameters are adjusted through the support vector machine incremental training algorithm. The support vector machine incremental training algorithm trains on the basis of the original model using the new incremental training set to adjust the hyperplane parameters and , so that the classification model can better adapt to new data and improve the accuracy of label mapping.

[0052] Example 5: In this embodiment, the process of pushing the result to the user terminal after generating the real-time analysis result and updating the label mapping rule of the classification model according to the user terminal interaction data is elaborated in detail.

[0053] The real-time analysis result is pushed to the user terminal through a preset feedback interface. The feedback interface can be implemented by a variety of technologies, such as the WebSocket protocol, which can establish a real-time two-way communication channel between the client (user terminal) and the server. The server encapsulates the generated real-time analysis result into a specific data format and sends it to the user terminal through the WebSocket connection. After receiving the data, the user terminal presents the real-time analysis result to the user in an intuitive manner according to its own interface display logic, such as displaying the real-time status evaluation of players and the prediction of the event trend on the event live broadcast interface.

[0054] Update the label mapping rule of the classification model according to the interaction data of the user terminal. During the use process, the user may perform interaction operations such as liking, commenting, and sharing on the real-time analysis result. These interaction data contain the feedback information of the user on the analysis result. For example, if a large number of users give negative comments on a certain real-time analysis result, it means that the result may be inaccurate and the label mapping rule of the classification model needs to be updated.

[0055] The specific update process is as follows: First, collect the interaction data of the user terminal, analyze and filter this data, and extract the data related to the classification model label mapping. For example, extract the keywords related to label accuracy in the user comments, count the number of likes and comments under different labels, etc. Then, according to the extracted data, in accordance with the classification model label mapping rule update method in Embodiment 4, convert the relevant data into corrected labels and associated context features, form an incremental training set, and update the decision boundary of the classification model through an online learning algorithm, thereby continuously optimizing the label mapping rule of the classification model and improving the accuracy and practicality of the real-time analysis results. During the whole process, continuously monitor the interaction data of the user terminal, and update the label mapping rule regularly or in real time to ensure that the classification model can adapt to the changes in the event data and the needs of users in a timely manner.

[0056] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device.

[0057] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A sports event multimodal data fusion analysis method based on AI real-time feedback, characterized in that: include: Acquire real-time multimodal data from at least one data source, wherein the data source includes a video acquisition device, an audio sensor, a motion sensor, and a user terminal, and the multimodal data includes video stream data, audio stream data, motion trajectory data, and user interaction data; Preprocessing the multimodal data to eliminate noise and extract initial features; Inputting the initial features into a preset fusion analysis model to generate a fusion feature vector, wherein the fusion analysis model includes a parallel convolutional neural network branch and a recursive neural network branch, the convolutional neural network branch is used to extract spatial features, and the recursive neural network branch is used to extract temporal features; The fused feature vector is weighted and integrated based on a preset dynamic weight allocation algorithm to obtain a target feature vector, wherein the dynamic weight allocation algorithm adjusts the weight coefficient of each branch feature according to the data quality and real-time update frequency of the multimodal data; Inputting the target feature vector into a preset real-time feedback model to generate real-time analysis results, wherein the real-time feedback model is built based on a streaming data processing architecture and is used to dynamically adjust the feature extraction strategy according to the current event status; The real-time analysis results are labeled using a preset classification model.

2. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: The method further includes training the fusion analysis model in the following manner: Acquire a historical multimodal data set, wherein sample data in the historical multimodal data set is annotated with corresponding fusion feature labels; divide the historical multimodal data set into a plurality of training subsets based on a sliding time window, and sequentially input the subsets into an initial fusion model in chronological order for iterative training, and adjust the model parameters according to the difference rate between the predicted features and the fusion feature labels after each iteration until the average value of the difference rate is less than or equal to a preset threshold; During the training process, the size of the training subset of each iteration is dynamically reduced according to the difference rate of the previous iteration, specifically including: calculating the difference rate of each sample in the previous iteration, arranging them in ascending order of difference rate, and retaining samples with difference rate lower than the middle value to form a new training subset.

3. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: Acquiring real-time multimodal data from at least one data source includes: Capturing panoramic video stream data and local close-up video stream data of the competition venue at a first sampling rate through a video acquisition device; collecting audio data associated with the motion state through an audio sensor deployed on the athlete's equipment, and collecting acceleration, angular velocity and position coordinate data at a second sampling rate through a motion sensor; The real-time comment data and interaction request data input by the audience are received through the user terminal, and the comment data are converted into a text feature vector.

4. The sports event multimodal data fusion analysis method according to claim 3, characterized in that: Preprocessing the multimodal data further includes: Extract key frames from the panoramic video stream data and the local close-up video stream data, and calculate motion vectors between adjacent frames based on an optical flow method; Performing frame and window processing on the audio stream data, and extracting acoustic features through Mel-frequency cepstral coefficients; The motion trajectory data is processed by Kalman filtering to eliminate measurement noise and complete missing coordinate points.

5. The sports event multimodal data fusion analysis method according to claim 2, characterized in that: The training process of the fusion analysis model also includes: In each iteration, calculating branch contributions according to output features of the convolutional neural network branch and the recursive neural network branch; The number of network layers and the number of neurons of each branch in the initial fusion model are dynamically adjusted based on the branch contribution.

6. The sports event multimodal data fusion analysis method according to claim 5, characterized in that: The branch contribution is calculated as follows: Respectively calculating the first cosine similarity between the spatial feature output by the convolutional neural network branch and the fused feature label, and the second cosine similarity between the temporal feature output by the recursive neural network branch and the fused feature label; The ratio of the first cosine similarity to the second cosine similarity is used as the branch contribution.

7. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: The implementation of the dynamic weight allocation algorithm includes: Generate a quality assessment coefficient according to the image clarity of the video stream data, the signal-to-noise ratio of the audio stream data, and the sampling integrity of the motion trajectory data; Determine the initial weight of each branch feature based on the product of the quality assessment coefficient and a preset weight reference value; The initial weight is adjusted by exponential decay according to the real-time update frequency to obtain a final weight coefficient.

8. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: The dynamic adjustment strategy of the real-time feedback model includes: Monitor the key event trigger signals of the current stage of the game. If a goal, foul or substitution event is detected, increase the feature extraction dimension of the corresponding modal data; The number of computing nodes of the streaming data processing architecture is dynamically expanded or compressed according to the request frequency of the user terminal.

9. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: The label mapping rule updating method of the classification model is as follows: Extracting a correction label and associated context features of the user terminal for the real-time analysis result; The modified labels and the context features are combined into an incremental training set, and the decision boundary of the classification model is updated through an online learning algorithm; The implementation of the online learning algorithm includes: Calculate the distance between the feature vector of each sample in the incremental training set and the hyperplane of the current classification model; If the distance is less than the preset tolerance threshold, the original decision boundary is retained; otherwise, the hyperplane parameters are adjusted through the support vector machine incremental training algorithm.

10. The sports event multimodal data fusion analysis method according to claim 1, characterized in that: After generating the real-time analysis result, the method further includes: pushing the real-time analysis result to a user terminal through a preset feedback interface, and updating the label mapping rule of the classification model according to the interaction data of the user terminal.

Images