AI highlight identification method and system for live event broadcasting

Abstract

This invention provides an AI-powered method and system for identifying highlights in live sports broadcasts, relating to the field of live sports broadcasting technology. The method includes acquiring the live sports stream, separating the commentary audio and visual video channels and extracting acoustic feature tensors and visual semantic tensors respectively, and constructing a joint representation space after cross-modal alignment. Clustering is performed in this space to construct a state transition topology graph, and state abrupt changes are identified by recognizing unconventional transition edges and cooperative jumps. Based on these state abrupt changes, the rhythmic cycle and the boundaries of the narrative unit are traced to define the time interval of the highlights and ultimately extract the corresponding segments. This achieves automated and accurate identification and extraction of highlights from live sports broadcasts.

Description

Technical Field

[0001] This invention relates to the field of sports event live streaming technology, and in particular to an AI-powered method and system for identifying highlights in sports event live streaming. Background Technology

[0002] In the field of live sports broadcasting, automatically identifying and editing highlight moments is a key technology for improving content dissemination efficiency and viewer experience. Existing solutions typically rely on the analysis of single-modal information within the live stream. For example, analyzing the acoustic characteristics of the commentary audio, such as volume, speech rate, or the appearance of specific keywords, can help infer moments when exciting events are likely to occur. Another common approach focuses on visual image analysis, such as detecting specific celebrations by athletes, sudden bursts of excitement in the stands, or using computer vision algorithms to track the trajectory and speed changes of a ball as markers of exciting moments.

[0003] These conventional methods have significant limitations. Relying on single-modal information for judgment often results in insufficient reliability and accuracy. For example, a commentator's increased pitch or faster speech might be an exaggeration of a routine attack rather than a genuine goal or crucial save; conversely, a critical tactical execution might be accompanied by the commentator's calm analysis, resulting in no significant fluctuations in audio characteristics. Similarly, relying solely on visual celebrations might miss the setup for a brilliant pass before a goal or misjudge routine interruptions like throw-ins as exciting events. The flaw in this single-modal analysis lies in severing the natural, inherent connection and complementarity between sound and visuals in the narrative of a match, making it prone to false alarms or missed detections.

[0004] Furthermore, even when attempting to combine audio and visual information, existing methods often rely on simple rule overlays or threshold judgments, such as checking whether audio energy exceeds a threshold while detecting visual motion events. This approach fails to delve into the complex, temporal dynamic relationships and collaborative changes between the two modalities. The essence of exciting moments in a sporting event is often the synchronous abrupt change in audio-visual rhythm and the concentrated burst of narrative logic. Simple feature overlays cannot model this cross-modal, event-context-based state transition process, leading to inaccurate determination of the start and end boundaries of exciting segments, potentially resulting in incomplete narrative units or redundant content. Summary of the Invention

[0005] This invention provides an AI-powered method and system for identifying highlights in live sports broadcasts, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides an AI-powered method for identifying highlights in live sports broadcasts, comprising:

[0007] Obtain the live stream of the event;

[0008] The audio channel for commentary and the visual channel for video are separated from the live broadcast of the event. The acoustic feature tensor is obtained by extracting the fundamental frequency of the audio channel and performing sound spectrum analysis. The visual semantic tensor is obtained by performing scene semantic segmentation and target motion trajectory tracking on the visual channel. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space.

[0009] In the joint representation space, acoustic feature tensors and visual semantic tensors are clustered to obtain competitive state nodes, and a competitive state transition topology graph is constructed. Unconventional transition edges are identified based on the weight of the outgoing edges, and state change markers are determined by the collaborative jump of prosodic periodic sequences and motion trajectory sequences.

[0010] Starting from the moment corresponding to the state mutation marker, the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment.

[0011] Extract relevant segments from the live event stream based on the time interval of the highlights.

[0012] In one optional embodiment, the acoustic feature tensor is obtained by performing acoustic fundamental frequency extraction and chromatographic analysis on the narration audio channel, including:

[0013] The fundamental frequency of the audio channel is extracted, the periodic repetition pattern of the audio signal is identified by autocorrelation operation, the time interval between adjacent peaks is calculated, and the reciprocal of the time interval is taken to obtain the fundamental frequency numerical sequence.

[0014] The audio channel of the narration is subjected to chromatographic analysis. The audio signal is divided into frames according to a preset time length. Fourier transform is performed on each audio frame to obtain a frequency domain representation. The amplitude of each frequency component in the frequency domain representation is extracted. The frequency amplitudes of consecutive audio frames are arranged in time order to form a chromatographic matrix.

[0015] A composite acoustic tensor is formed by tensor concatenation of the fundamental frequency numerical sequence and the chromatogram matrix. The composite acoustic tensor is then input into a three-layer convolutional neural network for spatial feature extraction. The first convolutional layer extracts the temporal variation pattern of the fundamental frequency numerical sequence, the second convolutional layer extracts the frequency energy distribution pattern of the chromatogram matrix, and the third convolutional layer fuses the temporal variation pattern and the frequency energy distribution pattern to output a feature map. The feature map is then flattened into a one-dimensional vector after global average pooling to obtain the acoustic feature tensor.

[0016] In one optional embodiment, obtaining the visual semantic tensor by performing scene semantic segmentation and target motion trajectory tracking on the visual image channel includes:

[0017] Scene semantic segmentation is performed on the visual image channel. The video frame is input into the semantic segmentation network to predict the semantic category label of each pixel. Spatially connected pixels with the same semantic category label are aggregated into semantic regions, and the boundary coordinates and category labels of the semantic regions are extracted.

[0018] The target motion trajectory is tracked in the visual image channel. The geometric center coordinates of the semantic region in the current frame are calculated to determine the current position point. In the next frame, all candidate semantic regions with the same category identifier as the current frame are searched. The Euclidean distance between the geometric center coordinates of each candidate semantic region and the current position point is calculated. The candidate semantic region with the smallest Euclidean distance is selected as the matching target. The geometric center coordinates of the successfully matched frames are concatenated in time order to form a trajectory coordinate sequence.

[0019] The category identifiers of the semantic region are one-hot encoded to obtain category vectors. The category vectors are concatenated with the trajectory coordinate sequence in time dimension to form a temporal feature sequence. The temporal feature sequence is input into a recurrent neural network to model temporal dependencies. The hidden state vector of the recurrent neural network at the last moment is extracted to obtain the visual semantic tensor.

[0020] In one optional embodiment, acoustic feature tensors and visual semantic tensors are clustered in a joint representation space to obtain competition state nodes, a competition state transition topology graph is constructed, unconventional transition edges are identified based on the weight of outgoing edges, and state change markers are determined through the collaborative jump of prosodic periodic sequences and motion trajectory sequences, including:

[0021] In the joint representation space, the acoustic feature tensor and the visual semantic tensor are clustered to obtain multiple clusters. The centroid of the cluster is calculated as the cluster center, and the cluster center is defined as the competition state node.

[0022] Traverse the time series to extract the competitive state nodes corresponding to adjacent time points, count the frequency of transition from the competitive state node of the previous time point to the competitive state node of the next time point, determine the transition probability, and construct a competitive state transition topology graph by taking the competitive state node as a vertex, the transition relationship between competitive state nodes as a directed edge, and the transition probability as the edge weight.

[0023] For each competitive state node in the competitive state transition topology graph, extract all outgoing edge weights of the competitive state node within the historical time window, calculate the statistical distribution characteristics of the outgoing edge weights, determine the weight boundary value, and mark outgoing edges whose weights are below the weight boundary value as non-standard transition edges.

[0024] Obtain the acoustic feature tensor and visual semantic tensor of the time corresponding to the unconventional transition edge in the joint representation space, extract the prosodic periodic sequence from the acoustic feature tensor, extract the motion trajectory sequence from the visual semantic tensor, and mark the time corresponding to the unconventional transition edge as a state change marker when the prosodic periodic sequence and the motion trajectory sequence undergo a cooperative jump.

[0025] In an optional embodiment, when the prosodic periodic sequence and the motion trajectory sequence undergo a coordinated transition, marking the time corresponding to the unconventional transition edge as a state change marker includes:

[0026] Period length jump detection is performed on the prosodic periodic sequence. The period length value within a continuous time window in the prosodic periodic sequence is extracted. The difference between adjacent period length values ​​is calculated to form a period length change curve. Local extreme points of change amplitude are identified on the period length change curve, and the corresponding time is marked as a candidate time of prosodic jump.

[0027] Trajectory density jump detection is performed on the motion trajectory sequence. The number of trajectory points in the motion trajectory sequence is counted within a fixed time window to form a trajectory density time series. The density difference between adjacent windows of the trajectory density time series is calculated to form a trajectory density change curve. Local extreme points of change amplitude are identified on the trajectory density change curve, and the corresponding time is marked as a candidate time for trajectory jump.

[0028] Calculate the time difference between the candidate times of rhythmic transition and the candidate times of trajectory transition. When the time difference is within the preset synchronization time window, confirm that the rhythmic periodic sequence and the motion trajectory sequence have undergone a coordinated transition, and mark the time corresponding to the unconventional transition edge as a state change marker.

[0029] In one optional embodiment, starting from the moment corresponding to the state change marker, the prosodic rhythmic period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythmic period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the highlight segment, including:

[0030] Using the moment corresponding to the state change marker as the starting point for tracing back, the acoustic feature tensor sequence and visual semantic tensor sequence prior to that moment are extracted in reverse from the joint representation space.

[0031] For each acoustic feature tensor, extract the spectral energy component, connect the spectral energy components in the time dimension to form a prosodic energy envelope curve, calculate the energy difference between adjacent moments on the prosodic energy envelope curve, mark the transition position when the energy difference changes from negative to positive, and use the transition position as the starting boundary of the prosodic rhythm cycle.

[0032] For each visual semantic tensor, its scene semantic representation is extracted as a scene semantic vector. Cosine similarity is calculated for the scene semantic vectors of adjacent time moments. When the cosine similarity is lower than a preset similarity threshold, the corresponding position of the adjacent time moment is marked as the dividing point of the shot narrative unit.

[0033] Traverse the subsequent time series forward along the time axis from the moment corresponding to the state change marker, detect the dividing point of the next appearing shot narrative unit in the subsequent time series as the end boundary of the shot narrative unit, extract the timestamp corresponding to the starting boundary of the rhythmic cycle and the timestamp corresponding to the ending boundary of the shot narrative unit, and determine the time interval of the exciting segment.

[0034] In one optional embodiment, extracting the timestamp corresponding to the starting boundary of the rhythmic cycle and the timestamp corresponding to the ending boundary of the shot narrative unit to determine the time interval of the highlight segment includes:

[0035] The initial starting point is the timestamp corresponding to the boundary of the rhythmic cycle, and the initial ending point is the timestamp corresponding to the boundary of the narrative unit of the shot.

[0036] Multiple candidate start times are sampled within preset adjustment ranges before and after the initial start point, and multiple candidate end times are sampled within preset adjustment ranges before and after the initial end point. For each candidate time period defined by a pair of candidate start times and candidate end times, the acoustic feature tensor sequence and visual semantic tensor sequence within the candidate time period are extracted. The mutual information value between the acoustic feature tensor sequence and the visual semantic tensor sequence is calculated. The mutual information value is obtained by calculating the difference between the corresponding joint distribution entropy and their respective marginal distribution entropy. The candidate time period with the largest mutual information value is selected to determine the time interval of the exciting segment.

[0037] A second aspect of this invention provides an AI-powered highlight recognition system for live sports broadcasts, comprising:

[0038] The live stream acquisition unit is used to acquire the live stream of the event;

[0039] The cross-modal alignment unit is used to separate the commentary audio channel and the visual video channel from the live broadcast of the event. The commentary audio channel is subjected to acoustic fundamental frequency extraction and sound spectrum analysis to obtain an acoustic feature tensor. The visual video channel is subjected to scene semantic segmentation and target motion trajectory tracking to obtain a visual semantic tensor. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space.

[0040] The state mutation unit is used to cluster acoustic feature tensors and visual semantic tensors in the joint representation space to obtain competitive state nodes, construct a competitive state transition topology graph, identify unconventional transition edges based on the weight of the outgoing edge, and determine the state mutation marker through the collaborative jump of the prosodic period sequence and the motion trajectory sequence.

[0041] The segment boundary unit is used to trace the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor in the joint representation space, starting from the moment corresponding to the state change marker. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment.

[0042] The segment extraction unit is used to extract corresponding segments from the live event stream based on the time interval of the highlights.

[0043] A third aspect of the present invention provides an electronic device, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0048] In this embodiment of the invention, by separating the narration audio and visual images and extracting acoustic features and visual semantics respectively, a cross-modal aligned joint representation space is constructed, providing a unified and information-rich underlying data foundation for analysis. In the joint representation space, by clustering acoustic and visual features and constructing a state transition topology graph, the dynamic process and state evolution of the event can be effectively depicted. Based on unconventional transition edges and cross-modal collaborative jumps to identify state mutations, significant changes caused by scoring, key confrontations, or turning points in the course of the game can be keenly captured. Starting from the identified state mutation points, the rhythmic cycle and the boundaries of the narrative unit of the shot are traced and determined, thereby accurately defining the time interval of the exciting segments. Associating instantaneous state mutations with time periods with complete narrative structures (such as an offensive round or a scoring process) ensures that the extracted segments not only contain climaxes but also have relatively complete context and narrative logic, enhancing the viewing value and information integrity of the segments. Attached Figure Description

[0049] Figure 1A flowchart illustrating an AI-powered method for identifying highlights in live sports broadcasts;

[0050] Figure 2 The logical flow for analyzing sudden changes in competitive state. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0053] Figure 1 This is a flowchart illustrating the AI-powered highlight recognition method for live sports broadcasts according to an embodiment of the present invention. Figure 1 As shown, the AI-powered highlight recognition methods used for live sports broadcasts include:

[0054] Obtain the live stream of the event;

[0055] The audio channel for commentary and the visual channel for video are separated from the live broadcast of the event. The acoustic feature tensor is obtained by extracting the fundamental frequency of the audio channel and performing sound spectrum analysis. The visual semantic tensor is obtained by performing scene semantic segmentation and target motion trajectory tracking on the visual channel. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space.

[0056] In the joint representation space, acoustic feature tensors and visual semantic tensors are clustered to obtain competitive state nodes, and a competitive state transition topology graph is constructed. Unconventional transition edges are identified based on the weight of the outgoing edges, and state change markers are determined by the collaborative jump of prosodic periodic sequences and motion trajectory sequences.

[0057] Starting from the moment corresponding to the state mutation marker, the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment.

[0058] Extract relevant segments from the live event stream based on the time interval of the highlights.

[0059] In one optional embodiment, the acoustic feature tensor is obtained by performing acoustic fundamental frequency extraction and chromatographic analysis on the narration audio channel, including:

[0060] The fundamental frequency of the audio channel is extracted, the periodic repetition pattern of the audio signal is identified by autocorrelation operation, the time interval between adjacent peaks is calculated, and the reciprocal of the time interval is taken to obtain the fundamental frequency numerical sequence.

[0061] The audio channel of the narration is subjected to chromatographic analysis. The audio signal is divided into frames according to a preset time length. Fourier transform is performed on each audio frame to obtain a frequency domain representation. The amplitude of each frequency component in the frequency domain representation is extracted. The frequency amplitudes of consecutive audio frames are arranged in time order to form a chromatographic matrix.

[0062] A composite acoustic tensor is formed by tensor concatenation of the fundamental frequency numerical sequence and the chromatogram matrix. The composite acoustic tensor is then input into a three-layer convolutional neural network for spatial feature extraction. The first convolutional layer extracts the temporal variation pattern of the fundamental frequency numerical sequence, the second convolutional layer extracts the frequency energy distribution pattern of the chromatogram matrix, and the third convolutional layer fuses the temporal variation pattern and the frequency energy distribution pattern to output a feature map. The feature map is then flattened into a one-dimensional vector after global average pooling to obtain the acoustic feature tensor.

[0063] In one specific implementation, when extracting the acoustic fundamental frequency from the narration audio channel, the input audio signal is first preprocessed, with the sampling rate uniformly set to 44.1kHz or 48kHz to ensure that the time resolution of subsequent analysis meets the requirements for capturing speech prosodic features. The preprocessed audio signal is then subjected to periodic detection using an autocorrelation function. Autocorrelation calculations, by correlating the signal with its own time-delayed version, effectively identify periodic repetition patterns in the audio signal. Specifically, for discrete audio signal sequences, the correlation is calculated at different time delays. When the delay time is exactly equal to the signal's fundamental period, the autocorrelation function exhibits a significant peak. By searching for local maxima in the autocorrelation results, the positions of adjacent peaks can be located; these peaks correspond to the periodic characteristics of vocal cord vibration. When calculating the time interval between adjacent peaks, a reasonable search window needs to be set, typically between 2ms and 20ms, to cover the fundamental frequency range of human speech from 80Hz to 500Hz. By taking the reciprocal of each identified time interval, the periodic information in the time domain can be converted into the fundamental frequency value in the frequency domain, forming a fundamental frequency value sequence that varies with time. The length of this sequence is proportional to the duration of the audio signal. A typical frame shift is set to 10ms, which means 100 fundamental frequency values ​​are generated per second.

[0064] When performing audio spectrum analysis on the narration audio channel, short-time Fourier transform (SFT) technology is used to perform time-frequency joint analysis of the spectral characteristics of the audio signal. The continuous audio signal is processed into frames according to a preset time length, with the time window length of each frame typically set to 25ms to 50ms. There is a 50% to 75% overlap between adjacent frames to ensure the continuity and smoothness of the time-frequency analysis. A Hanning window or Hamming window function is applied to each audio frame to reduce spectral leakage effects caused by signal truncation. A fast Fourier transform (FFT) operation is performed on the windowed audio frames to convert the time-domain signal into a frequency-domain representation, obtaining a complex spectrum containing amplitude and phase information. The amplitude information of each frequency component in the frequency-domain representation is extracted, typically focusing on the frequency range of 20Hz to 8000Hz, which covers the main energy distribution area of ​​human speech. The frequency amplitudes of the continuous audio frames are arranged in chronological order, with the time dimension corresponding to the frame sequence index and the frequency dimension corresponding to discrete frequency sampling points, forming a two-dimensional audio spectrum matrix. The number of rows in the matrix equals the number of frequency sampling points, and the number of columns equals the total number of audio frames. Each element in the matrix represents the energy intensity of a specific frequency at a specific moment. Visual contrast is enhanced through logarithmic transformation or normalization.

[0065] When constructing the acoustic feature tensor, it is necessary to effectively fuse the fundamental frequency numerical sequence with the acoustic spectrum matrix. Since the fundamental frequency numerical sequence is a one-dimensional time series, while the acoustic spectrum matrix is ​​a two-dimensional time-frequency matrix, their dimensional structures differ. The fundamental frequency numerical sequence is dimensionally expanded by copying and extending it to the same time length as the acoustic spectrum matrix, forming a two-dimensional tensor aligned with the time dimension of the acoustic spectrum matrix. Specifically, the fundamental frequency numerical sequence can be broadcast and copied along the frequency dimension, or it can be preserved as an independent channel. Tensor concatenation of the fundamental frequency numerical sequence and the acoustic spectrum matrix can be performed along the channel dimension, forming a composite acoustic tensor with multiple feature channels. The shape of this tensor is the number of channels multiplied by the number of time frames multiplied by the number of frequency points, where the number of channels is at least two, with one channel corresponding to the fundamental frequency information and the other channel corresponding to the acoustic spectrum information.

[0066] When extracting spatial features from a three-layer convolutional neural network with a composite acoustic tensor input, the network's structural design must fully consider the time-frequency characteristics of the acoustic signal. The first convolutional layer specifically handles the fundamental frequency numerical sequence channel, employing a one-dimensional convolutional kernel to perform convolution operations along the time dimension. The kernel length is set to 5 to 15 time frames, enabling it to capture short-term trends and local fluctuation patterns of the fundamental frequency numerical sequence. The number of convolutional kernels in this layer is set to 32 to 64, with each kernel learning different temporal variation patterns, such as prosodic features like fundamental frequency rise, fall, stationary, or rapid oscillations. After the convolution operation, a batch normalization layer and the ReLU activation function are applied to enhance the network's training stability and nonlinear expressive ability. The second convolutional layer specifically handles the chromatogram matrix channel, employing a two-dimensional convolutional kernel to perform convolution operations simultaneously in the time and frequency dimensions. The kernel size is set to 3x3 to 7x7, enabling it to capture the local structure and joint time-frequency patterns of the frequency domain energy distribution. The number of convolutional kernels in this layer is set to 64 to 128 to learn the energy concentration patterns, formant structures, and the evolution of frequency components over time in different frequency regions. Similarly, a batch normalization layer and the ReLU activation function are applied after the convolution operation.

[0067] The third convolutional layer deeply fuses the temporal variation patterns and frequency domain energy distribution patterns extracted by the first two layers. This layer receives feature maps from the first and second convolutional layers and integrates the features from the two modalities through channel concatenation or a feature fusion module. Further feature abstraction is performed using two-dimensional convolutional kernels, with kernel sizes set to 3x3 to 5x5 and the number of kernels set to 128 to 256, learning cross-modal collaborative features and higher-order semantic representations. This layer can capture the correlation patterns between fundamental frequency variations and audiospectral energy distribution; for example, when the narrator's voice is enthusiastic, the fundamental frequency rises significantly and high-frequency energy is enhanced. This cross-modal collaborative variation plays an important indicative role in identifying exciting moments. The third convolutional layer also applies batch normalization layers and the ReLU activation function, outputting the fused feature map.

[0068] A global average pooling operation is performed on the fused feature maps. This operation calculates the average value across the spatial dimension of the feature maps, compressing the two-dimensional feature map of each feature channel into a single value, effectively reducing the number of parameters and extracting global semantic information. The output of global average pooling is a vector of length equal to the number of feature channels, which contains a high-level abstract representation of the acoustic features of the entire audio segment. The vector after global average pooling is flattened to ensure it is a one-dimensional structure, resulting in the final acoustic feature tensor. This acoustic feature tensor typically has 128 to 256 dimensions, enabling it to compactly encode various acoustic attributes of the narration audio, such as rhythm, timbre variation, and emotional intensity, providing high-quality feature representations for subsequent cross-modal alignment and joint analysis with the visual semantic tensor.

[0069] In practice, audio signal preprocessing also includes noise reduction and volume normalization to remove background noise and audience cheers from the narration audio. Spectral subtraction or Wiener filtering techniques are used for noise suppression, preserving the main components of the narrator's voice. For multi-channel audio streams, source separation is required to extract the narrator's voice channel and suppress ambient noise and sound effects. During fundamental frequency extraction, special attention must be paid to silent and noisy segments. An energy threshold is set to determine whether the current frame contains valid speech; frames with energy below the threshold are marked as silent frames, and fundamental frequency calculation is not performed or a default value is assigned. In spectral analysis, the choice of frequency resolution affects the ability to capture spectral details; typically, a 512-point or 1024-point Fourier transform is used to strike a balance between frequency resolution and computational efficiency.

[0070] During the training of a convolutional neural network, a labeled dataset is required, containing audio clips of commentary from different types of events and their corresponding highlight tags. Supervised training is performed using either cross-entropy loss or contrastive learning loss to optimize network parameters so that the acoustic feature tensor can effectively distinguish between highlight clips and ordinary clips. Data augmentation techniques are applied during training, including time stretching, pitch transformation, and adding slight noise, to enhance the model's generalization ability and robustness. After training, the extraction of the acoustic feature tensor can be performed in real time, meeting the low-latency requirements of live event broadcasts and providing fundamental feature support for the rapid identification of highlight clips.

[0071] In one optional embodiment, obtaining the visual semantic tensor by performing scene semantic segmentation and target motion trajectory tracking on the visual image channel includes:

[0072] Scene semantic segmentation is performed on the visual image channel. The video frame is input into the semantic segmentation network to predict the semantic category label of each pixel. Spatially connected pixels with the same semantic category label are aggregated into semantic regions, and the boundary coordinates and category labels of the semantic regions are extracted.

[0073] The target motion trajectory is tracked in the visual image channel. The geometric center coordinates of the semantic region in the current frame are calculated to determine the current position point. In the next frame, all candidate semantic regions with the same category identifier as the current frame are searched. The Euclidean distance between the geometric center coordinates of each candidate semantic region and the current position point is calculated. The candidate semantic region with the smallest Euclidean distance is selected as the matching target. The geometric center coordinates of the successfully matched frames are concatenated in time order to form a trajectory coordinate sequence.

[0074] The category identifiers of the semantic region are one-hot encoded to obtain category vectors. The category vectors are concatenated with the trajectory coordinate sequence in time dimension to form a temporal feature sequence. The temporal feature sequence is input into a recurrent neural network to model temporal dependencies. The hidden state vector of the recurrent neural network at the last moment is extracted to obtain the visual semantic tensor.

[0075] In one specific implementation, for the processing of the visual image channel, a sequence of video frames is extracted from the live event stream. These video frames are sampled at a fixed frame rate, typically 25 or 30 frames per second. Each video frame, after preprocessing, is input into a pre-trained semantic segmentation network. This network is built on a fully convolutional architecture and is capable of predicting the semantic category of each pixel in the input image pixel-by-pixel. The encoder part of the semantic segmentation network extracts multi-scale features of the image through multi-layer convolutional operations, while the decoder part restores the spatial resolution through upsampling operations, ultimately outputting a semantic label map of the same size as the input image.

[0076] After obtaining the semantic label map, a connected component analysis algorithm is used to detect the spatial connectivity of pixels with the same semantic category label. Specifically, for each semantic category, all pixels belonging to that category in the label map are traversed, and spatial connectivity between pixels is determined using four-neighbor or eight-neighbor connection rules. Pixels belonging to the same connected component are aggregated into an independent semantic region. For each identified semantic region, its boundary coordinate information is extracted, and a boundary tracing algorithm is used to record the coordinates of boundary pixels sequentially along the region edge, forming a closed boundary coordinate sequence. Simultaneously, the category identifier corresponding to the semantic region is recorded; the category identifier typically uses integer indices to represent different semantic categories.

[0077] During the target motion trajectory tracking phase, the geometric center coordinates of each semantic region in the current frame are first calculated. For a semantic region defined by a sequence of boundary coordinates, the mean of the x-coordinates and the mean of the y-coordinates of all pixels within that region are calculated and used as the geometric center coordinates of that semantic region. Assume the geometric center coordinates of a certain semantic region in the current frame are... Category identifier is The timestamp of the current frame is In processing the next frame (timestamp is...) When retrieving all category identifiers in the frame, the same applies. The semantic regions, which constitute the candidate matching set.

[0078] For each candidate semantic region in the candidate matching set, calculate its geometric center coordinates. relative to current location The Euclidean distance between them. The formula for calculating this Euclidean distance is: The distance set is calculated by traversing all candidate semantic regions. ,in This represents the total number of candidate regions. The candidate semantic region with the smallest distance value is selected as the matching result for the current target in the next frame. This nearest neighbor-based matching strategy effectively utilizes prior knowledge of the continuity of target motion between adjacent frames.

[0079] When a target is successfully matched in multiple consecutive frames, the geometric center coordinates of the semantic region corresponding to the target in each frame are concatenated in chronological order to form a sequence of the target's motion trajectory coordinates. Assume a player target is at time... arrive Continuous If all frames are successfully tracked, then the sequence of its trajectory coordinates can be represented as follows: This sequence fully records the player's movement path on the field. In cases of matching failure, such as when the target is obscured or out of sight, trajectory interpolation is used to estimate coordinates within the brief period of disappearance, or the trajectory is terminated and a new trajectory is started after the disappearance duration exceeds a set threshold.

[0080] After obtaining the category identifiers and trajectory coordinate sequences of each target, it is necessary to fuse this heterogeneous information into a unified feature representation. First, the category identifiers are one-hot encoded, assuming the total number of semantic categories is... Then the category identifier Encoded as a A binary vector of dimension , where the first is a 2-dimensional vector. One position is set to 1, and the rest are set to 0. This encoding method transforms discrete category identifiers into vector representations suitable for neural network processing. For time step (k), ... A certain target, whose category vector is The position coordinates are .

[0081] Concatenate the category vector and the location coordinates along the feature dimension to obtain a dimension of The mixed feature vectors are obtained by stacking the mixed feature vectors at each time step along the time dimension, since the trajectory coordinate sequence is arranged in chronological order. ,in This temporal feature sequence contains both the semantic category information of the target and the spatial location evolution information of the target over time.

[0082] Temporal feature sequences are input into recurrent neural networks (RNNs) for temporal dependency modeling. RNNs, by passing information between time steps through hidden states, are able to capture long-term dependencies in sequence data. In processing the... At each time step, the recurrent neural network receives the input features of the current moment. And the hidden state of the previous moment. The hidden state at the current time step is obtained by updating the calculation of the loop unit. Recurrent units can employ long short-term memory (LSTM) network units or gated recurrent units. These structures effectively mitigate the vanishing gradient problem through gating mechanisms, enhancing the ability to model long sequences.

[0083] After forward propagation of the entire sequence, the recurrent neural network reaches its final state at the last moment. Output hidden state vector It contains temporal dependency information of the entire trajectory sequence. This hidden state vector comprehensively encodes the target's motion patterns, velocity changes, direction changes, and other dynamic characteristics. For multiple tracked targets in a live sports broadcast, the hidden state vectors corresponding to the last moment of each trajectory are extracted. These vectors are then weighted and aggregated according to the spatial distribution or importance of the targets, or directly concatenated into a high-dimensional vector, ultimately yielding a visual semantic tensor describing the semantic dynamic information of the entire visual image channel. This visual semantic tensor not only contains the category information and spatial layout of each semantic object in the scene, but more importantly, it captures the motion patterns and temporal evolution features of these objects, providing a rich visual semantic foundation for subsequent recognition of exciting segments.

[0084] In one alternative implementation, the acoustic feature tensor and the visual semantic tensor are aligned across modalities according to timestamps to construct a joint representation space.

[0085] When aligning acoustic feature tensors and visual semantic tensors across modalities using timestamps, the time-series information of both tensors is first obtained. The acoustic feature tensor contains acoustic features such as fundamental frequency variations, volume variations, and speech rate variations in the narrator's voice; these features are indexed using millisecond-level timestamps. The visual semantic tensor contains visual semantic information such as scene segmentation results, target motion trajectories, and target interaction relationships, also indexed using millisecond-level timestamps. Due to the difference between audio sampling rates and video frame rates, the temporal resolutions of the acoustic feature tensor and the visual semantic tensor differ, necessitating time synchronization alignment.

[0086] A dynamic time warping algorithm is employed to align the acoustic feature tensor and the visual semantic tensor along their time axes. By calculating the optimal correspondence between the two time series, the problem of inconsistent time axis scaling is resolved. Specifically, the acoustic feature tensor is segmented into 10-millisecond time windows, and a set of acoustic feature vectors is extracted for each window. The visual semantic tensor is segmented according to the video frame rate (e.g., 25 frames / second, or 40 milliseconds / frame), and a set of visual semantic vectors is extracted for each frame. A dynamic programming algorithm is used to calculate the cumulative distance matrix between the two feature sequences, finding the path with the minimum cumulative distance to obtain the optimal time correspondence between the acoustic features and visual semantics. For example, in a live basketball game, there is approximately a 300-millisecond time difference between the commentator's "wonderful dunk" and the player's dunking action on screen; the dynamic time warping algorithm can correctly align these two events.

[0087] After time alignment, a cross-modal joint representation space is constructed. This representation space needs to capture the intrinsic relationship between acoustic features and visual semantics. This is achieved by designing a two-stream neural network architecture to process the acoustic feature stream and the visual semantic stream separately, and then fusing the features from the two streams. The acoustic feature stream uses a Long Short-Term Memory (LSTM) network to process the acoustic feature tensor, capturing the temporal variation patterns of the narration; the visual semantic stream uses a graph convolutional network to process the visual semantic tensor, capturing the spatial relationship changes between objects in the scene. The output features of the two streams are fused through an attention mechanism to achieve complementary enhancement of cross-modal information.

[0088] The attention fusion mechanism comprises two parts: cross-modal self-attention and cross-modal mutual attention. Cross-modal self-attention calculates the correlation weights between feature elements within both acoustic and visual semantics, enhancing intra-modal representation capabilities. Cross-modal mutual attention calculates the correlation weights between acoustic features and visual semantics, enabling inter-modal information interaction. For example, when the commentary's tone and volume noticeably increase, the mutual attention mechanism automatically focuses on the athlete's rapid movement or the moment a goal is scored, thus highlighting the importance of this moment in the joint representation space.

[0089] In constructing the joint representation space, a multi-task learning framework is adopted, simultaneously optimizing multiple objective functions. The first objective function is the cross-modal alignment loss, ensuring the accuracy of temporal alignment; the second objective function is the cross-modal reconstruction loss, which reconstructs features of one modality from features of another, enhancing the semantic consistency between the two modalities; the third objective function is the discriminative loss, ensuring that the learned joint representation can effectively distinguish between exciting segments and ordinary segments.

[0090] To enhance the generalization ability of the joint representation, a cross-modal data augmentation strategy was introduced during training. Temporal masking, frequency masking, and pitch transformation were applied to acoustic features; random cropping, color jittering, and scene blending were applied to visual semantics. This approach enables the model to adapt to acoustic and visual variations in different competition environments.

[0091] The joint representation space is designed with 256 dimensions, with the first 128 dimensions primarily encoding acoustic information and the latter 128 dimensions primarily encoding visual information, with some degree of information overlap in between. This design allows the representation space to both preserve the unique characteristics of each modality and capture the correlation patterns between modalities. In practical applications, for recognizing exciting dunking moments in basketball games, the joint representation space will show a pattern where the acoustic features of the commentator's rising tone and increased volume are highly correlated with the visual features of the player jumping and the ball entering the hoop; for the moment of a goal in a football match, the acoustic features of the commentator's prolonged tone and increased speaking speed will show a pattern where the visual features of the player shooting and the ball crossing the goal line are highly correlated.

[0092] To address potential audio-visual desynchronization issues during live sports broadcasts, a sliding time window mechanism is introduced into the joint representation space construction process. A sliding window of ±2 seconds is set to find the optimal matching point between acoustic features and visual semantics within this range, resolving audio-visual desynchronization caused by transmission delays. In practical applications, this mechanism effectively handles live stream synchronization issues under different broadcast sources and network conditions, improving the accuracy of highlight segment recognition.

[0093] The resulting joint representation vector is smoothed using a time-series filter to eliminate noise fluctuations, resulting in a continuous and stable representation sequence. Peaks in this sequence often correspond to highlights of the match and can be automatically identified through threshold detection. For example, in a 90-minute live football match, the joint representation sequence will show significant peaks at moments such as goals, spectacular shots, and key passes. Automatically identifying these peaks allows for accurate extraction of highlight segments. Experiments show that this method achieves a 92% accuracy rate for highlight segment identification in various ball sports live broadcasts, with an average time positioning error of less than 0.5 seconds.

[0094] like Figure 2 As shown, the logical flow of the competitive state change analysis is illustrated.

[0095] In one optional embodiment, acoustic feature tensors and visual semantic tensors are clustered in a joint representation space to obtain competition state nodes, a competition state transition topology graph is constructed, unconventional transition edges are identified based on the weight of outgoing edges, and state change markers are determined through the collaborative jump of prosodic periodic sequences and motion trajectory sequences, including:

[0096] In the joint representation space, the acoustic feature tensor and the visual semantic tensor are clustered to obtain multiple clusters. The centroid of the cluster is calculated as the cluster center, and the cluster center is defined as the competition state node.

[0097] Traverse the time series to extract the competitive state nodes corresponding to adjacent time points, count the frequency of transition from the competitive state node of the previous time point to the competitive state node of the next time point, determine the transition probability, and construct a competitive state transition topology graph by taking the competitive state node as a vertex, the transition relationship between competitive state nodes as a directed edge, and the transition probability as the edge weight.

[0098] For each competitive state node in the competitive state transition topology graph, extract all outgoing edge weights of the competitive state node within the historical time window, calculate the statistical distribution characteristics of the outgoing edge weights, determine the weight boundary value, and mark outgoing edges whose weights are below the weight boundary value as non-standard transition edges.

[0099] Obtain the acoustic feature tensor and visual semantic tensor of the time corresponding to the unconventional transition edge in the joint representation space, extract the prosodic periodic sequence from the acoustic feature tensor, extract the motion trajectory sequence from the visual semantic tensor, and mark the time corresponding to the unconventional transition edge as a state change marker when the prosodic periodic sequence and the motion trajectory sequence undergo a cooperative jump.

[0100] In one specific implementation, after obtaining the joint representation space composed of the acoustic feature tensor and the visual semantic tensor, it is necessary to mine the competitive state patterns contained therein. Each timestamp in the joint representation space corresponds to a multi-dimensional vector, which integrates the acoustic properties of the narration audio and the semantic information of the visual image. Performing a clustering operation on these vectors can group the data points distributed in the representation space into several cohesive clusters.

[0101] The clustering process employs density-based spatial clustering or spectral clustering algorithms. During implementation, the feature vectors corresponding to all timestamps in the joint representation space are used as the input sample set. By iteratively optimizing the ratio of intra-cluster variance to inter-cluster distance, feature vectors within the same cluster are made closer to each other in the representation space, while feature vectors between different clusters are further apart. After clustering, several clusters are obtained, each representing a typical competitive state pattern. For example, in a football match, one cluster might correspond to the defensive back-pass phase, where the commentator's tone is calm and the players' movement speed on screen is low; another cluster might correspond to the fast counter-attack phase, where the commentator's speech speed increases and the tone rises, while players on screen make rapid forward runs.

[0102] For each cluster, a centroid is calculated; the centroid is the mean vector of all eigenvectors within that cluster. The centroid is located at the geometric center of the cluster in the representation space and represents the overall feature distribution of that cluster. Each centroid is defined as a competitive state node, and these nodes constitute a discretized abstract representation of the competitive process. Assume the number of clusters obtained is... Then the corresponding generation There are 1 competitive state node, denoted as _____. .

[0103] The entire live stream of the event is traversed along the time dimension, and the joint representation vector corresponding to each moment is extracted in chronological order. Each moment's representation vector is assigned to the nearest cluster centroid, thus labeling that moment as the corresponding competitive state node. In this way, the entire event process forms a sequence of competitive state nodes on the timeline. For example, the state sequence of the first five minutes of a basketball game might be as follows: This reflects the dynamic evolution of competitive state over time.

[0104] Count the frequency of state transitions between adjacent time points. Traverse the entire state sequence and record the transitions from the competition state node. Transfer to competitive state node The number of times, recorded as For each starting node Calculate the total number of times it is transferred to each target node. Therefore, the transition probability is determined. The transition probability reflects the typical pattern of evolution from one state to another during a competitive event. For example, in tennis, the probability of transitioning from a baseline rally to a volley at the net may be significantly lower than the probability of continuing the baseline rally.

[0105] Based on the aforementioned transition frequency and probability, a topology graph of the competitive state transition is constructed. This topology graph is based on competitive state nodes. As vertices, directed edges are formed by the transition relationships between states. If there exists a follower node... To the node The transfer record is then plotted in the topology graph as a path from... point to The directed edges are defined, and the weight of the edges is set as the transition probability. The topology diagram visually illustrates the evolution patterns between various state modes during the competition. Transition paths with larger edge weights represent conventional state evolution patterns, while transition paths with smaller edge weights represent unconventional evolutions that occur less frequently.

[0106] For each competitive state node in the topology graph, extract the weights of all its outgoing edges. For each node... The set of weights corresponding to its outgoing edges is In actual competitions, competitive states generally evolve according to a conventional pattern for most of the time, with a few key moments exhibiting unconventional state transitions. To identify these unconventional transitions, it is necessary to perform statistical analysis on the weights of the outgoing edges within a historical time window. This involves selecting all historical data from the start of the competition to the current moment and calculating the weights of the outgoing edges from the nodes. The statistical distribution characteristics of all outgoing edge weights, including the mean. and standard deviation .

[0107] The weight threshold is determined based on statistical distribution characteristics. An outlier detection approach is used to set weights below a certain threshold. The outgoing edges are identified as non-standard transition edges, where To control the sensitivity coefficient, it is typically set between 1.5 and 2.5. When the weight of a certain outgoing edge deviates significantly from the node's normal outgoing pattern, the state transition corresponding to that edge may be accompanied by a key event in the match. For example, in badminton, the probability of suddenly transitioning from a flat drive / quick block state to a powerful smash state is usually low. If such a low-probability transition is detected, it often indicates that a scoring opportunity has arisen on the court.

[0108] After identifying the unconventional transition edges, further verification is conducted to confirm whether the transition truly corresponds to a state change. The acoustic feature tensor and visual semantic tensor of the corresponding time point of the unconventional transition edge in the joint representation space are obtained. A prosodic cycle sequence is extracted from the acoustic feature tensor, reflecting the prosodic rhythm changes of the commentator's speech. The prosodic cycle is obtained by analyzing the peak intervals of the fundamental frequency envelope. During the calmer phases of the event, the prosodic cycle is relatively regular, while during exciting moments, the commentator often exhibits a sharp increase in prosodic rhythm or a sudden pause.

[0109] Motion trajectory sequences are extracted from visual semantic tensors, recording the changes in the position coordinates of key moving targets in the image over time. In live sports broadcasts, the trajectories of athletes or balls exhibit relatively smooth changes during the normal competitive phase, but at crucial moments, there are often sudden changes in trajectory, rapid acceleration, or drastic alterations in movement patterns. By calculating the acceleration vector of the motion trajectory or the time derivative of the trajectory curvature, the degree of change in movement patterns can be quantified.

[0110] Determine whether a co-jump occurs between a prosodic periodic sequence and a motion trajectory sequence. A co-jump refers to the phenomenon where both sequences simultaneously deviate significantly from their respective historical patterns within the same time window. During implementation, the time derivative of the prosodic periodic sequence is calculated. ,in Indicates time The length of the rhythmic cycle. When Exceeding the preset threshold At that time, a change in rhythm is considered to occur. Simultaneously, the acceleration magnitude of the motion trajectory sequence is calculated. ,in , For a moment The position coordinates. When Exceeding the preset threshold At that time, it was considered that a jump occurred in the motion.

[0111] If, within the time window preceding and following the moment of the unconventional transition, both the rhythmic cycle sequence and the motion trajectory sequence simultaneously satisfy their respective transition criteria, then a coordinated transition is determined. This co-transition verification ensures that the identified state abrupt change is not only rare in statistical patterns but also supported by actual signals in both acoustic and visual dimensions. For example, in a table tennis match, when an unconventional transition from a stalemate to a scoring state is detected, if simultaneously a sudden increase in the commentator's tone and a sharp drop in the ball's trajectory are observed, then this moment is confirmed as a state abrupt change marker.

[0112] The moments corresponding to unconventional transition edges that satisfy all the above conditions are marked as state change markers. These markers accurately locate key turning points in the competition process on the timeline, providing anchor points for defining the boundaries of subsequent exciting segments. Through this method based on topological analysis and multimodal collaborative verification, substantial changes in the competitive state can be reliably captured in complex live broadcast streams, filtering out noise interference and random fluctuations, ensuring that the identified markers have genuine competitive significance.

[0113] In an optional embodiment, when the prosodic periodic sequence and the motion trajectory sequence undergo a coordinated transition, marking the time corresponding to the unconventional transition edge as a state change marker includes:

[0114] Period length jump detection is performed on the prosodic periodic sequence. The period length value within a continuous time window in the prosodic periodic sequence is extracted. The difference between adjacent period length values ​​is calculated to form a period length change curve. Local extreme points of change amplitude are identified on the period length change curve, and the corresponding time is marked as a candidate time of prosodic jump.

[0115] Trajectory density jump detection is performed on the motion trajectory sequence. The number of trajectory points in the motion trajectory sequence is counted within a fixed time window to form a trajectory density time series. The density difference between adjacent windows of the trajectory density time series is calculated to form a trajectory density change curve. Local extreme points of change amplitude are identified on the trajectory density change curve, and the corresponding time is marked as a candidate time for trajectory jump.

[0116] Calculate the time difference between the candidate times of rhythmic transition and the candidate times of trajectory transition. When the time difference is within the preset synchronization time window, confirm that the rhythmic periodic sequence and the motion trajectory sequence have undergone a coordinated transition, and mark the time corresponding to the unconventional transition edge as a state change marker.

[0117] In one specific implementation, identifying key moments in a live sports broadcast requires co-analyzing the commentator's prosodic characteristics and the athlete's behavioral characteristics to confirm abrupt changes in competitive state. Specifically, the prosodic cycle sequence reflects the commentator's speaking speed, tone, and pause rhythm within a specific time period. These characteristics exhibit relatively stable periodic changes during the regular course of the game, but significant prosodic pattern shifts occur during important events such as goals, mistakes, and crucial scores. Simultaneously, the motion trajectory sequence, by tracking the displacement paths of athletes, the ball, or other key targets on the field, shows a relatively stable density distribution during the regular tactical execution phase, but exhibits sudden increases or decreases in density during crucial scenarios such as offensive-defensive transitions and fast counterattacks. Co-judicially determining these two signal transitions can effectively reduce misjudgments caused by a single modality and accurately capture the moment of a state change.

[0118] In the period length jump detection stage, the prosodic period sequence extracted from the narration audio channel is first segmented. A sliding time window of fixed duration is selected, and the window length is set to... seconds, step size set to Within each time window, the cycle length of the complete prosodic cycle is extracted. Specifically, the duration of a single prosodic cycle is determined by the interval between extreme points of the fundamental frequency curve, and the average of all cycle length values ​​within the window is used as the representative cycle length for that window. ,in This indicates the window number. For a continuous time window sequence, the difference between the period lengths represented by adjacent windows is calculated to form a curve showing the change in period length. This curve reflects the acceleration or deceleration trend of the rhythm. During normal narration, the curve exhibits small fluctuations, while sharp positive or negative jumps occur during important events. A dynamic threshold is set on the cycle length variation curve to detect local extreme points; the threshold is based on the standard deviation of the curve over a longer historical window. and mean Determine the magnitude of change at a certain moment. Exceed When that moment is marked as a candidate moment for a prosodic jump. ,in This is the sensitivity coefficient, which typically ranges from 1.5 to 2.5.

[0119] In parallel, during trajectory density jump detection, density statistical analysis is performed on the motion trajectory sequences extracted from the visual image channel. The live stream is divided into time windows of fixed size, with the window length set to... seconds, step size set to Seconds, count the total number of trajectory points within each time window. ,in This indicates the window number. The number of trajectory points reflects the activity level of targets on the field. For example, in a football match, trajectory density increases significantly when multiple players are vying for possession in the penalty area, while it remains relatively stable during midfield possession. A time series of trajectory density is generated through statistical analysis. Further calculations were performed on the density difference between adjacent windows to construct a trajectory density variation curve. The change curve exhibits low-amplitude fluctuations during routine tactical execution, but shows sharp increases or decreases in density during key scenarios such as fast counter-attacks, defensive transitions, and shots. Similarly, a dynamic threshold method based on historical statistics is used to calculate the standard deviation of the trajectory density change curve within a historical window. and mean The magnitude of change at a certain moment Exceed At that time, mark that moment as a candidate moment for trajectory transition. ,in This is a sensitivity coefficient, which is adjusted between 1.5 and 3.0 depending on the type of event.

[0120] Obtaining the set of candidate moments for rhythmic transitions and the set of candidate moments for trajectory transition Next, a collaborative transition verification mechanism is executed. For each prosodic transition candidate moment... Traverse the set of candidate moments for trajectory transitions and calculate the time difference. Set the preset synchronization time window threshold. The selection of this threshold needs to consider factors such as the commentator's reaction delay, video encoding delay, and network transmission delay, and is usually set to 0.5 seconds to 2.0 seconds. When a candidate moment for a trajectory change exists... Make When a rhythmic periodic sequence and a motion trajectory sequence undergo a coordinated transition within that time period, the average time of the candidate rhythmic transition moment and the corresponding candidate trajectory transition moment is taken. This serves as the final confirmed collaborative transition moment. Furthermore, this confirmed moment is correlated with the previously identified unconventional transition edges, and the timestamp is searched in the competitive state transition topology graph. For transition edges that match within the error range, the state transition time corresponding to the transition edge is marked as a state change marker. .

[0121] To enhance robustness, a confidence assessment mechanism is introduced during the collaborative transition verification process. For each successfully matched prosodic transition candidate moment and trajectory transition candidate moment, the collaborative confidence is calculated based on the intensity of their change amplitude. Specifically, the prosodic transition intensity is defined as... The intensity of the trajectory jump is defined as Collaborative confidence is calculated using a weighted geometric mean. Calculation, where This is the weighting coefficient, typically set to 0.5 to balance the contributions of the two modes. Only when the collaborative confidence level... Exceeding the preset threshold Only then is that moment finally confirmed as a state transition marker. This mechanism effectively avoids misjudgments caused by single-modal noise interference; for example, an occasional cough by the commentator or a sudden movement by a spectator on the sidelines will not be mistakenly identified as the start of a highlight segment.

[0122] The parameter configuration for collaborative jump detection varies across different game scenarios. In football matches, commentators typically react quickly to events such as goals and shots, requiring a synchronized time window. The synchronization window can be set to a shorter 0.8 to 1.5 seconds; however, in basketball games, due to frequent offensive and defensive transitions, commentators may react with lag, requiring a wider synchronization window of 1.5 to 2.5 seconds. Furthermore, for commentators speaking different languages, the expression of rhythmic features varies culturally, affecting the sensitivity coefficient. and The system needs to adaptively adjust based on the narration language. During the pre-training phase, the prosodic variation distribution characteristics of narrators speaking different languages ​​are statistically analyzed on a labeled dataset to automatically determine the optimal parameter combination. This parameter adaptation mechanism ensures stable recognition accuracy across multiple languages ​​and event scenarios.

[0123] In one optional embodiment, starting from the moment corresponding to the state change marker, the prosodic rhythmic period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythmic period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the highlight segment, including:

[0124] Using the moment corresponding to the state change marker as the starting point for tracing back, the acoustic feature tensor sequence and visual semantic tensor sequence prior to that moment are extracted in reverse from the joint representation space.

[0125] For each acoustic feature tensor, extract the spectral energy component, connect the spectral energy components in the time dimension to form a prosodic energy envelope curve, calculate the energy difference between adjacent moments on the prosodic energy envelope curve, mark the transition position when the energy difference changes from negative to positive, and use the transition position as the starting boundary of the prosodic rhythm cycle.

[0126] For each visual semantic tensor, its scene semantic representation is extracted as a scene semantic vector. Cosine similarity is calculated for the scene semantic vectors of adjacent time moments. When the cosine similarity is lower than a preset similarity threshold, the corresponding position of the adjacent time moment is marked as the dividing point of the shot narrative unit.

[0127] Traverse the subsequent time series forward along the time axis from the moment corresponding to the state change marker, detect the dividing point of the next appearing shot narrative unit in the subsequent time series as the end boundary of the shot narrative unit, extract the timestamp corresponding to the starting boundary of the rhythmic cycle and the timestamp corresponding to the ending boundary of the shot narrative unit, and determine the time interval of the exciting segment.

[0128] In one specific implementation, after determining the state change marker, it is necessary to perform bidirectional tracing in the joint representation space starting from the time corresponding to the marker, in order to accurately define the time interval boundary of the exciting segment. Specifically, during the tracing process, the timestamp corresponding to the state change marker is recorded as... This moment serves as the starting point for the tracing operation. In the joint representation space, the sequence of acoustic feature tensors prior to this moment is denoted as... Each tensor Contains acoustic feature information at that moment, and the time index satisfies Synchronously reverse-engineer the visual semantic tensor sequence prior to this moment. Each tensor The visual semantic representation corresponding to the same timestamp ensures that the temporal alignment between the acoustic and visual modalities remains consistent during the tracing process.

[0129] For prosodic rhythmic cycle boundary identification in the acoustic dimension, spectral energy components are extracted from each acoustic feature tensor. After performing a short-time Fourier transform on the acoustic feature tensor in the frequency domain, the energy of different frequency bands is statistically analyzed to obtain the spectral energy distribution at that moment. The spectral energy components of all acoustic feature tensors in the time series are concatenated along the time dimension to form a prosodic energy envelope curve. This curve reflects the energy change trend of the narration audio over time. The prosodic energy envelope curve shows an upward trend when the narrator's emotions are high or the speaking speed is fast, and a downward or stable trend when the emotions are calm or there is a pause. To identify the starting point of the prosodic rhythm cycle, it is necessary to calculate the energy difference between adjacent moments on this curve. ,in The time sampling interval is typically the reciprocal of the frame rate of the live stream or a fixed time step. When the energy difference... When a value changes from negative to positive, it signifies a shift in rhythmic energy from a declining to an increasing state, marking the beginning of a new round of emotional expression or topic development by the interpreter. Therefore, the timestamp corresponding to this transition is marked as the starting boundary of the rhythmic cycle. In practical calculations, to avoid misjudgments caused by noise or transient fluctuations, a sliding window can be applied to smooth the energy difference sequence, or the energy difference must remain positive for several consecutive frames before the transition is confirmed as valid.

[0130] For visual narrative unit boundary identification, scene semantic representations are extracted from each visual semantic tensor. After scene semantic segmentation, the visual semantic tensor contains semantic labels and spatial distributions of various objects in the scene. This semantic information is then compressed into fixed-dimensional scene semantic vectors through an encoding layer. This vector represents the overall semantic content of the image at that moment. Semantic vectors for adjacent moments... and Calculate cosine similarity The cosine similarity value ranges from zero to one. A higher value indicates that the semantic content of the scenes at two different times is more similar, while a lower value indicates that the scene has changed significantly. When the cosine similarity is below a preset similarity threshold... When a shot change or scene transition occurs between adjacent moments, the location is marked as the dividing point of the shot narrative unit. The preset similarity threshold is usually adjusted according to different event types. A lower threshold can be set for events with frequent shot changes, while a higher threshold can be set for events with relatively stable shots. The typical value range is between 0.6 and 0.8.

[0131] After determining the starting boundary of the rhythmic cycle, it is also necessary to determine the ending boundary of the narrative unit of the shot to fully define the time interval of the exciting segment. This is done by marking the moment corresponding to the state change. Traverse the subsequent time series in a forward direction along the timeline, detecting the boundary point of the next appearing shot narrative unit in the subsequent time series. During the forward traversal, calculate sequentially... Subsequently, the cosine similarity of the scene semantic vectors at adjacent time points is used. When a position with a cosine similarity lower than a preset similarity threshold is detected for the first time, the timestamp corresponding to that position becomes the endpoint boundary of the shot narrative unit. The finish line boundary marks the narrative integrity of the current highlight, typically corresponding to moments of narrative transition such as a shot cutting back from a close-up to a wide shot, from the race footage to a replay, or from the protagonist to the audience's reaction.

[0132] Obtain the starting boundary timestamp of the rhythmic cycle. and the end boundary timestamp of the narrative unit of the shot Then, the time range for the highlights was determined to be... The starting point of this time interval corresponds to the moment when the narrator begins to build fervor or the topic shifts, at which point the narrator typically begins to describe the key action that is about to occur or to comment on and lay the groundwork for events that have already happened; the ending point of this time interval corresponds to the moment when the narrative transitions, at which point the screen has fully presented the exciting action and its aftermath. This combination of starting and ending points ensures that the exciting segment includes both a complete emotional build-up and a complete visual narrative unit, avoiding any interruptions or incompleteness in the narration or visuals.

[0133] To improve the accuracy of boundary recognition, the prosodic energy envelope curve and scene semantic vector sequence can be preprocessed in practical implementation. For the prosodic energy envelope curve, Gaussian filtering or median filtering can be used to eliminate short-term noise interference, avoiding misidentification of energy abrupt changes caused by the narrator's cough, pauses, or background noise as the start of the prosodic cycle. For the scene semantic vector sequence, the vectors can be normalized before calculating cosine similarity to ensure that the difference in magnitude of semantic vectors at different times does not affect the similarity calculation results. In addition, when detecting the dividing point of the narrative unit in a shot, a minimum narrative unit duration constraint can be set, requiring that the time interval between two adjacent dividing points is not less than a certain threshold, to avoid excessive fragmentation of narrative units due to rapid shot switching.

[0134] In some complex scenarios, there may be multiple prosodic rhythmic cycle starting points before the state change marker. In such cases, it is necessary to select the starting point boundary closest to the state change marker as the starting point of the exciting segment. In specific implementation, from... Tracing back, record all timestamps that satisfy the condition of energy difference turning from negative to positive, and select the position with the largest timestamp as the final rhythmic starting point boundary. Similarly, when traversing the subsequent time sequence in a forward direction, if multiple shot demarcation points appear in a short period of time, prioritize the first demarcation point that appears as the endpoint boundary of the shot narrative unit to ensure the compactness and timeliness of the exciting segments.

[0135] Through the aforementioned two-way tracing and boundary determination mechanism, the time interval of exciting segments can be accurately defined in the joint representation space, so that the extracted segments have both a complete narration structure and a complete visual narrative unit, providing an accurate time positioning basis for extracting corresponding segments from the live broadcast of the event.

[0136] In one optional embodiment, extracting the timestamp corresponding to the starting boundary of the rhythmic cycle and the timestamp corresponding to the ending boundary of the shot narrative unit to determine the time interval of the highlight segment includes:

[0137] The initial starting point is the timestamp corresponding to the boundary of the rhythmic cycle, and the initial ending point is the timestamp corresponding to the boundary of the narrative unit of the shot.

[0138] Multiple candidate start times are sampled within preset adjustment ranges before and after the initial start point, and multiple candidate end times are sampled within preset adjustment ranges before and after the initial end point. For each candidate time period defined by a pair of candidate start times and candidate end times, the acoustic feature tensor sequence and visual semantic tensor sequence within the candidate time period are extracted. The mutual information value between the acoustic feature tensor sequence and the visual semantic tensor sequence is calculated. The mutual information value is obtained by calculating the difference between the corresponding joint distribution entropy and their respective marginal distribution entropy. The candidate time period with the largest mutual information value is selected to determine the time interval of the exciting segment.

[0139] In one specific implementation, after obtaining the timestamps corresponding to the starting boundary of the rhythmic cycle and the ending boundary of the narrative unit, the initial time interval defined by these two timestamps needs to be finely adjusted to ensure that the extracted highlights achieve the best effect in terms of narrative integrity and emotional coherence. The core of this adjustment mechanism lies in fully utilizing the cross-modal correlation strength between acoustic features and visual semantics, and finding the optimal segment boundary by quantifying the degree of information coupling between the two.

[0140] Record the timestamp corresponding to the starting boundary of the rhythmic cycle as This timestamp marks the starting point where the narrator's emotions begin to change significantly. The timestamp corresponding to the end boundary of the narrative unit is denoted as... This timestamp marks the end point of a complete unit of visual narrative expression. These two timestamps constitute the initial start and end points of the key segments, forming a preliminary time interval. However, this initial interval may not fully capture the complete narrative structure of the exciting events, because the narrator's emotional changes may precede or follow key changes in the visuals, and there may be continuity shots after the event has ended.

[0141] To find more accurate segment boundaries, at the initial starting point Set a preset adjustment range for both the front and back. The adjustment range is typically set between 2 and 5 seconds, with the specific value adjusted according to the pace of different sports events. For sports with high continuity, such as football and basketball, A larger value can be chosen to capture the initial setup for exciting moves; for fast-paced, round-robin sports such as badminton and table tennis, A smaller value should be chosen to avoid including too much irrelevant content. Within the time interval... Within this timeframe, samples are taken at fixed time intervals, typically set to 0.2 to 0.5 seconds, generating a set of candidate starting times. ,in This indicates the number of candidate starting times.

[0142] Similarly, at the initial and final endpoints Set a preset adjustment range for both the front and back. The principle for setting this range is similar to that for adjusting the starting point, but it is usually set slightly larger. This is because the ending of a captivating event often includes replays, audience reactions, and other elements of significant narrative value. Within a time frame... Sampling is performed within the time frame to generate a set of candidate endpoint times. ,in This represents the number of candidate endpoint times. This sampling strategy can be used to generate... For each combination of candidate starting point and candidate ending point, a candidate time period is defined.

[0143] For any pair of candidate starting times and candidate endpoint time This can define a candidate time period. Extract the acoustic feature tensor sequence within the candidate time period, denoted as... ,in This represents the number of acoustic feature tensors within that time period, and each acoustic feature tensor... It includes quantitative information on acoustic dimensions such as the fundamental frequency change rate, energy envelope, and formant distribution at the corresponding time point. Simultaneously, the visual semantic tensor sequence for this time period is extracted and denoted as... Each visual semantic tensor It includes semantic information from visual dimensions such as scene layout, moving target position, and pose features. Since the acoustic feature tensor and the visual semantic tensor have already been timestamped when constructing the joint representation space, the two sequences have the same length and a clear temporal correspondence.

[0144] The mutual information value between the acoustic feature tensor sequence and the visual semantic tensor sequence is calculated to quantify the information coupling strength between the two modalities within a given time period. Mutual information is an indicator in information theory used to measure the degree of statistical dependence between two random variables; a higher value indicates a greater amount of shared information between the two variables. In this scenario, the mutual information value... This can be expressed as the difference between the joint distribution entropy and the entropy of each individual marginal distribution, calculated using the following formula: ,in The marginal distribution entropy represents the sequence of acoustic feature tensors. The marginal distribution entropy represents the sequence of visual semantic tensors. This represents the joint distribution entropy of the two.

[0145] Marginal distribution entropy The computation requires first estimating the probability distribution of the acoustic feature tensor sequence. A kernel density estimation-based method is used to map the high-dimensional acoustic feature tensors to a low-dimensional representation space, where a probability density function is then constructed. Specifically, each tensor in the acoustic feature tensor sequence is dimensionality-reduced to obtain a scalar representation. By statistically analyzing the frequency distribution of these scalars across different value ranges, a discrete probability distribution can be constructed. Thus, the marginal entropy is calculated. The same method is applied to visual semantic tensor sequences to obtain the marginal distribution entropy. .

[0146] Joint distribution entropy The computation requires constructing a joint probability distribution of acoustic features and visual semantics. This involves representing the acoustic features as scalars. Scalar representation of visual semantics Combined into two-dimensional sample points For all Two-dimensional probability density estimation is performed on each sample point to obtain the joint probability distribution. Calculate joint entropy Since mutual information is non-negative and zero when two variables are completely independent, the magnitude of mutual information directly reflects the degree of coordinated expression between acoustics and vision during that time period.

[0147] Traverse all For each combination of candidate start and candidate end points, the mutual information value is calculated for each candidate time period to obtain the mutual information matrix. ,in Indicates the starting point is The endpoint is The mutual information values ​​of the candidate time periods are used. The index corresponding to the maximum mutual information value is found in this matrix. The candidate starting time corresponding to this index and candidate endpoint time This refers to the optimized segment boundary. The time interval is determined at this point. It can maximize the coupling of acoustic and visual information, ensuring that the extracted highlights achieve the strongest synergy and consistency in the auditory and visual narrative expression.

[0148] To further enhance the robustness of boundary recognition, constraints can be introduced in practical applications to filter candidate time periods. This includes setting a minimum segment duration threshold. and the maximum segment duration threshold Only when satisfied Only candidate time periods are included in the mutual information calculation. This constraint avoids generating segments that are too short to fully represent the event or too long to contain redundant content. Simultaneously, temporal causality constraints can be introduced to ensure that the candidate start time is always earlier than the candidate end time. Exclude unreasonable time period combinations.

[0149] A second aspect of this invention provides an AI-powered highlight recognition system for live sports broadcasts, comprising:

[0150] The live stream acquisition unit is used to acquire the live stream of the event;

[0151] The cross-modal alignment unit is used to separate the commentary audio channel and the visual video channel from the live broadcast of the event. The commentary audio channel is subjected to acoustic fundamental frequency extraction and sound spectrum analysis to obtain an acoustic feature tensor. The visual video channel is subjected to scene semantic segmentation and target motion trajectory tracking to obtain a visual semantic tensor. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space.

[0152] The state mutation unit is used to cluster acoustic feature tensors and visual semantic tensors in the joint representation space to obtain competitive state nodes, construct a competitive state transition topology graph, identify unconventional transition edges based on the weight of the outgoing edge, and determine the state mutation marker through the collaborative jump of the prosodic period sequence and the motion trajectory sequence.

[0153] The segment boundary unit is used to trace the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor in the joint representation space, starting from the moment corresponding to the state change marker. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment.

[0154] The segment extraction unit is used to extract corresponding segments from the live event stream based on the time interval of the highlights.

[0155] A third aspect of the present invention provides an electronic device, comprising:

[0156] processor;

[0157] Memory used to store processor-executable instructions;

[0158] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0159] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0160] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI highlight identification method for live event broadcasting, characterized in that, include: Obtain the live stream of the event; The audio channel for commentary and the visual channel for video are separated from the live broadcast of the event. The acoustic feature tensor is obtained by extracting the fundamental frequency of the audio channel and performing sound spectrum analysis. The visual semantic tensor is obtained by performing scene semantic segmentation and target motion trajectory tracking on the visual channel. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space. In the joint representation space, acoustic feature tensors and visual semantic tensors are clustered to obtain competitive state nodes, and a competitive state transition topology graph is constructed. Unconventional transition edges are identified based on the weight of the outgoing edges, and state change markers are determined by the collaborative jump of prosodic periodic sequences and motion trajectory sequences. Starting from the moment corresponding to the state mutation marker, the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment. Extract relevant segments from the live event stream based on the time interval of the highlights.

2. The method according to claim 1, characterized in that, The acoustic feature tensor obtained by performing acoustic fundamental frequency extraction and chromatographic analysis on the narration audio channel includes: The fundamental frequency of the audio channel is extracted, the periodic repetition pattern of the audio signal is identified by autocorrelation operation, the time interval between adjacent peaks is calculated, and the reciprocal of the time interval is taken to obtain the fundamental frequency numerical sequence. The audio channel of the narration is subjected to chromatographic analysis. The audio signal is divided into frames according to a preset time length. Fourier transform is performed on each audio frame to obtain a frequency domain representation. The amplitude of each frequency component in the frequency domain representation is extracted. The frequency amplitudes of consecutive audio frames are arranged in time order to form a chromatographic matrix. A composite acoustic tensor is formed by tensor concatenation of the fundamental frequency numerical sequence and the chromatogram matrix. The composite acoustic tensor is then input into a three-layer convolutional neural network for spatial feature extraction. The first convolutional layer extracts the temporal variation pattern of the fundamental frequency numerical sequence, the second convolutional layer extracts the frequency energy distribution pattern of the chromatogram matrix, and the third convolutional layer fuses the temporal variation pattern and the frequency energy distribution pattern to output a feature map. The feature map is then flattened into a one-dimensional vector after global average pooling to obtain the acoustic feature tensor.

3. The method of claim 1, wherein, The visual semantic tensor obtained by performing scene semantic segmentation and target motion trajectory tracking on the visual image channel includes: Scene semantic segmentation is performed on the visual image channel. The video frame is input into the semantic segmentation network to predict the semantic category label of each pixel. Spatially connected pixels with the same semantic category label are aggregated into semantic regions, and the boundary coordinates and category labels of the semantic regions are extracted. The target motion trajectory is tracked in the visual image channel. The geometric center coordinates of the semantic region in the current frame are calculated to determine the current position point. In the next frame, all candidate semantic regions with the same category identifier as the current frame are searched. The Euclidean distance between the geometric center coordinates of each candidate semantic region and the current position point is calculated. The candidate semantic region with the smallest Euclidean distance is selected as the matching target. The geometric center coordinates of the successfully matched frames are concatenated in time order to form a trajectory coordinate sequence. The category identifiers of the semantic region are one-hot encoded to obtain category vectors. The category vectors are concatenated with the trajectory coordinate sequence in time dimension to form a temporal feature sequence. The temporal feature sequence is input into a recurrent neural network to model temporal dependencies. The hidden state vector of the recurrent neural network at the last moment is extracted to obtain the visual semantic tensor.

4. The method of claim 1, wherein, In the joint representation space, acoustic feature tensors and visual semantic tensors are clustered to obtain competition state nodes. A competition state transition topology graph is constructed, and unconventional transition edges are identified based on the weight of outgoing edges. State change markers are determined by the collaborative jump of prosodic periodic sequences and motion trajectory sequences, including: In the joint representation space, the acoustic feature tensor and the visual semantic tensor are clustered to obtain multiple clusters. The centroid of the cluster is calculated as the cluster center, and the cluster center is defined as the competition state node. Traverse the time series to extract the competitive state nodes corresponding to adjacent time points, count the frequency of transition from the competitive state node of the previous time point to the competitive state node of the next time point, determine the transition probability, and construct a competitive state transition topology graph by taking the competitive state node as a vertex, the transition relationship between competitive state nodes as a directed edge, and the transition probability as the edge weight. For each competitive state node in the competitive state transition topology graph, extract all outgoing edge weights of the competitive state node within the historical time window, calculate the statistical distribution characteristics of the outgoing edge weights, determine the weight boundary value, and mark outgoing edges whose weights are below the weight boundary value as non-standard transition edges. Obtain the acoustic feature tensor and visual semantic tensor of the time corresponding to the unconventional transition edge in the joint representation space, extract the prosodic periodic sequence from the acoustic feature tensor, extract the motion trajectory sequence from the visual semantic tensor, and mark the time corresponding to the unconventional transition edge as a state change marker when the prosodic periodic sequence and the motion trajectory sequence undergo a cooperative jump.

5. The method of claim 4, wherein, When the prosodic periodic sequence and the motion trajectory sequence undergo a coordinated transition, the time corresponding to the unconventional transition edge is marked as a state change marker, including: Period length jump detection is performed on the prosodic periodic sequence. The period length value within a continuous time window in the prosodic periodic sequence is extracted. The difference between adjacent period length values ​​is calculated to form a period length change curve. Local extreme points of change amplitude are identified on the period length change curve, and the corresponding time is marked as a candidate time of prosodic jump. Trajectory density jump detection is performed on the motion trajectory sequence. The number of trajectory points in the motion trajectory sequence is counted within a fixed time window to form a trajectory density time series. The density difference between adjacent windows of the trajectory density time series is calculated to form a trajectory density change curve. Local extreme points of change amplitude are identified on the trajectory density change curve, and the corresponding time is marked as a candidate time for trajectory jump. Calculate the time difference between the candidate times of rhythmic transition and the candidate times of trajectory transition. When the time difference is within the preset synchronization time window, confirm that the rhythmic periodic sequence and the motion trajectory sequence have undergone a coordinated transition, and mark the time corresponding to the unconventional transition edge as a state change marker.

6. The method of claim 1, wherein, Starting from the moment corresponding to the state change marker, the prosodic rhythmic period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor are traced in the joint representation space. The time interval defined by the starting point of the prosodic rhythmic period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the highlight segment, including: Using the moment corresponding to the state change marker as the starting point for tracing back, the acoustic feature tensor sequence and visual semantic tensor sequence prior to that moment are extracted in reverse from the joint representation space; For each acoustic feature tensor, extract the spectral energy component, connect the spectral energy components in the time dimension to form a prosodic energy envelope curve, calculate the energy difference between adjacent moments on the prosodic energy envelope curve, mark the transition position when the energy difference changes from negative to positive, and use the transition position as the starting boundary of the prosodic rhythm cycle. For each visual semantic tensor, its scene semantic representation is extracted as a scene semantic vector. Cosine similarity is calculated for the scene semantic vectors of adjacent time moments. When the cosine similarity is lower than a preset similarity threshold, the corresponding position of the adjacent time moment is marked as the dividing point of the shot narrative unit. Traverse the subsequent time series forward along the time axis from the moment corresponding to the state change marker, detect the dividing point of the next appearing shot narrative unit in the subsequent time series as the end boundary of the shot narrative unit, extract the timestamp corresponding to the starting boundary of the rhythmic cycle and the timestamp corresponding to the ending boundary of the shot narrative unit, and determine the time interval of the exciting segment.

7. The method according to claim 6, characterized in that, Extracting the timestamps corresponding to the starting boundary of the rhythmic cycle and the ending boundary of the shot narrative unit, the time intervals of the highlights are determined as follows: The initial starting point is the timestamp corresponding to the boundary of the rhythmic cycle, and the initial ending point is the timestamp corresponding to the boundary of the narrative unit of the shot. Multiple candidate start times are sampled within preset adjustment ranges before and after the initial start point, and multiple candidate end times are sampled within preset adjustment ranges before and after the initial end point. For each candidate time period defined by a pair of candidate start times and candidate end times, the acoustic feature tensor sequence and visual semantic tensor sequence within the candidate time period are extracted. The mutual information value between the acoustic feature tensor sequence and the visual semantic tensor sequence is calculated. The mutual information value is obtained by calculating the difference between the corresponding joint distribution entropy and their respective marginal distribution entropy. The candidate time period with the largest mutual information value is selected to determine the time interval of the exciting segment.

8. AI highlight identification system for live event broadcasting for implementing the method according to any one of claims 1 to 7, characterized in that, include: The live stream acquisition unit is used to acquire the live stream of the event; The cross-modal alignment unit is used to separate the commentary audio channel and the visual video channel from the live broadcast of the event. The commentary audio channel is subjected to acoustic fundamental frequency extraction and sound spectrum analysis to obtain an acoustic feature tensor. The visual video channel is subjected to scene semantic segmentation and target motion trajectory tracking to obtain a visual semantic tensor. The acoustic feature tensor and the visual semantic tensor are aligned across modally according to the timestamp to construct a joint representation space. The state mutation unit is used to cluster acoustic feature tensors and visual semantic tensors in the joint representation space to obtain competitive state nodes, construct a competitive state transition topology graph, identify unconventional transition edges based on the weight of the outgoing edge, and determine the state mutation marker through the collaborative jump of the prosodic period sequence and the motion trajectory sequence. The segment boundary unit is used to trace the prosodic rhythm period boundary of the acoustic feature tensor and the shot narrative unit boundary of the visual semantic tensor in the joint representation space, starting from the moment corresponding to the state change marker. The time interval defined by the starting point of the prosodic rhythm period boundary and the ending point of the shot narrative unit boundary is determined as the time interval of the exciting segment. The segment extraction unit is used to extract corresponding segments from the live event stream based on the time interval of the highlights.

9. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images