Network sports match video round extraction and collection generation method integrating audio and video

Abstract

The invention provides an audio and video fused network sports match video round extraction and collection generation method, belongs to the technical field of video processing, and improves the automation level, robustness and processing efficiency of round identification. The method comprises the following steps: firstly, respectively extracting audio and video streams from a video, and constructing a smooth energy sequence and a main view confidence sequence; then, aligning and normalizing the two sequences, generating a one-dimensional discrimination sequence through dynamic weight fusion, and obtaining a turn state sequence after smoothing and self-adaptive threshold discrimination; and finally, positioning and optimizing a round time period based on the state sequence, and automatically intercepting corresponding fragments and synthesizing the fragments into a collection. According to the method, through complementary fusion and lightweight sequence processing of audio and video information, the influence of interference on a single mode is remarkably reduced, the defects that an existing method based on a complex visual model is large in calculation amount and difficult to deploy are overcome, and efficient, accurate and wide-application-range automatic collection generation is achieved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically a method for extracting rounds and generating highlights from netball sports competition videos that integrate audio and video. Background Technology

[0002] With the increasing popularity of live sports broadcasts, viewers' demand for highlights is growing. Traditional highlight production mainly relies on manual editing, which has problems such as being time-consuming, labor-intensive, costly, and highly subjective.

[0003] Existing automated highlight reel methods are mostly based on computer vision or deep learning models, analyzing video frame by frame through object detection, action recognition, and trajectory tracking. However, these methods are computationally complex, demanding on hardware resources, and difficult to generalize across projects, making them unstable for deployment on resource-constrained devices. On the other hand, network-based sports broadcast videos often contain both narration and fixed first-person perspective shots. The narration shows statistical differences between rounds and round intervals, while the first-person perspective shots account for a high proportion during the round-based combat phase and have a relatively stable image structure. However, relying solely on audio or video image features can be affected by noise, playback, and camera switching, leading to unstable round boundary determination. Therefore, there is an urgent need for a round extraction and highlight generation method that is computationally efficient, simple to implement, configurable and adaptable to multiple projects, and can integrate audio and video information to improve robustness. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes a method for extracting rounds and generating highlights from netball competition videos by integrating audio and video.

[0005] Therefore, the technical solution adopted by the present invention is as follows:

[0006] A method for extracting rounds and generating highlights from net sports competition videos by integrating audio and video, the method including:

[0007] S1: Acquire videos of tennis matches and extract audio signals and video frame sequences; perform frame-by-frame processing on the audio signals to calculate short-time energy, and further perform three-stage cascaded filtering to obtain a smooth energy sequence; extract image features based on the video frame sequences and calculate the main viewpoint confidence sequence;

[0008] S2: Align and normalize the smoothed energy sequence with the main view confidence sequence, and fuse the normalized smoothed energy sequence with the main view confidence sequence to generate a one-dimensional discriminant sequence; smooth the one-dimensional discriminant sequence and generate a round state sequence based on an adaptive threshold;

[0009] S3: Perform round location on the round state sequence to obtain candidate round time periods; perform three-level optimization processing based on the candidate round time periods to obtain an optimized round time period set; generate match highlights based on the optimized round time period set.

[0010] Furthermore, the net sports competition video includes a round-based phase and a round-interval phase;

[0011] The audio signal includes an audio track containing narration;

[0012] The framing process uses a sliding window method with a fixed frame length and frame shift to obtain short-time energy, represented as:

[0013]

[0014] in, Indicates the first The short-time energy of a frame; Indicates the first The number of audio sample points in a frame; Indicates the first The amplitude value of each sampling point; the audio signal is processed by frame segmentation to obtain a short-time energy sequence.

[0015] Furthermore, the three-stage cascaded filtering includes nonlinear filtering, low-pass filtering, and long-window averaging filtering;

[0016] The nonlinear filter performs impulse noise suppression on the short-time energy sequence as a first-level filter.

[0017] The low-pass filter smooths the short-time energy sequence after the first-stage filter as a second-stage filter.

[0018] The long-time window average filter extracts the short-time energy trend after the second-level filter as the output of the third-level filter, and the output of the third-level filter is a smoothed energy value.

[0019] A three-stage cascaded filtering process is performed on all frames in the short-time energy sequence to generate a smooth energy sequence.

[0020] Furthermore, the video frame sequence is subjected to frame extraction processing to form a temporal input sequence on the video side, represented as:

[0021]

[0022] in, Indicates the first Frame image; Represented as the first The corresponding timestamp assigned to the frame image;

[0023] The image features include brightness statistics features, color distribution features, global motion features, and region proportion features;

[0024] Based on the image features, a main viewpoint scoring function is constructed, and the first output is... Confidence of the main viewpoint of the frame image The confidence level of the main viewpoint is composed of the brightness stability sub-confidence level. Confidence level of motion stability and regional proportion of sub-confidence The result obtained through weighted fusion is expressed as:

[0025]

[0026] in, , and The weights of the brightness stability sub-confidence, motion stability sub-confidence, and region proportion sub-confidence are respectively represented; the main viewpoint confidence of all frames in the temporal input sequence is calculated to generate the main viewpoint confidence sequence.

[0027] Furthermore, the alignment is based on the video side, targeting the first [unclear] in the video side. The timestamp of the frame image is used to extract the energy value of the corresponding time period from the smooth energy sequence and aggregate it to obtain the aligned smooth energy sequence. The aligned smooth energy sequence and the main view confidence sequence are then normalized.

[0028] Furthermore, the round score on the audio side is obtained based on the normalized smooth energy sequence, and is expressed as:

[0029]

[0030] in, Indicates the first Round scores on the frame audio side; Indicates the first The smoothed energy value after frame normalization;

[0031] Based on the normalized first-person confidence sequence, the round score on the video side is obtained, expressed as:

[0032]

[0033] in, Indicates the first Round scores on the frame video side; Indicates the first Frame-normalized first-person perspective confidence;

[0034] By fusing the round scores from the audio side and the round scores from the video side, a one-dimensional discriminant sequence is generated, represented as:

[0035]

[0036] in, and These represent the weights of the round scores on the audio side and the round scores on the video side, respectively.

[0037] Furthermore, the adaptive threshold Determined based on the smoothed one-dimensional discriminant sequence, expressed as:

[0038]

[0039] in, This represents the median of the smoothed one-dimensional discriminant sequence; and This represents the quartile statistic; Indicates the regulating factor;

[0040] Based on the adaptive threshold, a round state sequence is generated, represented as follows:

[0041]

[0042]

[0043] in, Indicates the first One-dimensional discriminant value after frame smoothing; Indicates the first The round state of a frame; when , indicates that the round is in progress; when This indicates that the process is in a turn interval state.

[0044] Furthermore, the candidate round time period is obtained through a state change sequence;

[0045] The start and end points of the rounds in the round state sequence are located to obtain the state changes. , is represented as:

[0046]

[0047] when When this occurs, it indicates a transition from the turn interval state to the turn in progress state;

[0048] when When, it indicates a transition from the round in progress state to the round interval state;

[0049] when When the state remains unchanged;

[0050] The set of all positions with a value of 1 in the state change sequence represents the start time of the round. All values ​​in the state change sequence are The position is the set at the end of the round. Pair adjacent round start times with round end times to form a set of candidate round time periods. ;

[0051] The three-level optimization process includes time-constrained filtering, adjacent merging, and boundary expansion.

[0052] Compared with the prior art, the advantages of the present invention are as follows:

[0053] 1. This invention unifies audio and video information into a one-dimensional discrimination sequence for processing, avoiding the high computational overhead caused by complex frame-by-frame visual understanding.

[0054] 2. This invention reduces misjudgments caused by noise or camera switching affecting a single modality by integrating the differences in narration voice energy with the stability characteristics of the main viewpoint camera.

[0055] 3. The algorithm of this invention is based on deterministic signal processing, statistical features and threshold discrimination, and can run in real time or near real time on general-purpose CPU devices.

[0056] 4. This invention can be adapted to various net sports by configuring the sports type parameter, and supports quick adjustments under different broadcast styles. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of the video round extraction and highlight generation method of the present invention;

[0059] Figure 2 This is a schematic diagram of the short-time energy sequence and smooth energy sequence of the present invention;

[0060] Figure 3 This is a flowchart of the main perspective confidence sequence generation process of the present invention;

[0061] Figure 4 This is a flowchart of the one-dimensional discrimination sequence and round state sequence generation process of the present invention;

[0062] Figure 5The flowchart for generating the optimized round time period set of the present invention is shown below. Detailed Implementation

[0063] To achieve the above objectives, this invention provides a method for extracting rounds and generating highlights from netball competition videos by integrating audio and video. Please refer to [link to relevant documentation]. Figures 1-5 The method includes:

[0064] S1: Acquire videos of tennis matches and extract audio signals and video frame sequences; segment the audio signal into frames and calculate short-time energy, then perform a three-stage cascaded filter to obtain a smooth energy sequence; extract image features based on the video frame sequence and calculate the main viewpoint confidence sequence.

[0065] This step is fundamental to extracting rounds from tennis match videos. Its purpose is to extract discriminative features from the original tennis match videos, both audio and video, providing a data foundation for subsequent fusion and discrimination.

[0066] Tennis includes net-based competitive sports such as table tennis, tennis, badminton, and volleyball. First, obtain videos of tennis matches. These videos include the rally phase and the intervals between rallies. The rally phase corresponds to the time when athletes are in a competitive state, from the start of the serve to the end of the dead ball phase. The intervals between rallies include scenes of athletes wiping sweat, retrieving the ball, breaks between games, timeouts, close-ups of spectators or coaches, and replay footage.

[0067] The video formats for tennis competition videos include, but are not limited to, mainstream formats such as MP4, AVI, MKV, and MOV. Audio signals and video frame sequences are extracted from the competition videos. The audio signals include commentary. When the competition video contains multiple audio tracks, the audio track containing the commentary is selected as the audio signal; when the competition video contains only a single audio track, that audio track is directly used as the audio signal. The audio signals are uniformly converted to mono, PCM format with a sampling rate of 16kHz or higher and 16-bit quantization precision to ensure consistency in subsequent processing.

[0068] Simultaneously, the video frame image sequence of tennis competition videos is extracted, and frame extraction is performed according to a preset frame rate or windowing is performed according to a preset time window.

[0069] The audio signal is processed by frame segmentation, which is performed using a sliding window method with fixed frame length and frame shift. In this embodiment, the frame length is set to 20-50ms and the frame shift is set to 40%-50% of the frame length to ensure smooth transition between frames and reduce information loss.

[0070] For the The short-time energy of a frame is obtained by calculating the sum of squares of the audio samples within the frame. , is represented as:

[0071]

[0072] in, Indicates the first The number of audio sample points within a frame; Indicates the first The amplitude values ​​at each sampling point; by repeating the above calculation for all frames, a short-time energy sequence is formed, represented as:

[0073]

[0074] in Indicates the first The timestamp corresponding to a frame is calculated from the frame sequence number and the frame shift parameter;

[0075] A three-stage cascaded filtering process is performed on the short-time energy sequence to suppress noise impulses, smooth the energy curve, and extract the energy trend. The three-stage cascaded filtering includes nonlinear filtering, low-pass filtering, and long-window averaging filtering. Specifically,

[0076] The first-level filtering uses nonlinear filtering to suppress impulse noise. In this embodiment, a median filter is used with a window length of 5-11 sample points. The median filter takes the short-time energy of each frame in the short-time energy sequence before and after the frame. There are 10 adjacent points, totaling 1000 points. Count the points, sort them by value, and take the median as the filter output. Median filtering can effectively suppress sudden impulse noise in audio, such as sudden applause, whistles, or equipment collisions, while preserving the main trend of energy sequence variation.

[0077] The secondary filter uses a low-pass filter for smoothing. In this embodiment, a Butterworth low-pass filter with a cutoff frequency of 0.1-0.5Hz is used. The Butterworth filter smooths the output of the primary filter. Frequency domain filtering is performed to remove high-frequency jitter components while preserving the low-frequency energy variation trend, resulting in a two-stage filter output. ;

[0078] The third-level filter uses a long-time-window averaging filter to extract the energy trend. In this embodiment, a moving average filter with a window length of 1-5 seconds is used. Frames, on the timeline Centered on the time taken forward and backward, the duration is... The time window in which Indicates the window length, extracts the second-level filter output of all frames within that time window, calculates the mean as the third-level filter output, i.e., the smoothing energy value. ;

[0079] By performing the above three-stage cascaded filtering process on all frames, a smooth energy sequence is finally obtained. This smooth energy sequence effectively characterizes the periodic energy differences corresponding to the relatively silent phase of the narration and the relatively active phase between phases.

[0080] Frame extraction is performed on the video frame sequence of netball competition videos to form a temporal input sequence on the video side, reducing computational load and ensuring the efficiency of subsequent feature extraction. In this embodiment, the frame extraction rate is set to 1-10fps. When the video frame rate is 25fps or 30fps, uniform sampling is performed according to the preset frame extraction rate; the first frame is extracted. Frame images and assign corresponding timestamps. ,

[0081] For the first sample Frame Image Image features are extracted to characterize the main viewpoint shot. The main viewpoint shot refers to a fixed-angle shot covering the main playing area of ​​the table or court. It typically occupies the majority of the time during a rally and is characterized by stable frame structure, a high proportion of the playing area, and relatively smooth overall motion. The image features include four categories, specifically:

[0082] The first category is brightness statistical features, for the... Frame Image The brightness value of all pixels is calculated using a grayscale conversion formula. Based on the brightness values ​​of all pixels, the frame brightness mean, brightness variance, peak percentage of the brightness histogram, and quantile difference are calculated. The frame brightness mean represents the average brightness of the entire image; the brightness variance represents the dispersion of pixel brightness, and the smaller the variance, the more uniform the brightness distribution of the image; the peak percentage of the brightness histogram represents the proportion of pixels corresponding to the highest peak in the brightness histogram to the total number of pixels; the quantile difference represents the difference between the 90th quantile and the 10th quantile of the brightness distribution, characterizing the dynamic range of the image.

[0083] The second category is color distribution characteristics, for the first... Frame Image For all pixels, hue and saturation components are extracted in the HSV color space, and a hue-saturation two-dimensional histogram is constructed. Based on this histogram, the dominant color ratio, color distribution entropy value, or color cluster center stability is calculated. The dominant color ratio represents the proportion of pixels corresponding to the color interval with the largest number of pixels. The color distribution entropy value is used to measure the uniformity of color distribution; the smaller the entropy value, the more concentrated the color distribution. The color cluster center stability is evaluated by matching the color cluster centers of adjacent frames and calculating the Euclidean distance or cosine similarity of the center positions to assess the temporal stability of the color distribution.

[0084] The third category is global motion features, which are applied to two adjacent frames. and Global motion is estimated using optical flow methods. In this embodiment, the Farneback dense optical flow algorithm is used to calculate the pixel-by-pixel optical flow vector field. ,in and Representing positions respectively The motion velocities of pixels in the horizontal and vertical directions are calculated, and the global motion energy is calculated based on the optical flow vector field. , is represented as:

[0085]

[0086] in, This represents the total number of pixels in the image; summation iterates through all pixel positions in the image; global motion energy represents the overall motion amplitude of the scene. During the main-view shot, the camera usually remains relatively still or only shakes slightly, so the global motion energy is relatively small; while during shot transitions or playback, the global motion energy increases significantly.

[0087] The fourth category is regional proportion characteristics, for the first Frame Image The system employs methods such as geometric structure detection, color segmentation, or semantic segmentation to locate the playing area of ​​the table tennis table or court. In this embodiment, for table tennis, a table edge detection method based on Hough line transform is used to extract the rectangular area of ​​the table; for tennis or badminton, a threshold segmentation method based on prior court color is used to extract the court area. After the playing area is located, the ratio of the playing area area to the total image area is calculated to obtain the playing area proportion. The playing area proportion feature characterizes the degree of coverage of the playing area by the main viewpoint camera. The main viewpoint camera usually makes the playing area occupy the center of the image and has a high proportion, while the playing area proportion of close-up shots or spectator shots is low or zero.

[0088] Construct a main viewpoint scoring function based on image features, and output the first... Confidence of the main viewpoint of the frame image The confidence level of the main viewpoint is composed of the brightness stability sub-confidence. Confidence level of motion stability and regional proportion of sub-confidence The result obtained through weighted fusion is expressed as:

[0089]

[0090] in, , and Let represent the weighting coefficients of the brightness stability sub-confidence, motion stability sub-confidence, and region proportion sub-confidence, respectively, and satisfy the condition that their sum is 1; the specific calculation method for each sub-confidence is as follows:

[0091] Brightness stability sub-confidence Based on luminance statistical features, in this embodiment, a luminance distribution model is established for the luminance mean of all frames across the entire video range, and the global mean of the luminance mean is calculated. Compared with global standard deviation , for the Frame, calculate the average brightness Deviation from global mean And normalized to a confidence score, expressed as:

[0092]

[0093] This formula uses a Gaussian kernel function form, when the... When the average brightness of a frame is close to the global average, A value close to 1 indicates high brightness stability for that frame; when the deviation is large, A decrease in the frame indicates that the frame may be a non-first-person perspective shot.

[0094] The motion stationarity sub-confidence is calculated based on global motion features. In this embodiment, a statistical measure of global motion energy, including the median, is calculated for all frames across the entire video. Interquartile range , for the Frame, based on global motion energy The relationship with the statistic is used to calculate the confidence score, which is expressed as:

[0095]

[0096] When the When the global motion energy of a frame is low, A value close to 1 indicates that the frame is stable; when the global motion energy is high, A lowered value indicates that the frame may be a camera cut or a fast-moving scene;

[0097] The regional proportion sub-confidence is calculated directly based on the regional proportion feature of the competition. In this embodiment, a linear mapping method is used to represent the regional proportion of the competition. The mapping is to a confidence score, expressed as:

[0098]

[0099] in, This represents the threshold for the typical playing area proportion of a main-view shot; for example, it could be set to 0.3-0.5 for a table tennis match. Reaching or exceeding hour, =1; when When smaller, Reduce proportionally;

[0100] All frames are associated with and stored with their corresponding timestamps to form a first-view confidence sequence. This sequence represents the visual difference characteristics of the video footage, which maintains a stable first-view during the round and switches frequently between rounds.

[0101] S2: Align the smoothed energy sequence with the main view confidence sequence, and fuse the aligned smoothed energy sequence with the main view confidence sequence to generate a one-dimensional discriminant sequence; smooth the one-dimensional discriminant sequence and generate a round state sequence based on an adaptive threshold;

[0102] The purpose of this step is to fuse the smooth energy sequence on the audio side with the main view confidence sequence on the video side. The smooth energy sequence is used to characterize the periodic difference of the narration voice being relatively silent during the round and relatively active during the round interval. The main view confidence sequence is used to characterize the visual difference of the video image maintaining a stable main view during the round and frequently switching during the round interval. This step fuses the round silence and interval activity differences on the audio side with the main view stability differences on the video side into a unified one-dimensional discriminant sequence, and further generates a round state sequence based on an adaptive threshold.

[0103] Because the audio and video signals have different sampling rates, the smoothed energy sequence and the main viewpoint confidence sequence need to be aligned to a unified time base. The sampling rate of the audio-side smoothed energy sequence is determined by the framing parameters. When the frame length is 20-50ms and the frame shift is 40%-50% of the frame length, the corresponding sampling rate is approximately 40-125Hz. The sampling rate of the video-side main viewpoint confidence sequence is the frame rate of the subtraction frame, typically 1-10fps, which is much lower than the sampling rate of the audio-side.

[0104] Alignment is performed based on the video side with the lower sampling rate, for the first... Timestamp of frame image The energy values ​​for the corresponding time periods are extracted from the smooth energy sequence and aggregated. In this embodiment, an interval aggregation method is used to aggregate the energy values ​​for the time periods. The smoothed energy values ​​within the range are averaged to obtain aligned energy values. For example, when the frame rate on the video side is 2fps, there are approximately 20-60 smoothed energy values ​​within each 0.5s time interval. Averaging these values ​​effectively smooths short-term fluctuations while preserving energy trends. In another embodiment, a central window approach is used... Centered on the video sample interval, smooth energy values ​​within a time window with a width equal to the video sampling interval are extracted and aggregated.

[0105] After time alignment, an aligned energy sequence is formed. To ensure that the two sequences have comparable numerical ranges, the aligned smooth energy sequence and the main view confidence sequence are normalized. In this embodiment, the min-max normalization method is used. After normalization, the numerical ranges of both the smooth energy sequence and the main view confidence sequence are mapped to... The interval is defined as follows: a larger smooth energy value indicates higher energy, and a larger main view confidence value indicates higher main view confidence.

[0106] A one-dimensional discriminant sequence is obtained by fusing the normalized smooth energy sequence and the main perspective confidence sequence according to weights. The audio-side round score sequence is constructed based on the normalized smooth energy sequence. Since the narration is relatively silent during the round's progress and relatively active during the round's interval, low-energy periods correspond to the round's progress, and high-energy periods correspond to the round's interval. To ensure a positive correlation between the sub-sequence and the round's progress state, the normalized energy is inverted, as shown below:

[0107]

[0108] in, Indicates the first Round scores on the frame audio side; Indicates the first Smoothing energy after frame normalization; The larger the value, the more likely it is to be in the ongoing phase, and the corresponding commentary is relatively silent.

[0109] The round score sequence on the video side is constructed based on the normalized first-person perspective confidence sequence. Since the first-person perspective shots have a high and stable proportion in the round combat phase, the periods with high first-person perspective confidence correspond to the round progress phase. The normalized first-person perspective confidence is directly used as the round score on the video side, expressed as:

[0110]

[0111] in, Indicates the first Round scores on the frame video side; Indicates the first Frame-normalized first-person perspective confidence; The larger the value, the more likely it is to be in a first-person perspective turn-based confrontation phase;

[0112] By fusing the round scores from the audio side and the round scores from the video side, a one-dimensional discriminant sequence is generated, represented as:

[0113]

[0114] in, and These represent the weights of the round scores on the audio side and the round scores on the video side, respectively; in this embodiment, a fixed weight fusion is used, for example, setting... =0.6, =0.4, to reflect the dominant role of the main perspective confidence in round discrimination; in another embodiment, adaptive weight fusion is adopted, and the weights are dynamically adjusted according to the reliability of each modality. When the audio discrimination is unstable due to audience noise, background music, or continuous speaking, the variance or fluctuation frequency of the smooth energy sequence is detected to reduce the impact. And improve When the main-view shot is affected by frequent switching or broadcast playback, causing instability in video-side discrimination, the amplitude of jumps in the main-view confidence sequence or the proportion of low-confidence segments can be detected to reduce the impact. And improve .

[0115] The one-dimensional discriminant sequence is smoothed to suppress short-term fluctuations and preserve the main trend; median filtering is used for smoothing, with a window length of 3-7 sampling points. Each sampling point is taken before and after it. There are 10 adjacent points, totaling 1000 points. The discriminant values ​​of these points are sorted by size and the median is taken as the smoothed output. Through median filtering, a smoothed one-dimensional discriminant sequence is obtained, which can effectively eliminate isolated high or low value noise points and maintain the main trend characteristics of the discriminant sequence.

[0116] Based on the statistical characteristics of the smoothed one-dimensional discriminant sequence, an adaptive threshold is determined. Statistical features include the median of the smoothed one-dimensional discriminant sequence. With quartile statistics and The adaptive threshold is expressed as:

[0117]

[0118] in, Indicates the moderating factor related to exercise type; median This represents the middle value after all smoothed one-dimensional discriminant values ​​are sorted by size; quartile statistic. This represents the 25th percentile. The 75th percentile represents the interquartile range, which characterizes the dispersion of the discriminant sequence; the adjustment factor... Used to control adaptive threshold The offset relative to the median, The larger the value, the higher the threshold. The higher the confidence level, the more likely the discrimination will identify high-confidence round segments; The smaller the value, the lower the threshold. The lower the value, the more lenient the judgment; for different types of net sports, The value can be set based on experience, such as the settings for a table tennis match. =0.3-0.5, tennis match settings =0.4-0.6;

[0119] Based on an adaptive threshold, the smooth discrimination sequence is binarized for discrimination, generating a round state sequence. The discrimination rule is as follows.

[0120] when hour, (Round in progress)

[0121] when hour, (Round Interval Status)

[0122] in, Indicates the first One-dimensional discriminant value after frame smoothing; Indicates the first The round state of a frame; the round state sequence divides the continuous time axis into round execution segments and round interval segments, providing a binary state marker for subsequent round positioning.

[0123] S3: Perform round location on the round state sequence to obtain candidate round time periods; based on the candidate round time periods, perform duration constraint filtering, adjacent merging, and boundary expansion processing to obtain an optimized set of round time periods; generate match highlights based on the optimized set of round time periods.

[0124] The purpose of this step is to locate candidate round time segments based on the round state sequence, and to optimize them through duration constraints, adjacent merging, and boundary expansion processing. Finally, round segments are extracted from tennis match videos to generate match highlights.

[0125] Determine the start and end points of a round based on the round state sequence; Perform first-order difference calculations to obtain the state changes. , is represented as:

[0126]

[0127] when When, it indicates a transition from the turn interval state to the turn in progress state, corresponding to the start edge of the turn; when When, it indicates a transition from the ongoing round state to the intermission round state, corresponding to the edge of the round's end; when When the state remains unchanged;

[0128] By detecting all positions with a value of 1 in the state change sequence, we obtain the set of times at the start of the round. By detecting all positions in the state change sequence that take a value of -1, the set of times at the end of the round is obtained. Pair adjacent round start times with round end times to form a set of candidate round time periods. .

[0129] A three-level optimization process is performed on the candidate round time period, including duration constraint filtering, adjacent merging, and boundary expansion. Specifically,

[0130] 1) Duration-constrained filtering, calculate the first... Duration of each candidate round , represented as

[0131]

[0132] Set round duration constraints ,in Indicates the shortest duration of an effective round. This represents the maximum duration of a valid rally, and the duration constraint range is configured according to the rally characteristics of different tennis sports; in this embodiment, for table tennis, it is set as follows: =3s、 =60s, to filter out falsely detected segments that are too short (such as the instant of a camera transition) and abnormal segments that are too long (such as interruptions or pauses in the match); for tennis matches, set to =5s、 =180s, to adapt to the generally longer rallies in tennis matches; for badminton matches, set to =4s、 =120s;

[0133] Iterate through the set of candidate round time periods and retain those that meet the requirements. The candidate rounds are selected, and those that do not meet the conditions are removed to obtain the set of candidate round time periods after being filtered by the duration constraint.

[0134] 2) Adjacent merging: For the candidate round time period set after time constraint filtering, adjacent round merging is performed; the candidate round time period set is traversed, and for two adjacent candidate rounds, the time interval between them is calculated and a merging threshold is set. When the time interval is less than the merging threshold, it is considered that the interval between the two candidate rounds is too short, which may be due to a brief shot switch, replay insertion or judgment fluctuation, causing the complete round to be incorrectly segmented. Therefore, the two candidate rounds are merged into one consecutive round. This merging operation is repeated until there are no more adjacent round pairs that meet the merging conditions in the candidate round time period set.

[0135] 3) Boundary expansion: For the set of candidate round time segments after adjacent merging, boundary expansion is performed to ensure that each round segment contains the complete match action sequence. Expansion includes forward expansion and backward expansion.

[0136] Forward expansion is used to include initial actions such as serve preparation, shifting the start time of the rally forward to obtain the expanded start time of the rally; backward expansion is used to retain ending actions such as scoring reactions, shifting the end time of the rally backward to obtain the expanded end time of the rally.

[0137] A boundary expansion operation is performed on all candidate rounds after adjacent merging to obtain an optimized round time period set. The optimized round time period set filters out false detection segments and ensures the integrity and watchability of each round.

[0138] Based on the optimized set of round time periods, corresponding video segments are extracted from tennis match videos. Video processing tools such as FFmpeg are used to perform precise time period segmentation operations on the tennis match videos. Each optimized round time period Extracting time ranges from tennis competition videos The video clips were saved as separate temporary video files.

[0139] Subsequently, the optimized set of rally time periods was sorted. This could be done by sorting chronologically to maintain the continuity of the match, or by sorting by rally duration to prioritize exciting long rallies, or by combining rally duration and rally score for a comprehensive sort to select the top ten rallies.

[0140] Then, all the sorted round segments are spliced ​​together in sequence to generate a complete match highlights video. In this embodiment, fade-in and fade-out transitions are added between adjacent round segments, with a transition duration of 0.5 seconds, to improve the smoothness of the viewing experience. In another embodiment, particularly exciting scoring moments are slowed down, for example, by slowing down the shot by 2x speed 1 second before the end of the round to highlight the exciting moments.

[0141] Finally, the spliced ​​highlight video is output in mainstream formats such as MP4, AVI, and MOV, completing the automatic generation of match highlights. The generated highlight video retains all the exciting rounds of the match, and through audio and video fusion discrimination, it effectively reduces false detections and missed detections caused by noise or camera switching in a single modality, improving the robustness and applicability of round recognition.

[0142] This invention proposes a method for extracting rounds and generating highlights from tennis match videos by fusing audio and video. By mining the inherent complementary audio-video patterns in tennis match broadcast videos, it achieves efficient and robust automated round recognition. First, audio signals and video frame sequences are extracted from the match video. Based on the pattern that commentary is relatively silent during rounds and relatively active during intervals, the audio is segmented, short-time energy is calculated, and multi-level cascaded filtering is performed to construct a smooth energy sequence representing the periodic differences in speech activity. Simultaneously, based on the characteristics of stable image structure, high proportion of the competition area, and stable global motion in the main viewpoint during rounds, multi-dimensional image features such as brightness, global motion, and area proportion are extracted from the video frames and weighted and fused to generate a main viewpoint confidence sequence representing visual stability. Next, the two sequences are time-aligned and normalized, and according to a dynamic weighted fusion strategy, the audio-side round score and the video-side round score are combined into a unified one-dimensional discriminant sequence. This sequence is then smoothed and an adaptive threshold based on its statistical distribution is used to generate an accurate round state sequence. Finally, edge detection is performed on the state sequence to locate candidate rounds, and then project-based duration constraint filtering, anti-segmentation adjacent merging, and boundary expansion optimization for complete actions are performed in sequence to obtain an accurate set of round time segments. Based on this, video clips are automatically extracted, transitions and effects are added to generate high-quality match highlights.

[0143] In summary, this invention transforms the complex round recognition problem into signal processing of a one-dimensional discriminative sequence, avoiding the high computational overhead and deployment difficulties associated with relying on frame-by-frame complex visual understanding or deep learning models in existing technologies. By complementary fusion of audio and video dual-modal features, it effectively suppresses false positives and false negatives caused by interference from noise, continuous narration, camera switching, or playback in a single modality, significantly improving the accuracy, robustness, and cross-project applicability of round recognition. This method is computationally lightweight and logically clear, and can run efficiently on general-purpose computing devices, providing a practical solution for the automated editing and content generation of sports videos.

[0144] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for extracting rounds and generating highlights from online sports competition videos that integrates audio and video, characterized in that, The method includes: S1: Acquire videos of tennis matches and extract audio signals and video frame sequences; perform frame-by-frame processing on the audio signals to calculate short-time energy, and further perform three-stage cascaded filtering to obtain a smooth energy sequence; extract image features based on the video frame sequences and calculate the main viewpoint confidence sequence; S2: Align and normalize the smoothed energy sequence with the main view confidence sequence, and fuse the normalized smoothed energy sequence with the main view confidence sequence to generate a one-dimensional discriminant sequence; smooth the one-dimensional discriminant sequence and generate a round state sequence based on an adaptive threshold; S3: Perform round location on the round state sequence to obtain candidate round time periods; perform three-level optimization processing based on the candidate round time periods to obtain an optimized round time period set; generate match highlights based on the optimized round time period set.

2. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 1, characterized in that, The tennis competition video includes the round-run phase and the round-interval phase; The audio signal includes an audio track containing narration; The framing process uses a sliding window method with a fixed frame length and frame shift to obtain short-time energy, represented as: in, Indicates the first The short-time energy of a frame; Indicates the first The number of audio sample points in a frame; Indicates the first The amplitude value of each sampling point; perform frame segmentation processing on all frames in the audio signal to obtain the short-time energy sequence.

3. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 2, characterized in that, The three-stage cascaded filtering includes nonlinear filtering, low-pass filtering, and long-window averaging filtering; The nonlinear filter performs impulse noise suppression on the short-time energy sequence as a first-level filter. The low-pass filter smooths the short-time energy sequence after the first-stage filter as a second-stage filter. The long-time window average filter extracts the short-time energy trend after the second-level filter as the output of the third-level filter, and the output of the third-level filter is a smoothed energy value. A three-stage cascaded filtering process is performed on all frames in the short-time energy sequence to generate a smooth energy sequence.

4. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 3, characterized in that, The video frame sequence is subjected to frame extraction processing to form a time-series input sequence on the video side, represented as follows: in, Indicates the first Frame image; Represented as the first The corresponding timestamp assigned to the frame image; The image features include brightness statistics features, color distribution features, global motion features, and region proportion features; Based on the image features, a main viewpoint scoring function is constructed, and the first output is... Confidence of the main viewpoint of the frame image The confidence level of the main viewpoint is composed of the brightness stability sub-confidence level. Confidence level of motion stability and regional proportion of sub-confidence The result obtained through weighted fusion is expressed as: in, , and The weights of the brightness stability sub-confidence, motion stability sub-confidence, and region proportion sub-confidence are respectively represented; the main viewpoint confidence of all frames in the temporal input sequence is calculated to generate the main viewpoint confidence sequence.

5. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 4, characterized in that, The alignment is based on the video side and targets the first [unclear] in the video side. The timestamp of the frame image is used to extract the energy value of the corresponding time period from the smooth energy sequence and aggregate it to obtain the aligned smooth energy sequence. The aligned smooth energy sequence and the main view confidence sequence are then normalized.

6. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 5, characterized in that, Based on the normalized smooth energy sequence, the round score on the audio side is obtained, expressed as: in, Indicates the first Round scores on the frame audio side; Indicates the first The smoothed energy value after frame normalization; Based on the normalized first-person confidence sequence, the round score on the video side is obtained, expressed as: in, Indicates the first Round scores on the frame video side; Indicates the first Frame-normalized first-person perspective confidence; By fusing the round scores from the audio side and the round scores from the video side, a one-dimensional discriminant sequence is generated, represented as: in, and These represent the weights of the round scores on the audio side and the round scores on the video side, respectively.

7. The method for extracting rounds and generating highlights from netball sports competition videos by integrating audio and video as described in claim 6, characterized in that, The adaptive threshold Determined based on the smoothed one-dimensional discriminant sequence, expressed as: in, This represents the median of the smoothed one-dimensional discriminant sequence; and This represents the quartile statistic; Indicates the regulating factor; Based on the adaptive threshold, a round state sequence is generated, represented as follows: in, Indicates the first One-dimensional discriminant value after frame smoothing; Indicates the first The round state of a frame; when , indicates that the round is in progress; when This indicates that the process is in a turn interval state.

8. The method for extracting rounds and generating highlights from netball competition videos by integrating audio and video as described in claim 7, characterized in that, The candidate round time period is obtained through a state change sequence; The start and end points of the rounds in the round state sequence are located to obtain the state changes. , is represented as: when When this occurs, it indicates a transition from the turn interval state to the turn in progress state; when When, it indicates a transition from the round in progress state to the round interval state; when When the state remains unchanged; The set of all positions with a value of 1 in the state change sequence represents the start time of the round. All values ​​in the state change sequence are The position is the set at the end of the round. Pair adjacent round start times with round end times to form a set of candidate round time periods. ; The three-level optimization process includes time-constrained filtering, adjacent merging, and boundary expansion.

Images