The application relates to a basketball video highlight automatic generation method and
system based on multi-
modal feature fusion, and belongs to the technical field of
computer vision and
multimedia intelligent analysis. The method comprises the following steps: S1, extracting and aligning a visual feature sequence and an audio feature sequence; S2, using bidirectional LSTM to perform
time sequence coding on the audio-visual feature, constructing a cross-
modal attention mechanism taking audio as a query and vision as a key value, and obtaining multi-
modal fusion features; S3, inputting the fusion features into a 1D CNN to output a frame-by-frame highlight event probability sequence; S4, performing
peak value searching based on an adaptive threshold of the probability sequence to obtain a coarse-grained
timestamp; S5, taking the coarse-grained
timestamp as a center to intercept local features, inputting the local features into bidirectional LSTM to predict starting and ending offsets, and obtaining accurate start and end time boundaries; and S6, clipping and splicing to generate a highlight video according to the accurate boundaries. The application realizes audio-visual semantic depth alignment through an audio-guided
visual attention mechanism, and improves boundary positioning accuracy through a coarse-precision two-stage positioning architecture.