News stripping control method and system based on multimodality

Abstract

The present invention relates to the field of data processing and artificial intelligence technology, and discloses a news stripping control method and system based on multimodality, including: receiving a news short video stream, and splitting the news short video stream into multimodal data; generating corresponding semantic representations based on the multimodal data, fusing the features of each modality to form a unified semantic representation, and generating an initial stripping boundary list; performing multimodal semantic consistency verification on each segment corresponding to the initial segmentation boundary, and optimizing the initial segmentation boundary in combination with the quality assessment results of each segment, and outputting high-quality segments. The accuracy and real-time performance of stripping are significantly enhanced; combined with the structural consistency verification mechanism, the semantic coherence and structural rationality of the stripping segments are effectively guaranteed; further introducing dynamic semantic drift control and comprehensive segment quality assessment guided by cyclic consistency, the multimodal semantic consistency and content quality of the output segments are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing and artificial intelligence technology, and in particular to a news stripping control method and system based on multimodality. Background Art

[0002] The rapid development of short video platforms and smart media technologies has profoundly transformed the production and distribution of news content. Traditional long-form news videos are gradually being replaced by shorter, more user-friendly formats. News segmentation technology, which automatically breaks down complete news videos into several semantically complete, thematically clear short video segments, has become a core component of content operations.

[0003] Existing news segmentation methods mostly rely on rule-based, shallow feature extraction methods, primarily detecting surface features such as screen switching, silent segments, keyframe changes, or voice pauses to determine the boundaries of video segments. However, such methods have significant limitations: First, surface features cannot effectively reflect the true semantic structure of news content, which can easily lead to a mismatch between segmentation boundaries and semantic turning points, resulting in semantic discontinuities or information omissions. Second, single-modality-driven segmentation ignores the synergistic relationship between multimodal information such as audio, video, and text, making it difficult to address the boundary determination challenges brought about by multimodal information interaction in complex news scenarios. Furthermore, most existing technologies use static thresholds or fixed rules and lack the ability to perceive dynamic changes in contextual semantics. This results in a lack of flexibility and intelligence in segmentation results, making it difficult to ensure the semantic integrity and audiovisual coordination of the output segments. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is that the existing news segmentation method lacks multimodal deep semantic analysis and dynamic control mechanism, resulting in inaccurate boundary judgment, poor semantic coherence and inability to effectively guarantee segment quality.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a multimodal news stripping control method, comprising: receiving a news short video stream, and splitting the news short video stream into multimodal data;

[0007] Generate corresponding semantic representation based on multimodal data, fuse the features of each modality to form a unified semantic representation, and generate an initial striping boundary list;

[0008] For each segment corresponding to the initial segmentation boundary, multimodal semantic consistency verification is performed, and the initial segmentation boundary is optimized based on the quality assessment results of each segment to output high-quality segments.

[0009] As a preferred solution of the multi-modal based news strip control method described in this invention, wherein: the receiving of the news short video stream includes setting the sliding window length to H seconds and the sliding step to K seconds, satisfying 0 < K < H; and intercepting a news short video stream segment of H seconds every K seconds for preprocessing; where H is the sliding window length, representing the duration of each intercepted segment; K is the sliding step, representing the time interval between two adjacent interceptions.

[0010] The multi-modal data includes visual modality, audio modality and text modality data.

[0011] Extract video frames from the preprocessed news short video stream at a preset frame rate, and perform normalization processing on the video frames to generate visual features ;

[0012] Extract audio modality data from the preprocessed news short video stream at a fixed sampling frequency, convert it into a mel spectrogram, and generate audio features , and call an automatic speech recognition model to generate a text transcription result when there is no subtitle in the news short video stream;

[0013] Based on the text transcription result, use a word segmentation model to generate a text embedding feature T;

[0014] At each time step, combine to generate the corresponding multi-modal feature vector ; where represents the visual feature at time step , represents the audio feature at time step , represents the text embedding feature at time step ; preprocess the multi-modal feature vectors for all time steps and construct a multi-modal feature sequence ; represents the multi-modal feature vector corresponding to the th time step; represents the th time step, represents the time step number.

[0015] As a preferred solution of the multimodal news stripping control method described in the present invention, the unified semantic representation includes receiving the multimodal feature sequence, including visual features, audio features and text embedding features; using linear mapping functions to convert the visual features, audio features and text embedding features into semantic spaces of the same dimension, and adding time position coding to each modal feature vector; adding the linearly mapped visual features, audio features and text embedding features to the corresponding time position coding to obtain a multimodal fusion input vector for each time step.

[0016] As a preferred solution of the multimodal news stripping control method of the present invention, the unified semantic representation is input into a multimodal deep semantic transformer to perform deep semantic modeling and identify semantic changes; based on the semantic changes, candidate boundary time steps are identified to generate an initial stripping boundary list;

[0017] The multimodal deep semantic transformer introduces a trainable gating factor to adjust the fusion ratio of text embedding features, visual features and audio features according to contextual semantics, thereby achieving adaptive deep fusion of multimodal features. Through the modal attention change monitoring mechanism, the modal attention entropy is embedded in the cross-attention process, the changes in modal dominance are perceived in real time, and semantic mutation detection is dynamically triggered to optimize the determination of stripping boundaries. According to the preset structural consistency verification mechanism, a local temporal semantic relationship graph is constructed based on the boundary candidate time step, and the graph structure modeling is performed in combination with the edge weight. The semantic coherence and structural rationality of the boundary candidate time step are verified through the graph neural network.

[0018] As a preferred solution of the multimodal news stripping control method described in the present invention, the multimodal deep semantic transformer architecture includes a backbone encoding layer, a gated cross-attention fusion layer, a modal attention change monitoring mechanism, and a structural consistency verification mechanism;

[0019] The backbone encoding layer is composed of a preset number of layers of Transformer encoder stacks, which captures long-range dependencies in time series based on the self-attention mechanism and extracts global semantic features;

[0020] Each a-layer Transformer encoder is inserted into a gated cross attention fusion layer, with text modality features as query vectors, visual modality and audio modality features as keys and values, to perform cross-modal attention calculations, and based on the trainable gating factor Dynamically adjust the information injection ratio of each modal feature;

[0021] Wherein, a represents the preset number of interval layers; Indicates the A trainable gating factor for each time step.

[0022] As a preferred solution of the multimodal news stripping control method described in the present invention, the modal attention change monitoring mechanism is embedded in the gated cross attention fusion layer, which calculates the attention distribution of text, visual and audio modal features in each time step, and calculates the modal attention entropy based on the attention distribution. , and in consecutive time steps, when the modal attention entropy change amplitude is detected Exceeding the preset threshold When , local semantic trajectory offset detection is performed;

[0023] The structural consistency verification mechanism includes constructing a local temporal semantic relationship graph based on the deep semantic representation sequence after the local semantic trajectory offset detection. , combining edge weights to model graph structure, and performing weighted feature aggregation and structural analysis through a two-layer lightweight graph neural network to calculate the structural fracture confidence of the candidate stripping boundary , verify the semantic coherence and structural rationality of the boundary area; among them, represents a set of points, represents the edge set, Indicates the The confidence level of structural fracture at each time step.

[0024] As a preferred embodiment of the multimodal news segmentation control method described in the present invention, the multimodal semantic consistency includes: constructing a cross-modal mapping relationship with the text modality as the core based on each segment corresponding to the initial segmentation boundary; mapping the text features to the visual modality space and the audio modality space through a cross-modal encoder, and then reversely mapping them back to the text space from the respective modal spaces, and comparing the difference between the original text features and the reversely mapped features as a cycle consistency loss value;

[0025] Adjusting the semantic topic drift detection threshold according to the cycle consistency loss value , the formula is:

[0026]

[0027] in, represents the basic drift threshold, is the control coefficient; represents the exponential function, Representation fragment The cycle consistency loss value of ; Indicates the jth strip segment; j represents the segment index, Representation fragment The semantic topic drift detection threshold;

[0028] Based on the deep semantic representation sequence of the fragment, the cosine similarity between consecutive time points is calculated. When the cosine similarity drops below the set threshold, the fragment semantic coherence is determined to be insufficient, and the current strip boundary is redefined.

[0029] A news stripping control system based on multi-modality, wherein:

[0030] A data module receives a news short video stream and splits the news short video stream into multimodal data;

[0031] The initial segmentation boundary module generates corresponding semantic representations based on multimodal data, fuses the features of each modality to form a unified semantic representation, and generates an initial split boundary list;

[0032] The output module performs multimodal semantic consistency verification on each segment corresponding to the initial segmentation boundary, and optimizes the initial segmentation boundary based on the quality assessment results of each segment to output high-quality segments.

[0033] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0034] A computer-readable storage medium stores a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0035] Beneficial effects of the present invention: The multimodal news stripping control method provided by the present invention realizes the adaptive deep fusion of video, audio and text features by introducing a multimodal deep semantic transformer, thereby improving the semantic modeling accuracy and the context perception ability of boundary judgment in the news stripping process; through the modal attention change monitoring mechanism and dynamically triggered semantic mutation detection, it avoids the boundary misjudgment problem caused by traditional static rules, and significantly enhances the accuracy and real-time performance of stripping; combined with the structural consistency verification mechanism, it effectively guarantees the semantic coherence and structural rationality of the stripping segments; further introduces the dynamic semantic drift control and segment quality comprehensive evaluation guided by cyclic consistency, thereby improving the multimodal semantic consistency and content quality of the output segments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1Overall flowchart of the multi-modal based news strip control method provided for the first embodiment of the present invention. Detailed implementation manners

[0038] To make the above objects, features and advantages of the present invention more obvious and understandable, the following will describe the detailed implementation manners of the present invention in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0039] Example 1, referring to Figure 1 , which is an embodiment of the present invention, provides a multi-modal based news strip control method, including:

[0040] S1: Receive a news short video stream and split the news short video stream into multi-modal data.

[0041] Receive the news short video stream through a streaming media protocol, and dynamically process the video stream using a sliding window mechanism. Set the sliding window length to H seconds and the sliding step to K seconds, where 0 < K < H; and intercept a news short video stream segment of H seconds every K seconds for preprocessing; where H is the sliding window length, representing the duration of each intercepted segment; K is the sliding step, representing the time interval between two adjacent interceptions; realizing the streaming processing of the continuous video stream, reducing system latency and memory pressure.

[0042] The multi-modal data includes visual modality, audio modality and text modality data.

[0043] For the accessed video stream, the system synchronously extracts visual modality, audio modality and text modality data.

[0044] Extracting visual modality data includes extracting video frames from the preprocessed news short video stream at a preset frame rate Adjusting each extracted frame image uniformly to 224×224 pixels; performing normalization processing on the pixel values and mapping them to the interval [0, 1].

[0045] Extract audio modality data, extract the video audio track within the corresponding time period from the preprocessed news short video stream at a fixed sampling frequency, and the fixed sampling frequency can be set to 16kHz.

[0046] [[ID=3۳]]Convert the audio signal into a Mel spectrogram using the short-time Fourier transform to extract audio features; when there is no subtitle information, call the automatic speech recognition (ASR) model to generate a text transcription result.

[0047] Extracting text modal data includes parsing and extracting time-aligned subtitle text when the video contains a subtitle track; performing BERT word segmentation on the transcribed text or subtitle text to form a token sequence, adding [CLS] and [SEP] tags to obtain text embedding features; limiting the length of the text sequence, and truncating the excess part using a sliding window.

[0048] At each time step Within, the corresponding multimodal feature vector is generated by combination:

[0049]

[0050] in, Represents the time step The visual characteristics of Represents the time step The audio characteristics of Represents the time step The text embedding features of the multimodal data are aligned in time steps using a repeated filling strategy to ensure that each time step The system contains complete trimodal features to avoid fusion deviation caused by missing features. After completing the time alignment, in order to improve the stability of feature data and model processing efficiency, the system performs and audio characteristics Perform mean-variance normalization processing respectively to normalize the characteristic values ​​to the standard normal distribution interval to eliminate the influence of the scale differences of different modal characteristics. The formula is as follows:

[0051]

[0052] in, represents the feature mean, represents the characteristic standard deviation, is the original feature vector.

[0053] At the same time, the text modality is mapped into a fixed-dimensional vector representation through the embedding layer to ensure a balance between dimensional consistency and semantic expression ability. A standardized and time-aligned multimodal feature sequence is constructed in consecutive time steps. The overall structure is expressed as:

[0054]

[0055] in, Represents normalized and time-aligned multimodal feature sequences; Indicates in The multimodal feature vector corresponding to each time step; Indicates the time steps, represents the time step number, Indicates the total number of time steps.

[0056] Furthermore, to address the shortcomings of existing technologies in news video processing, which are single-minded and lack multimodal collaboration and real-time, efficient processing capabilities, a method for dynamic segmentation of news short video streams and synchronous extraction of multimodal features based on a sliding window mechanism is proposed. By setting the sliding window length and step size, streaming processing of continuous video streams is achieved, significantly reducing system latency and memory usage, and improving the real-time performance and resource utilization of large-scale news video processing. At the same time, the method for extracting multimodal data is improved, supporting the synchronous acquisition of visual, audio, and text modal features within the same time step, avoiding the feature alignment deviation problem caused by asynchronous modal processing in traditional methods.

[0057] Furthermore, after completing multimodal feature extraction, a time-step-based alignment strategy and mean-variance normalization were introduced to address the heterogeneity of different modal features in scale and time sequence, ensuring a high degree of consistency and stability in feature fusion within each time step. Compared to existing technologies that rely solely on static frame extraction or single-modal feature input, this effectively improves the integrity of multimodal feature expression and model processing efficiency, providing standardized, unified, high-quality input for subsequent deep semantic analysis and boundary determination, significantly enhancing the accuracy, robustness, and overall system performance of the news segmentation process.

[0058] S2: Generate corresponding semantic representations based on multimodal data and fuse the features of each modality to form a unified semantic representation to determine the initial segmentation boundary.

[0059] Receive the normalized and time-aligned multimodal feature sequence output by step S1:

[0060]

[0061] in, 、 、 They are visual features, audio features, and text embedding features. The system unifies different modal features into a semantic space of the same dimension through linear mapping, which can be expressed as:

[0062]

[0063] Combined with time step position encoding Generate the fused input vector:

[0064]

[0065] in, represents the overall multimodal feature sequence, Indicates that at time step A single multimodal feature vector at represents the index of the i-th time step; Represents the time step The visual feature vector of Represents the time step The audio feature vector of Represents the time step The text embedding feature vector of Representing visual features Through linear mapping The result after Representing audio features Eigenvector after linear mapping; Represents text embedding features Eigenvector after linear mapping; , , They represent the linear mapping weight matrices corresponding to visual, audio, and text modalities, respectively, and are used to transform the original features of different modalities into a unified semantic space of the same dimension, facilitating subsequent feature fusion and deep processing. Represents the time step The multimodal fusion input vector at .

[0066] The fused multimodal input vector sequence is fed into the Multimodal Deep Semantic Transformer for processing. The Multimodal Deep Semantic Transformer architecture consists of a backbone encoding layer, a gated cross-attention fusion layer, a modality attention change monitoring mechanism, and a structural consistency verification mechanism.

[0067] The backbone encoding layer uses a 60-layer Transformer structure to perform deep semantic modeling on the uniformly mapped multimodal feature sequence. To address the problem that traditional shallow models have insufficient context capture capabilities in long sequence processing, the enhanced self-attention mechanism is used to improve the perception of long-distance semantic dependencies in time series, achieve accurate extraction and dynamic understanding of global semantic features in the complex context of news short videos, and provide a more complete and coherent semantic basis for subsequent boundary judgment.

[0068] A gated cross-attention fusion layer is introduced in the backbone encoding process to dynamically adjust the information injection ratio of text, visual and audio modalities according to the context to achieve adaptive multimodal deep fusion.

[0069] The modal attention change monitoring mechanism generates perceptual signals reflecting the changes in modal dominance by real-time statistics of the attention distribution dynamics of each modality in cross-attention, which is used to trigger semantic mutation detection and boundary judgment.

[0070] By integrating a gated cross-attention fusion layer in the backbone encoding process, the ratio of information injection from different modalities can be dynamically adjusted based on contextual semantics. This solves the problem of over-reliance on noise modalities or insufficient information utilization under traditional fixed fusion strategies, significantly improving the adaptability of multimodal deep fusion and the accuracy of semantic modeling.

[0071] At the same time, by building a modal attention change monitoring mechanism, we track the dynamics of modal distribution in cross-attention in real time and generate perceptual signals of modal dominance changes. This replaces the traditional passive threshold detection method, achieving active triggering and context-aware optimization of the boundary determination process, effectively improving the responsiveness to complex semantic transitions and the robustness of boundary recognition during news stripping. To improve the structural rationality and semantic coherence of boundary determination, a structural consistency verification mechanism based on a graph neural network (GNN) is introduced in the candidate stripping boundary area.

[0072] The input multimodal feature sequence formula is expressed as:

[0073]

[0074] The backbone encoding layer consists of 60 layers of Transformer encoders with 32 self-attention heads per layer for deep semantic modeling. Each layer implements a standard self-attention mechanism to capture long-range dependencies within the time series and extract global semantic features. The formula is expressed as:

[0075]

[0076] in, Represents the overall multimodal fusion feature sequence; Indicates that at time step A single multimodal fusion feature vector at ; Indicates the time steps, Indicates the time step number; Represents the total number of time steps in the feature sequence. SelfAttention(·) represents the self-attention mechanism function, Q represents the query matrix, K represents the key matrix (Key), and V represents the value matrix (Value); represents the transpose of the key matrix K; softmax(·) represents the softmax normalization function; Indicates the dimension of the key vector.

[0077] In the backbone layer, a is set to 5; a gated cross-attention fusion layer is inserted every 5 layers of Transformer encoder, for a total of 12 layers, which are used to dynamically fuse text, visual and audio modality features.

[0078] Using text features as query vectors , visual and audio features as keys Sum , calculate the cross-modal attention output , the formula is:

[0079]

[0080] Introducing Trainable gating factor for time steps , to achieve dynamic fusion control:

[0081]

[0082] in, Indicates the The fused feature vector of time steps; Represents the dimension of the key vector

[0083] In each cross-attention layer, the attention distribution of different modalities is counted in real time, and a modal attention change monitoring mechanism is introduced to calculate the modal attention entropy. :

[0084]

[0085] in, Represents the time step When the mode (visual or audio) attention share. Indicates that at time step The modal attention entropy is used to quantify the distribution of attention between different modalities (visual, audio) in the cross-attention mechanism, reflecting the balance or bias of the current multimodal information fusion. The table continuously tracks the change in modal attention entropy when the following conditions are met:

[0086]

[0087] This indicates that in the current multimodal information fusion process, the dominance ratio of vision and audio has fluctuated significantly, which usually corresponds to the turning point of the news semantic structure or the switching of content focus.

[0088] Therefore, the system no longer performs a global traversal, but instead locates locally near the time step, starts the semantic trajectory offset analysis and structural consistency verification mechanism, and accurately determines whether there is a real semantic break in the area, avoiding invalid calculations and improving real-time and response efficiency.

[0089] in, Indicates that at the previous time step The modal attention entropy on ; The trigger threshold for changes in modal attention entropy is set based on experience. When the change exceeds this threshold, the system determines that the modal dominance has changed significantly and triggers the local boundary detection process. Indicates that at time step The mode at is concerned with the magnitude of entropy change.

[0090] Multimodal input After the 60-layer backbone Transformer encoding layer and the 12-layer gated cross attention fusion layer, the model is processed at each time step of the 60th layer. Output the final hidden state, recorded as:

[0091]

[0092] And the set of hidden states at all time steps is called a deep semantic representation sequence:

[0093]

[0094] in, Represents the time step The corresponding deep semantic feature vector. express dimensional real number space.

[0095] The deep semantic representation sequence is considered as the temporal trajectory of news content in the semantic space, which is used to reflect the dynamic process of semantic evolution over time. Based on this semantic trajectory, the directional offset of adjacent time steps is calculated to identify potential semantic mutation points. The specific formula is:

[0096]

[0097] when When , the current semantic flow direction changes significantly, it is determined to be a potential topic switching node, recorded as a candidate split boundary, and enters the structural consistency verification stage. Represents the time step The semantic trajectory direction offset, similarity and angle relationship between them. and Represents the Euclidean norm (L2 norm) of the vector. Represents the time step Deep semantic feature vector of Represents the time step The deep semantic feature vector of . represents the semantic trajectory offset threshold, which is set based on experience. .

[0098] The structural consistency verification mechanism includes building a local temporal semantic relationship graph for the time window of candidate stripping boundaries screened by semantic trajectory offset detection. .

[0099] The deep semantic representation vector within the window is selected as the node of the graph, and the formula is expressed as:

[0100]

[0101] in, represents the candidate split boundary center time step, Represents a set of points.

[0102] For each pair of adjacent nodes and , define the edge , and assign a weight to the edge. The formula is expressed as:

[0103]

[0104] in, Represents a set of edges. Representation node With node The edge weight between , the value range is between [-1, 1]; Represents the time step Deep semantic feature vector of Represents the time step The deep semantic feature vector of .

[0105] when When , it is marked as a strong connection, reflecting the high semantic correlation between nodes, and is given a higher weight in the GNN feature aggregation process to enhance the feature stability of the semantic continuous region. When , it is marked as a weak connection, reflecting the low semantic association between nodes. During the graph neural network feature aggregation process, the weight contribution of this edge is dynamically adjusted to reduce its influence on node feature updates and inhibit invalid information transfer between low-association nodes. The proportion of weak connections in the neighborhood of candidate split boundary nodes is counted, and the structural break confidence is calculated based on the node feature difference, improving the accuracy and robustness of boundary judgment.

[0106] Local temporal semantic relationship graph Input into the two-layer lightweight GNN model, perform weighted feature aggregation based on edge weights, and update the node feature representation. The formula is expressed as:

[0107]

[0108] in, Represents the normalized edge weight:

[0109]

[0110] in, is the weight matrix of the lth layer, represents the activation function; Representation node The initial node features. Indicates in In the layer GNN, nodes The updated feature vector of ; Indicates in In layer GNN, neighbor nodes The feature vector of is used as the input of feature aggregation. Representation node The set of neighbor nodes of Represents the GNN No. The trainable weight matrix of the layer is used to linearly transform the features of neighbor nodes and extract high-order semantic relationship features; Representation node Its neighboring nodes The normalized edge weights between The traversal variable representing the neighbor node index is used to normalize the sum operation of the denominator to ensure that the sum of the edge weights of all neighbor nodes is 1, forming a weighted average mechanism.

[0111] After the GNN feature update is completed, the structural consistency analysis of the candidate split boundary nodes is performed to calculate the feature difference between the candidate nodes and the adjacent nodes:

[0112]

[0113] Statistics Node The average edge weight of:

[0114]

[0115] The structural confidence score is obtained by synthesis:

[0116]

[0117] in, Indicates that at time step Confidence score of structural fracture calculated based on graph neural network (GNN). Represents the balance parameter of the structural confidence score, which is used to adjust the influence weight of feature difference and edge weight weakening on the overall score, satisfying . Represents candidate split boundary nodes The feature difference between its adjacent time step nodes, Representation node The feature vector after GNN update, Represents the Euclidean distance of vectors. Representation node The average edge weight of Representation node With neighboring nodes The edge weights between .

[0118] when When , it is determined that the candidate point has a significant semantic structure break, which is the initial segmentation boundary, and enters the final multi-signal fusion scoring link to ensure the comprehensiveness and rigor of the boundary judgment.

[0119] when When , it is determined that there is no significant semantic structure break at the candidate point.

[0120] Represents the structural fracture confidence threshold calculated based on the graph neural network (GNN), which is set according to requirements.

[0121] Integrate multi-dimensional judgment signals to perform boundary confidence scoring

[0122]

[0123] when , the time step is confirmed as a valid stripping boundary and is included in the initial stripping boundary list.

[0124] when , the time step is confirmed to be an invalid stripping boundary.

[0125] in, Indicates that at time step The comprehensive boundary confidence score at , with a value range of [0, 1]; Represents: Sigmoid function; Indicates that at time step The semantic trajectory offset at ; Indicates that at time step The modal focus at is the magnitude of entropy change; Indicates that at time step At , the structural fracture confidence score obtained based on the structural consistency verification of the graph neural network (GNN); Indicates that at time step The basic semantic difference at is defined as the average Euclidean distance between the deep semantic vectors of the current time step and its adjacent time steps, and the formula is expressed as: ; Indicates that the basic semantic difference is squared as a penalty term; Represents the boundary confidence threshold, which is determined by training set optimization and is usually set at interval.

[0126] All boundary points that meet the effective stripping boundary are set, and redundancy is removed. The non-maximum suppression strategy is adopted to eliminate the repeated boundaries caused by dense judgment, and finally a structured initial stripping boundary list is output:

[0127]

[0128] in, Represents the initial strip boundary list. Indicates the first time steps, Indicates the specific index position of the boundary point in the time series, , number all confirmed boundary points in sequence, Indicates the total number of initial stripping boundary points.

[0129] By building a unified multimodal semantic representation mechanism, the accuracy and robustness of news segment boundary recognition have been significantly improved. Unlike the existing technologies that mostly use shallow feature splicing or simple weighted fusion, this system introduces linear mapping of multimodal features, time position encoding enhancement, and deep semantic modeling mechanisms. Specifically, the system first maps the visual, audio, and text embedding features to a unified semantic space to eliminate the differences in the original feature dimensions and inconsistent semantic expressions between the modalities; then, combined with the time position encoding, the contextual relationship of the time dimension is explicitly embedded into the fusion representation to obtain a multimodal fusion input vector with sequence perception capabilities.

[0130] Furthermore, the fused vector is fed into a deep Transformer encoder structure, where a gated cross-attention fusion layer is periodically introduced. Trainable gating factors dynamically adjust the proportion of modal information injected under contextual awareness. Compared to traditional fusion strategies, this design boasts stronger modality adaptability and contextual semantic alignment, effectively suppressing interference signals from redundant modalities and enhancing the expressive weight of key modalities.

[0131] Furthermore, the computation of modal attention entropy and its change monitoring mechanism replaces traditional boundary detection methods that rely on fixed threshold strategies. This proactively triggers semantic mutation detection when modal dominance fluctuates significantly, significantly improving the stripping system's responsiveness to semantic turning points. Combined with a structural consistency verification mechanism, this step uses a graph neural network to conduct an in-depth analysis of the structural coherence of boundary candidate regions, ensuring that initial stripping boundaries are appropriately broken at semantically changing locations. This overcomes the underutilization of structural information by existing solutions.

[0132] Furthermore, by introducing a number of innovative mechanisms such as unified modeling of multimodal features, gated dynamic fusion and structural coherence verification, the boundary recognition method was systematically reconstructed from the feature layer, structure layer and trigger mechanism layer, which significantly improved the accuracy, robustness and generalization ability of the news segmentation task in complex semantic scenarios, and achieved obvious technological progress.

[0133] S3: For each segment corresponding to the initial segmentation boundary, perform multimodal semantic consistency verification, and combine the quality assessment results of each segment to optimize the initial segmentation boundary and output high-quality segments.

[0134] Based on the initial stripping boundary list , divide the news short video stream into several segments:

[0135]

[0136] For each fragment , re-extract normalized and time-aligned multimodal feature sequences The formula is:

[0137]

[0138] For fragments , construct cross-modal mapping and calculate cycle consistency loss, the formula is expressed as:

[0139]

[0140] in, represents the cross-modal mapping function from text to vision, Represents the inverse mapping from vision to text; represents the cross-modal mapping function from text to audio, Represents a cross-modal mapping function from audio to text. represents the L2 norm. represents the jth stripping fragment; Indicates the total number of strip segments. Representation fragment At time step The multimodal feature vector at ; Represents the time step index, ranging from , Represents fragments The start and end time steps. Representation fragment The cycle consistency loss value of Representation fragment The overall representation of the text modality feature; j represents the fragment index.

[0141] Dynamically adjust the semantic drift detection threshold based on the cycle consistency loss results :

[0142]

[0143] in, represents the basic drift threshold, is the control coefficient; Represents the exponential function.

[0144] Sequence representation based on deep semantics of fragments , calculate the semantic trajectory offset:

[0145]

[0146] in, Represents the time step The deep semantic feature vector at ; Representation fragment The maximum deviation of the semantic trajectory.

[0147] when When the fragment If there is a problem of insufficient semantic coherence, the segment will be marked as a semantic abnormal segment, and the boundary optimization mechanism will be triggered to recalculate the segmentation boundary of the segment to avoid semantic confusion affecting the content quality of the short video. No operation is required.

[0148] Pair of fragments Calculate the comprehensive quality score, the formula is expressed as:

[0149]

[0150] in, It represents the effective information density, which is calculated as the ratio of the effective information duration to the total duration of the segment, reflecting the information richness of the segment. The higher the density, the more refined the content and the less redundancy. It indicates the coordination of audio-visual synchronization. It calculates the semantic matching degree of audio and video in the clip at each time node and counts its fluctuation score. The higher the score, the more stable the audio-visual matching. It represents redundancy by detecting the similarity of deep semantic features of consecutive time periods in the segment, counting the proportion of highly repeated segments, and measuring the degree of information redundancy within the segment. is the weight coefficient. Representation fragment The overall quality rating of .

[0151] when When , it is marked as a low-quality fragment. , it is marked as a high-quality segment. Indicates the judgment threshold of the fragment quality score, which can be set according to requirements.

[0152] Integrate all high-quality clips and output a high-quality clip collection.

[0153] in, Indicates the optimized confirmed High-quality clips; Indicates the number of optimized fragments.

[0154] Compared with existing technologies, a multimodal semantic consistency optimization method that integrates cycle consistency verification and semantic drift control mechanism is proposed to address the problems of insufficient semantic coherence of fragments, poor modal coordination, and inability to dynamically optimize fragment quality in the process of news segmentation. By constructing a cross-modal bidirectional mapping with text modality as the core, the cycle consistency loss of fragments is calculated in real time, and quantitative judgment of multimodal semantic consistency is achieved, breaking through the technical limitations of traditional fragment evaluation that only relies on static rules or single modality similarity. At the same time, the semantic trajectory offset detection threshold is dynamically adjusted based on consistency loss, which improves the intelligent perception of semantic mutations and coherence anomalies within the fragment, avoiding misjudgment or missed detection problems caused by fixed thresholds.

[0155] Based on the initial stripping boundaries, multimodal semantic consistency verification is performed on each segmented segment, and combined with comprehensive quality assessment, the accuracy and expression quality of segmentation are effectively improved. First, by constructing a bidirectional cross-modal mapping mechanism with text modality as the core, a semantic mapping relationship is established between text-vision and text-audio, and bidirectional cycle consistency verification is performed separately to quantify the semantic reconstruction errors in the two directions, and accordingly obtain the cycle consistency loss value. Compared with traditional methods that only consider one-way alignment errors, this mechanism can fully reflect the closed-loop consistency of semantic transfer between modalities, which helps to identify potential risks of semantic disconnection.

[0156] After obtaining the loss results, the system dynamically adjusts the threshold used for semantic trajectory drift detection, adapting it to the semantic stability of the current segment. Specifically, when cycle consistency loss is high, the threshold is automatically tightened to increase sensitivity to semantic mutations; when loss is low, the system relaxes the detection boundary to avoid misjudgments and enhance the robustness of the overall decision. This adaptive drift detection mechanism effectively overcomes the shortcomings of existing methods that rely on static thresholds and lack contextual awareness.

[0157] Furthermore, the system combines the deep semantic representation of the clips to analyze the evolution direction of the semantic vectors at each time step in the time series, constructing a semantic trajectory and calculating the degree of semantic drift based on this. If the degree of drift exceeds an adjusted drift threshold, the system automatically identifies the risk of internal semantic discontinuity in the clip and repositions the segmentation boundaries to ensure semantic consistency and integrity within the clip. This mechanism overcomes the technical shortcomings of traditional methods for determining semantic coherence, which rely on static structures and are prone to misclassification or omission. Each clip undergoes a comprehensive quality assessment, with a weighted score based on three dimensions: effective information density, audiovisual coordination, and information redundancy. This scoring system not only comprehensively covers key metrics for short video clips in terms of information carrying capacity, modality matching, and expressive compactness, but also supports customized strategies for different usage scenarios through an adjustable weighting mechanism. High-quality clips are retained for output, while low-quality clips undergo boundary optimization, effectively improving the structural rationality and communication value of the final generated clips. The overall process embodies the deep integration of multimodal closed-loop verification, adaptive threshold adjustment, and semantic structure reconstruction, significantly outperforming traditional boundary judgment methods based solely on sudden changes in modal features.

[0158] Example 2 is an embodiment of the present invention, which provides a news strip splitting control system based on multimodality, including: a data module, which receives a news short video stream and splits the news short video stream into multimodal data.

[0159] The initial segmentation boundary module generates corresponding semantic representations based on multimodal data, fuses the features of each modality to form a unified semantic representation, and generates an initial split boundary list.

[0160] The output module performs multimodal semantic consistency verification on each segment corresponding to the initial segmentation boundary, and optimizes the initial segmentation boundary based on the quality assessment results of each segment to output high-quality segments.

[0161] Example 3, an embodiment of the present invention, is different from the previous two embodiments in that:

[0162] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0163] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0164] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in another suitable manner as necessary, and then storing it in a computer memory.

[0165] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0166] Example 4 is an embodiment of the present invention, which provides a news stripping control method and system based on multimodality. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0167] Ten long news videos published by a news media outlet in 2024 were selected as the test set. The videos had an average length of 300 seconds and covered a variety of genres, including politics, finance, and sports. The videos were complex in content and featured frequent modal interactions, making them suitable for testing segmentation accuracy and quality. The experimental environment was configured with an Intel Xeon processor, 32GB of memory, and an NVIDIA RTX 3080 graphics card. The system deployed the segmentation control algorithm described in this paper.

[0168] First, we set a sliding window length of 10 seconds and a step size of 5 seconds to receive a short news video stream and split it into three modal data types: visual, audio, and text. For the visual modality, we extracted and normalized video frames at 1fps. For the audio modality, we set the sampling rate to 16kHz and converted it into a mel-spectrogram. For videos without subtitles, we used an ASR model to generate text transcriptions. This was then combined with a word segmentation model to generate text embedding features, thereby constructing a standardized multimodal feature sequence.

[0169] The multimodal feature sequence is then fed into a deep semantic transformer. Leveraging a 60-layer Transformer and a gated cross-attention fusion layer, deep semantic modeling and dynamic boundary determination are achieved. Modal attention entropy monitoring and structural consistency verification are used to output an initial list of strip boundaries. For each segment, bidirectional cycle consistency verification and dynamic semantic drift control are performed. Combining effective information density, audiovisual synchronization coordination, and redundancy, segment quality assessment and boundary optimization are performed to ultimately output high-quality news strip segments.

[0170] In the experiment of decomposing 10 news videos, the proposed method generated 82 decomposition segments. The core indicators are as follows:

[0171] Average cycle consistency loss: 0.12.

[0172] Average semantic trajectory deviation: 0.35.

[0173] Average effective information density: 0.78.

[0174] Average audiovisual synchronization coordination score: 0.85.

[0175] Average redundancy: 0.08.

[0176] The proportion of high-quality clips: 93.90% (77 / 82).

[0177] Compared with the traditional rule-based segmentation method (high-quality segments account for about 76.50%), the present invention performs superiorly in segment quality, semantic coherence, and audio-visual coordination.

[0178] Experimental results demonstrate that this method demonstrates significant advantages in the task of decomposing complex news videos. First, the average effective information density reaches 0.78, an increase of approximately 15% compared to traditional methods, indicating that the output segments are more refined and invalid information is significantly reduced. The audiovisual synchronization and coordination score remains at 0.85, demonstrating that this method, through deep multimodal fusion and dynamic control mechanisms, can effectively ensure the semantic synchronization between commentary and images, avoiding the common audio-visual disconnection problem in existing technologies.

[0179] Furthermore, the redundancy is controlled at 0.08, significantly lower than the 0.20 of traditional methods, demonstrating that this method can accurately identify and remove repetitive information from segments, improving content compactness. Through a dynamic semantic drift control mechanism guided by cycle consistency, potential semantic coherence issues are accurately identified, preventing the output of semantically disconnected segments and ensuring the overall semantic integrity and structural rationality of the segmentation results.

[0180] In summary, the present invention realizes intelligent control of the news stripping process through an innovative multimodal deep semantic transformer architecture, a dynamic boundary judgment mechanism, and a segment quality optimization strategy, significantly improving the accuracy, real-time performance, and output segment quality of the stripping. It overcomes the shortcomings of the existing technology in terms of fixed rules, single modal processing, and lack of dynamic optimization capabilities. It has outstanding creativity and novelty, is suitable for various news short video automatic generation scenarios, and has good application prospects and promotion value.

[0181] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A news stripping control method based on multimodality is characterized by: including: Receiving a news short video stream and splitting the news short video stream into multimodal data; Generating corresponding semantic representations based on the multimodal data, fusing the features of each modality to form a unified semantic representation, and generating an initial strip boundary list; For each segment corresponding to the initial segmentation boundary, performing multimodal semantic consistency verification, and combining the quality evaluation results of each segment to optimize the initial segmentation boundary and output high-quality segments; Inputting the unified semantic representation into a multimodal deep semantic transformer for deep semantic modeling to identify semantic changes; based on the identified semantic change boundary candidate time steps, generating an initial strip boundary list; The multimodal deep semantic transformer, by introducing a trainable gating factor, adjusts the fusion ratio of text embedding features, visual features, and audio features according to the context semantics to achieve adaptive deep fusion of multimodal features; through a modality attention change monitoring mechanism, embedding modality attention entropy in the cross-attention process to real-time perceive the change of modality dominance and dynamically trigger semantic mutation detection to optimize the determination of strip boundaries; according to a preset structural consistency verification mechanism, constructing a local temporal semantic relationship graph based on the boundary candidate time steps, performing graph structure modeling in combination with edge weights, and verifying the semantic coherence and structural rationality of the boundary candidate time steps through a graph neural network.

2. The multimodal news stripping control method according to claim 1, wherein: The receiving the news short video stream includes setting a sliding window length of H seconds and a sliding step of K seconds, where 0 < K < H; and intercepting a news short video stream segment of H seconds every K seconds for preprocessing; where H is the sliding window length representing the duration of each intercepted segment; K is the sliding step representing the time interval between two adjacent interceptions; The multimodal data includes visual modality, audio modality, and text modality data; Extract video frames from the pre-processed news short video stream according to a preset frame rate, normalize the video frames, and generate visual features ; Extract audio modal data from the preprocessed news short video stream at a fixed sampling frequency, convert it into a Mel-spectrogram, and generate audio features , and when the news short video stream has no subtitles, calling the automatic speech recognition model to generate text transcription results; Based on the text transcription result, using a word segmentation model to generate text embedding features T; At each time step In the combination, the corresponding multimodal feature vector is generated ;in, Represents the time step The visual characteristics of Represents the time step The audio characteristics of Represents the time step Text embedding features; preprocess the multimodal feature vectors of all time steps and construct a multimodal feature sequence ; Indicates in The multimodal feature vector corresponding to each time step; Indicates the time steps, Indicates the time step number.

3. The multimodal news stripping control method according to claim 2, wherein: The unified semantic representation includes receiving the multimodal feature sequence, including visual features, audio features, and text embedding features; Respectively using a linear mapping function to transform the visual features, audio features, and text embedding features into a semantic space of the same dimension, and adding time position encoding to each modality feature vector; adding the linearly mapped visual features, audio features, and text embedding features and the corresponding time position encoding to obtain the multimodal fusion input vector at each time step.

4. The multimodal news stripping control method according to claim 3, wherein: The architecture of the multimodal deep semantic transformer includes a backbone encoding layer, a gating cross-attention fusion layer, a modality attention change monitoring mechanism, and a structural consistency verification mechanism; The backbone encoding layer is composed of a stack of Transformer encoders with a preset number of layers, capturing long-range dependencies in the time series based on the self-attention mechanism and extracting global semantic features; Each a-layer Transformer encoder is inserted into a gated cross attention fusion layer, with text modality features as query vectors, visual modality and audio modality features as keys and values, to perform cross-modal attention calculations, and based on the trainable gating factor Dynamically adjust the information injection ratio of each modal feature; Wherein, a represents the preset number of interval layers; Indicates the A trainable gating factor for each time step.

5. The multimodal news stripping control method according to claim 4, characterized in that: The modal attention change monitoring mechanism is embedded in the gated cross attention fusion layer, which calculates the attention distribution of text, visual and audio modal features in each time step, and calculates the modal attention entropy based on the attention distribution , and in consecutive time steps, when the modal attention entropy change amplitude is detected Exceeding the preset threshold When , local semantic trajectory offset detection is performed; The structural consistency verification mechanism includes constructing a local temporal semantic relationship graph based on the deep semantic representation sequence after the local semantic trajectory offset detection. , combining edge weights to model graph structure, and performing weighted feature aggregation and structural analysis through a two-layer lightweight graph neural network to calculate the structural fracture confidence of the candidate stripping boundary , verify the semantic coherence and structural rationality of the boundary area; among them, represents a set of points, represents the edge set, Indicates the The confidence level of structural fracture at each time step.

6. The multimodal news stripping control method according to claim 5, characterized in that: Performing the multimodal semantic consistency includes, for each segment corresponding to the initial segmentation boundary, constructing a cross-modal mapping relationship with the text modality as the core; respectively mapping the text features to the visual modality space and the audio modality space through a cross-modal encoder, and then mapping back to the text space from their respective modality spaces, and using the difference between the original text features and the features after the reverse mapping as the cyclic consistency loss value; Adjusting the semantic topic drift detection threshold according to the cycle consistency loss value , the formula is: ; in, represents the basic drift threshold, is the control coefficient; represents the exponential function, Representation fragment The cycle consistency loss value of ; Indicates the jth strip segment; j represents the segment index, Representation fragment The semantic topic drift detection threshold; Based on the deep semantic representation sequence of the fragment, the cosine similarity between consecutive time points is calculated. When the cosine similarity drops below the set threshold, the fragment semantic coherence is determined to be insufficient, and the current strip boundary is redefined.

7. A multimodal news stripping control system using the method according to any one of claims 1 to 6, characterized in that: A data module receives a news short video stream and splits the news short video stream into multimodal data; The initial segmentation boundary module generates corresponding semantic representations based on multimodal data, fuses the features of each modality to form a unified semantic representation, and generates an initial split boundary list; The output module performs multimodal semantic consistency verification on each segment corresponding to the initial segmentation boundary, and optimizes the initial segmentation boundary based on the quality assessment results of each segment to output high-quality segments.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multimodal news stripping control method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal news stripping control method described in any one of claims 1 to 6 are implemented.

Images