A video content structured disassembly and analysis method and system based on time series segmentation

Abstract

The present invention relates to the field of video recognition technology, specifically a method and system for video content structured decomposition and analysis based on time segmentation. First, the video and metadata are acquired, and a first artificial intelligence model integrating a spatiotemporal attention mechanism and a sliding window mechanism is used to perform time segmentation to generate time segments. The video is cut into several video segments using the FFmpeg tool, and a second artificial intelligence model integrating multimodal fusion and timestamp embedding is used to extract text descriptions of each segment to achieve semantic alignment of actions and texts. At the same time, FFmpeg is used to extract key frames to generate picture groups, and a third artificial intelligence model combining target detection and cross-frame consistency constraints performs semantic analysis to generate action labels. Finally, the time segments, key frames, text descriptions, and action labels are integrated to generate a complete structured output. The present invention can improve the recognition accuracy of video content.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a method and system for structured decomposition and analysis of video content based on time series segmentation. Background Art

[0002] With the explosive growth of video data, the structural decomposition of video content has become a core technical requirement in areas such as video understanding, intelligent monitoring, and media recommendation. Current technologies primarily focus on temporal action segmentation and multimodal feature fusion, aiming to semantically segment long videos into independent segments and extract structured information to support subsequent applications.

[0003] Existing methods still have the following core problems:

[0004] 1) Insufficient adaptability to dynamic scenes

[0005] Time scale inconsistency: A single model struggles to balance local details (small time scale) with global semantics (large time scale). For example, the action of opening a door requires combining the key rotation and door state changes from the previous and next frames. Existing TCNs or Transformers are prone to misjudgment due to the dominance of local features.

[0006] Sensitive to background interference: In scenes where the background action accounts for a large proportion, existing segmentation algorithms lack semantic constraints and are prone to mis-segmenting continuous background into multiple action segments.

[0007] 2) Roughness of multimodal fusion

[0008] Disconnection between text and visual features: Annotated text and video temporal features are only associated through shallow embedding, without fine-grained alignment, resulting in limited semantic guidance effect.

[0009] Insufficient utilization of manifold structure: Existing methods ignore the distribution characteristics of video data in low-dimensional manifold space and fail to jointly model temporal continuity and local manifold structure, which affects the accuracy of segmentation boundaries.

[0010] 3) Computational efficiency and generalization bottlenecks

[0011] High model complexity: Multi-stage TCN or Transformer requires a large number of parameters, resulting in poor real-time performance and difficulty in deployment on edge devices.

[0012] Data dependence: Supervised learning requires a large amount of labeled data, while unsupervised methods are sensitive to the quality of initial features and have limited generalization capabilities.

[0013] 4) Over-segmentation problem has not been solved

[0014] Large intra-action feature differences: The visual features of the same action at different execution stages vary significantly. Existing models are prone to redundant segmentation due to the lack of intra-segment consistency constraints.

[0015] Boundary fuzziness: Motion blur or lighting changes in first-person videos make it difficult to accurately detect action boundaries, and traditional thresholding methods are prone to missed detections or false detections.

[0016] To this end, a video content structured decomposition and analysis method and system based on time series segmentation is proposed. Summary of the Invention

[0017] The purpose of the present invention is to provide a method and system for video content structured decomposition and analysis based on time segmentation. First, the video and metadata are acquired, and a first artificial intelligence model integrating a spatiotemporal attention mechanism and a sliding window mechanism is used to perform time segmentation to generate time segments. The video is cut into several video segments using the FFmpeg tool, and a second artificial intelligence model integrating multimodal fusion and timestamp embedding is used to extract text descriptions of each segment to achieve semantic alignment of actions and text. At the same time, FFmpeg is used to extract key frames to generate image groups, and a third artificial intelligence model combining target detection and cross-frame consistency constraints performs semantic analysis to generate action labels. Finally, the time segments, key frames, text descriptions, and action labels are integrated to generate a complete structured output. The present invention can improve the recognition accuracy of video content, thereby ensuring the accuracy of content structured decomposition and analysis.

[0018] To achieve the above object, the present invention provides the following technical solutions:

[0019] A video content structured decomposition and analysis method based on time series segmentation, comprising:

[0020] Get videos uploaded by users and extract video metadata;

[0021] Performing temporal segmentation on the video using a first artificial intelligence model to generate a plurality of time segments, wherein the first artificial intelligence model utilizes a spatiotemporal attention mechanism and a sliding window mechanism in combination with the metadata to optimize action boundary recognition in the video content;

[0022] Using FFmpeg tool to split the video according to the time segment to generate a plurality of video segments;

[0023] The plurality of video clips are processed by: extracting text descriptions of the video clips using a second artificial intelligence model, wherein the second artificial intelligence model uses multimodal fusion and timestamp embedding for dynamic semantic alignment of actions and text; extracting key frames of the video clips using an FFmpeg tool to generate image groups, and performing semantic analysis on the image groups using a third artificial intelligence model to generate structured action labels, wherein the third artificial intelligence model combines object detection and cross-frame consistency constraints to identify objects in the video clips and their action relationships;

[0024] Integrate time segments, keyframes, text descriptions, and action labels to generate structured output.

[0025] Furthermore, the metadata of the video includes: resolution, frame rate and duration.

[0026] Furthermore, the process of performing time segmentation on the video using the first artificial intelligence model to generate multiple time segments includes:

[0027] resizing the video frame input based on the resolution in the metadata;

[0028] The spatiotemporal attention mechanism is used to capture short-term action details and long-term semantic dependencies of the video; wherein the short-term action details are realized by extracting local spatiotemporal features, and the long-term semantic dependencies are realized by modeling global spatiotemporal features;

[0029] By using the sliding window mechanism, the window step size is dynamically adjusted in combination with the frame rate in the metadata to optimize the initial demarcation of the action boundary;

[0030] The timestamps of action boundaries are predicted through the object detection regression module, and self-supervised learning is combined to generate time segments.

[0031] Furthermore, the process of extracting a text description of the video clip by the second artificial intelligence model includes:

[0032] Extracting visual features of the video clip based on the resolution in the metadata, and extracting audio features based on the audio stream;

[0033] Mapping action time points into action time vector representations through timestamp embedding;

[0034] Integrating the visual features, the audio features, and the action time vector representation through a multimodal fusion technique, and using a contrastive learning method to separate background interference;

[0035] A preliminary text description is generated based on the fused features, and the text fluency and semantic accuracy are optimized through the language model.

[0036] Furthermore, the process of using the FFmpeg tool to extract the key frames of the video clip to generate a picture group includes:

[0037] Setting FFmpeg extraction parameters based on the resolution and frame rate in the metadata, adjusting the keyframe output size and extraction frequency;

[0038] Extracting key frames of the video clip using a scene change detection algorithm, and selecting frames with significant semantic changes;

[0039] Using an artificial intelligence model to verify the semantic importance and image quality of the keyframes, retaining keyframes related to the action;

[0040] The extracted key frames are organized into image groups in chronological order and grouped by action stages using a semantic clustering algorithm;

[0041] The picture group is compressed based on the resolution in the metadata to generate an optimized picture group.

[0042] Furthermore, the process of performing semantic analysis on the image group by the third artificial intelligence model to generate structured action labels includes:

[0043] Adjusting the input size of the picture group based on the resolution in the metadata to optimize the computational efficiency of semantic analysis;

[0044] identifying objects and their spatial relationships in each frame of the image group using object detection technology, and extracting action-related visual features;

[0045] The conditional random field model is used to impose cross-frame consistency constraints and ensure the semantic consistency of action labels based on time series.

[0046] Based on the identified objects, spatial relationships and cross-frame consistency constraints, a structured action label is generated, which contains object, action and timestamp information.

[0047] A video content structured disassembly and analysis system based on time series segmentation, comprising:

[0048] Video data acquisition unit, used to obtain videos uploaded by users and extract video metadata;

[0049] a time segmentation unit, configured to perform time segmentation on the video using a first artificial intelligence model to generate a plurality of time segments, wherein the first artificial intelligence model utilizes a spatiotemporal attention mechanism and a sliding window mechanism in combination with the metadata to optimize action boundary recognition in the video content;

[0050] A video segmentation unit, configured to segment the video according to the time segments using an FFmpeg tool to generate a plurality of video segments;

[0051] a semantic analysis and structured processing unit, configured to process the video clips, the processing comprising: extracting text descriptions of the video clips using a second artificial intelligence model, wherein the second artificial intelligence model employs multimodal fusion and timestamp embedding for dynamic semantic alignment of actions and text; extracting key frames of the video clips using an FFmpeg tool to generate image groups, and performing semantic analysis on the image groups using a third artificial intelligence model to generate structured action labels, wherein the third artificial intelligence model combines target detection and cross-frame consistency constraints to identify objects in the video clips and their action relationships;

[0052] The integration and output unit is used to integrate time segments, key frames, text descriptions and action labels to generate structured outputs.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention proposes a temporal segmentation method for accurately segmenting videos. This method first utilizes a spatiotemporal attention mechanism that combines local and global attention to capture local action details and long-term semantic dependencies, and uses an adaptive sliding window to optimize action boundary division, thereby improving the accuracy of action boundary recognition in complex scenes. The timestamps of action boundaries are predicted through a target detection regression module, and time segments are generated in combination with self-supervised learning, thereby improving the accuracy of temporal segmentation.

[0055] 2. The present invention proposes a method for extracting video text descriptions. This method generates text descriptions by integrating visual features, audio features, and action time sequences in video clips. The action time sequence ensures that the text is aligned with the action time points, thereby improving the timing accuracy of the text description. Contrastive learning is used to separate interference, further improving the accuracy of semantic extraction, thereby improving the reliability and accuracy of the structured decomposition and analysis of video content.

[0056] 3. The present invention proposes a method for generating structured action labels. This method uses target detection to identify objects and their spatial relationships, extracts action-related visual features, and ensures the semantic accuracy of the labels. It then uses conditional random fields to model action dependencies based on time series to solve the problem of inconsistent actions across frames. This can enhance the quality of structured labels, thereby improving the reliability and accuracy of structured decomposition and analysis of video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of a flow chart of a method for structurally decomposing and analyzing video content based on time series segmentation according to the present invention;

[0058] Figure 2 A schematic diagram of a process for extracting a text description of a video clip using a second artificial intelligence model according to the present invention;

[0059] Figure 3 The diagram is a structural diagram of a video content structured disassembly and analysis system based on time series segmentation according to the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] See also Figures 1 to 3 The present invention provides a method and system for video content structured decomposition and analysis based on time series segmentation. The technical solution is as follows:

[0062] Example 1:

[0063] In order to improve the efficiency and accuracy of video content structured decomposition, a company used a video content structured decomposition and analysis method based on time series segmentation proposed in this invention. The process of this method is shown as follows: Figure 1 Shown, including:

[0064] Get videos uploaded by users and extract video metadata;

[0065] Furthermore, the metadata of the video includes: resolution, frame rate, and duration;

[0066] Furthermore, the FFmpeg command is executed on the uploaded video to output video metadata in JSON format; by parsing the JSON, information such as resolution, frame rate, and duration is obtained to meet subsequent processing requirements.

[0067] The metadata information of three different videos is given below, as shown in Table 1.

[0068] Table 1. Metadata information

[0069]

[0070] By using the video metadata, a standardized basis and constraints are provided for subsequent video content processing, thereby ensuring the feasibility of temporal segmentation and semantic analysis, and further improving the reliability and accuracy of the structured decomposition and analysis of video content.

[0071] The video is temporally segmented using a first artificial intelligence model to generate multiple time segments, wherein the first artificial intelligence model uses a spatiotemporal attention mechanism and a sliding window mechanism in combination with metadata to optimize the recognition of action boundaries in the video content;

[0072] Furthermore, the process of performing time-series segmentation on the video using the first artificial intelligence model to generate multiple time segments includes:

[0073] Resize video frame input based on resolution in metadata;

[0074] The short-term action details and long-term semantic dependencies of the video are captured through the spatiotemporal attention mechanism. The short-term action details are realized by extracting local spatiotemporal features, and the long-term semantic dependencies are realized by modeling global spatiotemporal features.

[0075] Through the sliding window mechanism, the window step size is dynamically adjusted in combination with the frame rate in the metadata to optimize the initial division of the action boundary;

[0076] The target detection regression module predicts the timestamps of action boundaries and combines it with self-supervised learning to generate time segments;

[0077] Furthermore, the resolution in the metadata is preprocessed using the FFmpeg tool, and the input size of the video frame is adjusted to reduce the model calculation time;

[0078] Furthermore, the pre-trained TimeSformer encoder is used in combination with the spatiotemporal attention mechanism to extract local and global spatiotemporal features in video frames;

[0079] Furthermore, the TimeSformer encoder is trained using MoCo v3, an algorithm for self-supervised learning. MoCo v3 trains the encoder via contrastive learning, which maximizes the similarity between positive samples (query and key features of the same action) while minimizing the similarity between negative samples (query and key features of background frames).

[0080] Furthermore, the TimeSformer encoder extracts local and global spatiotemporal features as follows: using 15 frames as a local window and dividing the input into 8×8 spatiotemporal patches, local attention is used to extract local spatiotemporal features of 15 frames; global attention is used to process the entire video frame, integrating the spatiotemporal dependencies between frames, and outputting global spatiotemporal features. The calculation formulas for local attention and global attention are the same, the difference lies in the input objects processed and the sequence size of the output features.

[0081] Furthermore, in order to enable the subsequent object detection and regression module to more accurately locate the action boundary, a sliding window mechanism is used to dynamically adjust the window step size in combination with the frame rate in the metadata, thereby optimizing the initial demarcation of the action boundary. The sliding window size is set to half a full frame, and the dynamic adjustment of the time step can be represented by taking the minimum value of the preset time step and the adjusted time step. The preset time step is set to 0.5 seconds, and the adjusted time step is the ratio of the frame rate coefficient to the frame rate. The frame rate coefficient is set to 15, and the frame step size is obtained by multiplying the time step by the frame rate for window movement.

[0082] Furthermore, the features of each window extracted by TimeSformer are input into the DETR model for prediction. The non-maximum suppression algorithm is then used to calculate the IoU value of the previous and next windows to determine the similarity of the boundary prediction timestamps. For example, boundaries with an IoU greater than 0.9 are merged, and the timestamp with the highest confidence is taken as the merged timestamp. The DETR model uses L1+cross entropy as the loss function and is trained on boundary annotation data to optimize timestamp regression.

[0083] Furthermore, in order to enhance the semantic relevance across windows, a graph neural network (GNN) is used to model the semantic relationship between windows. The TimeSformer features of each window after average pooling are used as graph nodes, and the temporal distance between windows is used as edges. GNN is used to aggregate the features of each window with the features of neighboring windows to generate enhanced features. The window features originally extracted by TimeSformer and the enhanced features are channel-joined and sent together to the DETR model for prediction.

[0084] By utilizing a spatiotemporal attention mechanism that combines local and global attention, we can capture local action details and long-term semantic dependencies, and use an adaptive sliding window to optimize action boundary division, which can improve the accuracy of action boundary recognition in complex scenes; the timestamps of action boundaries are predicted through the target detection regression module, and time segments are generated in combination with self-supervised learning, which can improve the accuracy of time series segmentation.

[0085] Use FFmpeg to split the video into time segments and generate several video segments;

[0086] Several video clips were processed by: extracting text descriptions of the video clips using a second AI model, which employed multimodal fusion and timestamp embedding for dynamic semantic alignment of actions and text; extracting key frames from the video clips using the FFmpeg tool to generate image groups, and performing semantic analysis on the image groups using a third AI model to generate structured action labels. The third AI model combined object detection and cross-frame consistency constraints to identify objects in the video clips and their action relationships;

[0087] Furthermore, the process of extracting the text description of the video clip through the second artificial intelligence model is as follows: Figure 2 Shown, including:

[0088] Extract visual features of video clips based on the resolution in metadata, and extract audio features based on the audio stream;

[0089] Mapping action time points into action time vector representations through timestamp embedding;

[0090] By integrating visual features, audio features and action time vector representation through multimodal fusion technology, the contrastive learning method is used to separate background interference;

[0091] Generate preliminary text description based on the fused features, and optimize text fluency and semantic accuracy through language model;

[0092] Furthermore, visual and audio features are extracted by inputting the video clip and the corresponding audio stream into the CLIP-ViT model and the Wav2Vec 2.0 model, respectively. The timestamps predicted by the object detection and regression module are input into a linear layer for processing and mapped into a time vector representation of the action. The audio stream is extracted using the FFmpeg tool. Both the CLIP-ViT model and the Wav2Vec 2.0 model are pre-trained models, using cross entropy as the loss function, a learning rate of 1e-4, and a batch size of 32.

[0093] Furthermore, a four-layer Transformer model structure is used to perform multimodal weighted fusion of visual features, audio features, and action time vector representations. The weights of each feature are learned through training. During the model training process, a contrastive learning method (SimCLR) is introduced. That is, by using the SimCLR loss function, action features are compared with background features to enhance action semantics and separate background interference.

[0094] Furthermore, to address the problem that video clips in real-world scenarios may lack certain modal data (such as silent videos or low-quality videos), an adaptive modality missing processing mechanism is introduced before multimodal weighted fusion. By adding a modality completion module (such as feature reconstruction based on generative adversarial networks) to the multimodal fusion stage, approximate features are generated when audio or visual information is missing;

[0095] Furthermore, the fused features were input into the pre-trained LLaMA 3.1 model to generate a preliminary text description. Then, the pre-trained LLaMA 3.1 model was used as a text optimization model, and the preliminary text description was used as the input of the optimization model to perform text modification and polishing. The text quality was evaluated using indicators such as BLEU, METEOR, and Grammatical Error Detection. The LLaMA 3.1 model was trained using a combined loss function of cross entropy and SimCLR, with the learning rate and batch size set to 5e-5 and 16, respectively.

[0096] By integrating the visual features, audio features and action time series in the video clips, text descriptions are generated. The action time series ensures that the text is aligned with the action time points, improving the timing accuracy of the text description. Contrastive learning is used to separate interference and further improve the accuracy of semantic extraction, thereby improving the reliability and accuracy of the structured decomposition and analysis of video content.

[0097] Furthermore, the process of using the FFmpeg tool to extract key frames of a video clip to generate a picture group includes:

[0098] Set FFmpeg extraction parameters based on the resolution and frame rate in the metadata, adjusting the keyframe output size and extraction frequency;

[0099] Use scene change detection algorithms to extract key frames from video clips and select frames with significant semantic changes;

[0100] Use artificial intelligence models to verify the semantic importance and image quality of keyframes and retain keyframes related to actions;

[0101] The extracted key frames are organized into image groups in chronological order and grouped by action stages using a semantic clustering algorithm;

[0102] Compress the image group based on the resolution in the metadata to generate an optimized image group;

[0103] Furthermore, the scene change detection algorithm uses a method based on pixel differences, namely the frame difference method. This method is used to perform inter-frame detection on the time-series frames extracted by FFmpeg. If the pixel difference exceeds a preset threshold (0.35), the frame is selected as the key frame.

[0104] Furthermore, the previously pre-trained CLIP-ViT was used to extract visual features from the keyframes, and then the action labels associated with the video were encoded as text features. Next, the cosine similarity score between the visual features and the text features of each keyframe was calculated. A higher score indicates action-relatedness, while a lower score indicates background-relatedness. The action labels used for verification were obtained through manual annotation rather than model generation. The semantically verified keyframes were input into the pre-trained EfficientNet, and the clarity probability was output. If the clarity probability is greater than 0.5, it indicates clarity, otherwise it is unclear. The learning rate used in the EfficientNet model training process is 1e-3, the number of training rounds is 100 epochs, and the cross entropy is used as the loss function.

[0105] Furthermore, the DINOv2 visual encoder is used to extract the visual features of the image group, and then the DBSCAN clustering algorithm is used to classify the visual features to obtain the grouping results; The value is set to 0.5;

[0106] By using scene change detection algorithms and intelligent models, the semantics and image quality of key frames extracted by FFmpeg can be detected, thereby improving the accuracy of subsequent action label generation and further improving the reliability and accuracy of video content structured decomposition and analysis.

[0107] Furthermore, the process of performing semantic analysis on the image group by the third artificial intelligence model and generating structured action labels includes:

[0108] Adjust the input size of the image group based on the resolution in the metadata to optimize the computational efficiency of semantic analysis;

[0109] Use object detection technology to identify objects and their spatial relationships in each frame of the image group and extract action-related visual features;

[0110] The conditional random field model is used to impose cross-frame consistency constraints and ensure the semantic consistency of action labels based on time series.

[0111] Generate structured action labels based on the identified objects, spatial relationships, and cross-frame consistency constraints. The structured action labels contain object, action, and timestamp information.

[0112] Furthermore, the third AI model is the DETR-CRF model, where the DETR model is used for target detection and the Conditional Random Field (CRF) model is used to impose time series constraints;

[0113] Furthermore, after adjusting the input size of the image group, pixel value normalization is performed to improve the model operation efficiency;

[0114] Furthermore, the DETR model uses the ResNet-50 backbone network to extract features from the input image group and processes the extracted features using 1×1 convolution operations to meet the input requirements of the Transformer encoder; the Transformer encoder uses self-attention to process features to obtain encoded features; the Transformer decoder uses a query vector (a hyperparameter set by DETR by default) to interact with the encoded features and uses a feedforward network to update the query vector; finally, the prediction head is used to predict the bounding box and category of the query vector; the relative position between the predicted bounding boxes is calculated to represent the spatial relationship; for example, the distance and size ratio between the "hand" box (x1, y1, w1, h1) and the "door handle" box (x2, y2, w2, h2) is used to represent the "contact" relationship;

[0115] Furthermore, the RoI-Align feature extraction technique is applied to each predicted bounding box, and then the extracted features are processed using a global average pooling operation to obtain object features, i.e., action-related visual features;

[0116] Furthermore, a linear chain CRF model is used to model the probability distribution of action labels in frame sequences. The DETR output and the action label set are used as input. The label sequence is optimized using single-frame features (objects and spatial relationships output by DETR) and cross-frame transfer constraints. The Viterbi algorithm is used to solve the optimal action label sequence during the CRF inference process.

[0117] Furthermore, the optimization processes of single-frame features and cross-frame transfer constraints are implemented through single-frame potential functions and cross-frame potential functions, respectively. The single-frame potential function first concatenates the features extracted by DETR into a vector, which is then input into a neural network (two fully connected layers) to generate an initial score for each action label. The higher the score, the more consistent the label is with the content of the frame. The cross-frame potential function defines the transition probability of each pair of labels through a transfer matrix and introduces a time penalty factor to adjust the transition probability according to the inter-frame interval, which can ensure the coherence of the action labels in the time series.

[0118] Furthermore, in order to optimize the action label sequence, a post-processing module is introduced after generating structured action labels. The label sequence is semantically optimized through a pre-trained language model to generate more natural action descriptions. In addition, a rule-based label smoothing mechanism is applied to the semantically optimized label sequence to reduce noise in the sequence and improve the quality of the label sequence.

[0119] By using target detection to identify objects and their spatial relationships, action-related visual features are extracted to ensure the semantic accuracy of labels. Then, conditional random fields are used to model action dependencies based on time series to solve the problem of inconsistent actions across frames. This can enhance the quality of structured labels, thereby improving the reliability and accuracy of structured decomposition and analysis of video content.

[0120] Integrate time segments, keyframes, text descriptions, and action labels to generate structured output.

[0121] This embodiment proposes a method for structured decomposition and analysis of video content based on time segmentation; first, the video and metadata are obtained, and a first artificial intelligence model that integrates a spatiotemporal attention mechanism and a sliding window mechanism is used to perform time segmentation to generate time segments; the video is cut into several video segments using the FFmpeg tool, and a second artificial intelligence model that integrates multimodal fusion and timestamp embedding is used to extract the text description of each segment to achieve semantic alignment of action and text; at the same time, FFmpeg is used to extract key frames to generate picture groups, and a third artificial intelligence model that combines target detection and cross-frame consistency constraints performs semantic analysis to generate action labels; finally, the time segments, key frames, text descriptions and action labels are integrated to generate a complete structured output; the present invention can improve the recognition accuracy of video content, thereby ensuring the accuracy of structured decomposition and analysis of content.

[0122] Example 2:

[0123] The present invention also proposes a video content structured disassembly and analysis system based on time series segmentation, the structure of the system is as follows: Figure 3 As shown, the details are as follows:

[0124] Video data acquisition unit, used to obtain videos uploaded by users and extract video metadata;

[0125] a time segmentation unit, configured to perform time segmentation on the video using a first artificial intelligence model to generate a plurality of time segments, wherein the first artificial intelligence model utilizes a spatiotemporal attention mechanism and a sliding window mechanism in combination with metadata to optimize action boundary recognition in the video content;

[0126] The video segmentation unit is used to segment the video according to the time segment using the FFmpeg tool to generate several video segments;

[0127] A semantic analysis and structured processing unit is used to process a number of video clips. The processing process is as follows: extracting text descriptions of the video clips through a second artificial intelligence model, wherein the second artificial intelligence model uses multimodal fusion and timestamp embedding for dynamic semantic alignment of actions and text; using the FFmpeg tool to extract key frames of the video clips to generate image groups, and performing semantic analysis on the image groups through a third artificial intelligence model to generate structured action labels, wherein the third artificial intelligence model combines object detection and cross-frame consistency constraints to identify objects in the video clips and their action relationships;

[0128] The integration and output unit is used to integrate time segments, key frames, text descriptions and action labels to generate structured outputs.

[0129] Furthermore, the process of extracting a text description of the video clip by the second artificial intelligence model includes:

[0130] Extract visual features of video clips based on the resolution in metadata, and extract audio features based on the audio stream;

[0131] Mapping action time points into action time vector representations through timestamp embedding;

[0132] By integrating visual features, audio features and action time vector representation through multimodal fusion technology, the contrastive learning method is used to separate background interference;

[0133] Generate preliminary text description based on the fused features, and optimize text fluency and semantic accuracy through language model;

[0134] Furthermore, the process of performing semantic analysis on the image group by the third artificial intelligence model and generating structured action labels includes:

[0135] Adjust the input size of the image group based on the resolution in the metadata to optimize the computational efficiency of semantic analysis;

[0136] Use object detection technology to identify objects and their spatial relationships in each frame of the image group and extract action-related visual features;

[0137] The conditional random field model is used to impose cross-frame consistency constraints and ensure the semantic consistency of action labels based on time series.

[0138] Based on the identified objects, spatial relationships and cross-frame consistency constraints, structured action labels are generated. The structured action labels contain object, action and timestamp information.

[0139] The performance comparison results under different third-party artificial intelligence models are given below, as shown in Table 2; the indicators include action label accuracy (ALA), the F1 score used to represent the average of precision and recall, and the BLEU-4 score used to evaluate the similarity with the true label; the comparison models include the DETR-CRF model proposed in this paper, the DETR-Only model without the conditional random field model, and the traditional CNN-LSTM model.

[0140] Table 2. Performance comparison results

[0141]

[0142] It can be seen from Table 2 that the DETR-CRF model proposed in this paper can obtain more accurate action label sequences, thereby improving the reliability and accuracy of the structured decomposition analysis of video content.

[0143] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A video content structured disassembly and analysis method based on time series segmentation, characterized in that: include: Get videos uploaded by users and extract video metadata; The video is temporally segmented using a first artificial intelligence model to generate a plurality of time segments, wherein the first artificial intelligence model uses a spatiotemporal attention mechanism and a sliding window mechanism, and optimizes the recognition of action boundaries in the video content in combination with the metadata; the process includes: adjusting the video frame input size based on the resolution in the metadata; capturing the short-term action details and long-term semantic dependencies of the video using the spatiotemporal attention mechanism; wherein the short-term action details are realized by extracting local spatiotemporal features, and the long-term semantic dependencies are realized by modeling global spatiotemporal features; dynamically adjusting the window step size using the sliding window mechanism in combination with the frame rate in the metadata to optimize the initial division of the action boundary; predicting the timestamp of the action boundary using a target detection regression module, and generating time segments in combination with self-supervised learning; Using FFmpeg tool to split the video according to the time segment to generate a plurality of video segments; The plurality of video clips are processed by: extracting text descriptions of the video clips using a second artificial intelligence model, wherein the second artificial intelligence model uses multimodal fusion and timestamp embedding for dynamic semantic alignment of actions and text; extracting key frames of the video clips using an FFmpeg tool to generate image groups, and performing semantic analysis on the image groups using a third artificial intelligence model to generate structured action labels, wherein the third artificial intelligence model combines object detection and cross-frame consistency constraints to identify objects in the video clips and their action relationships; The process of performing semantic analysis on the picture group by a third artificial intelligence model to generate a structured action label includes: adjusting the input size of the picture group based on the resolution in the metadata; identifying the objects and their spatial relationships in each frame of the picture group by object detection technology, and extracting action-related visual features; applying cross-frame consistency constraints to the frame sequence by a conditional random field model; and generating a structured action label based on the identified objects, spatial relationships, and cross-frame consistency constraints, wherein the structured action label includes object, action, and timestamp information. Integrate time segments, keyframes, text descriptions, and action labels to generate structured output.

2. The video content structured disassembly and analysis method based on time series segmentation according to claim 1 is characterized in that: The metadata of the video includes: resolution, frame rate and duration.

3. The video content structured disassembly and analysis method based on time series segmentation according to claim 1 is characterized in that: The process of extracting a text description of the video clip by the second artificial intelligence model includes: Extracting visual features of the video clip based on the resolution in the metadata, and extracting audio features based on the audio stream; Mapping action time points into action time vector representations through timestamp embedding; Integrating the visual features, the audio features, and the action time vector representation through a multimodal fusion technique, and using a contrastive learning method to separate background interference; A preliminary text description is generated based on the fused features, and the text fluency and semantic accuracy are optimized through the language model.

4. The video content structured disassembly and analysis method based on time series segmentation according to claim 1 is characterized in that: The process of using the FFmpeg tool to extract the key frames of the video clip to generate a picture group includes: Setting FFmpeg extraction parameters based on the resolution and frame rate in the metadata, adjusting the keyframe output size and extraction frequency; Extracting key frames of the video clip using a scene change detection algorithm, and selecting frames with significant semantic changes; Using an artificial intelligence model to verify the semantic importance and image quality of the keyframes, retaining keyframes related to the action; The extracted key frames are organized into image groups in chronological order and grouped by action stages using a semantic clustering algorithm; The picture group is compressed based on the resolution in the metadata to generate an optimized picture group.

5. A video content structured disassembly and analysis system based on time series segmentation, characterized in that: include: Video data acquisition unit, used to obtain videos uploaded by users and extract video metadata; A temporal segmentation unit is configured to temporally segment the video using a first artificial intelligence model to generate a plurality of time segments, wherein the first artificial intelligence model employs a spatiotemporal attention mechanism and a sliding window mechanism, and optimizes the recognition of action boundaries in the video content in combination with the metadata; the process comprises: adjusting the video frame input size based on the resolution in the metadata; capturing the short-term action details and long-term semantic dependencies of the video using the spatiotemporal attention mechanism; wherein the short-term action details are realized by extracting local spatiotemporal features, and the long-term semantic dependencies are realized by modeling global spatiotemporal features; dynamically adjusting the window step size using the sliding window mechanism in combination with the frame rate in the metadata to optimize the initial division of the action boundary; predicting the timestamps of the action boundaries using an object detection regression module, and generating time segments in combination with self-supervised learning; A video segmentation unit, configured to segment the video according to the time segments using an FFmpeg tool to generate a plurality of video segments; A semantic analysis and structured processing unit is used to process the several video clips, and the processing process is as follows: extracting a text description of the video clip through a second artificial intelligence model, wherein the second artificial intelligence model adopts multimodal fusion and timestamp embedding for dynamic semantic alignment of action and text; using the FFmpeg tool to extract the key frames of the video clip to generate a picture group, and performing semantic analysis on the picture group through a third artificial intelligence model to generate a structured action label, wherein the third artificial intelligence model combines target detection and cross-frame consistency constraints to identify objects and their action relationships in the video clip; wherein the process of performing semantic analysis on the picture group through the third artificial intelligence model to generate a structured action label includes: adjusting the input size of the picture group based on the resolution in the metadata; identifying the objects and their spatial relationships in each frame of the picture group through target detection technology, and extracting action-related visual features; imposing cross-frame consistency constraints on the frame sequence through a conditional random field model; generating a structured action label based on the identified objects, spatial relationships and cross-frame consistency constraints, wherein the structured action label contains object, action and timestamp information; The integration and output unit is used to integrate time segments, key frames, text descriptions and action labels to generate structured outputs.

Images