Video content reproduction method, system, device and medium based on ai information perception

Abstract

This invention discloses a method, system, device, and medium for video content reproduction based on AI information perception. The method specifically includes: performing frame sequence analysis on the original video, combining audio features for multimodal fusion, and outputting a set of segmented video clips; dynamically selecting keyframes based on the video clip set according to content complexity to obtain a keyframe sequence; generating a script description with causal logic based on the keyframe sequence to obtain a structured script and action description text; combining the structured script, action description text, and a user preference model to generate a personalized image sequence; and using motion fidelity metrics for motion control and optimization based on the personalized image sequence and the action information of the original video to synthesize a target video. This invention achieves the maximum preservation of the original video's storyboard, plot, and actions when the main body of the video is changed, generating a target video with high visual consistency and narrative coherence.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, system, device, and medium for reproducing video content based on AI information perception. Background Technology

[0002] With the continuous development of video processing technology, video reconstruction technology has shown great application potential in many fields such as film and television production, creative design, and cultural dissemination. Existing video reconstruction technologies are mainly divided into two categories: one focuses on the overall visual effect transformation of the video, and the other generates video content based on text descriptions. However, both of these technologies have significant limitations in practical applications.

[0003] Currently, most video reconstruction technologies focus on converting the overall color, texture, or artistic style of a video. These methods alter the visual effects of a video by adjusting its color parameters, texture features, or applying specific artistic style filters. However, this approach often struggles to preserve the original video's storyboard, plot, and action. During style conversion, the structural information of the original video, such as the rhythm of shot transitions, the progression of the storyline, and the continuity of character movements, is easily disrupted. For example, when converting a film from a realistic style to a cartoon style, although the overall visual style changes, the sequence of shots in the film may be disrupted, the continuity of the plot may be affected, and character movements may become unnatural, resulting in a significant reduction in the quality of the final video and failing to meet the requirements for video content integrity and continuity in practical applications.

[0004] Some video generation technologies can generate video content based on text descriptions, offering new ideas for video creation. However, these technologies have significant shortcomings when handling complex scenes and actions. Because complex scenes contain a wealth of details and diverse elements, and actions exhibit rich variations and dynamic characteristics, existing video generation models struggle to accurately understand and reproduce this information. For example, when generating a video featuring multi-person interaction, complex actions, and rich scenes, the resulting video often suffers from unnatural movements, missing scene details, or discrepancies with the text description, resulting in limited video quality and fidelity, failing to achieve the desired effect.

[0005] Most existing video processing methods are based on image or pixel-level processing. This approach focuses on extracting and modifying local features of each frame in a video, lacking an understanding of the overall plot and semantics. As a dynamic visual medium, video conveys its plot and semantics through a series of continuous shots and actions. Image or pixel-level processing cannot capture this deeper information, resulting in videos that lack coherence and logic. For example, when processing a story-driven video, simply performing style transfer or feature modification on each frame cannot guarantee the logical progression of the plot and the continuity of character actions, making it difficult for the resulting video to form a complete and meaningful narrative. Summary of the Invention

[0006] The purpose of this invention is to provide a method, system, device and medium for video content reproduction based on AI information perception, which achieves the maximum preservation of the original video's storyboard, plot and actions when the main body of the video is changed, and generates a target video with high visual consistency and plot coherence, thereby solving at least one of the above-mentioned problems of the prior art.

[0007] In a first aspect, the present invention provides a video content reproduction method based on AI information perception, the method specifically comprising:

[0008] The original video is analyzed frame sequence using a spatiotemporal attention model, and multimodal fusion is performed by combining audio features to output a set of segmented video clips.

[0009] Based on a set of video clips, an adaptive keyframe extraction strategy driven by reinforcement learning is adopted to dynamically select keyframes according to the content complexity and obtain a keyframe sequence that represents the video content.

[0010] Based on the keyframe sequence, a pre-built video-text-action knowledge graph is invoked to generate a script description with causal logic, thereby obtaining a structured script and action description text;

[0011] By combining structured scripts and action description texts with a pre-set user preference model, the image diffusion model is driven to generate personalized image sequences that match the original video actions and user style preferences.

[0012] Based on personalized image sequences and motion information from original videos, motion control and optimization are performed using motion fidelity metrics to synthesize target videos with high visual consistency and narrative coherence.

[0013] Secondly, the present invention provides a video content reproduction system based on AI information perception, the system specifically comprising:

[0014] The video segmentation module is used to perform frame sequence analysis on the original video using a spatiotemporal attention model, and to perform multimodal fusion by combining audio features, outputting a set of segmented video clips.

[0015] The frame extraction module is used to extract keyframes based on a set of video clips using a reinforcement learning-driven adaptive keyframe extraction strategy. It dynamically selects keyframes according to the complexity of the content to obtain a keyframe sequence that represents the video content.

[0016] The knowledge graph module is used to generate script descriptions with causal logic based on keyframe sequences by calling pre-built video-text-action knowledge graphs, and obtain structured scripts and action description texts.

[0017] The image diffusion module is used to combine structured scripts and action description texts with a preset user preference model to drive the image diffusion model to generate personalized image sequences that match the original video actions and user style preferences.

[0018] The video optimization module is used to perform motion control and optimization based on personalized image sequences and motion information from the original video, using motion fidelity metrics to synthesize target videos with high visual consistency and narrative coherence.

[0019] Thirdly, the present invention provides a computer device, including: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements the video content reproduction method based on AI information perception as described in any of the above methods.

[0020] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video content reproduction method based on AI information perception as described in any of the above methods.

[0021] Compared with the prior art, the present invention has at least one of the following technical effects:

[0022] 1. This invention achieves the maximum preservation of the original video's storyboard, plot, and actions when the main body of the video is changed, generating a target video with high visual consistency and plot coherence.

[0023] 2. This invention utilizes a spatiotemporal attention model to extract spatiotemporal features from video and combines them with audio features for multimodal fusion, which can accurately predict shot switching points, thereby reasonably segmenting the original video into a set of segmented video segments, providing a structured foundation for subsequent processing.

[0024] 3. This invention constructs a network architecture that includes a spatiotemporal attention layer to sample, segment, encode, and propagate video segments. It updates parameters based on the loss value of the classification task, enabling the TimeSformer spatiotemporal attention model to effectively extract and fuse video spatiotemporal features.

[0025] 4. This invention adopts a reinforcement learning-driven adaptive keyframe extraction strategy, which dynamically selects keyframes based on content complexity, adjusts the decision model parameters by calculating composite reward signals, and finally obtains a keyframe sequence that accurately represents the video content, effectively preserving the core information of the video.

[0026] 5. This invention performs visual semantic encoding on keyframe sequences, retrieves relevant knowledge triples from the knowledge graph, and fuses them to generate a structured script and action description text, so that the generated script has causal logic and provides a semantic basis for subsequent personalized image generation.

[0027] 6. This invention combines structured scripts and action description texts with a preset user preference model to drive an image diffusion model to generate personalized image sequences that match the original video actions and user style preferences. This ensures consistency between the generated images and the original video actions while meeting the user's personalized needs.

[0028] 7. Based on personalized image sequences and motion information from original videos, this invention uses motion fidelity metrics for motion control and optimization, ensuring high visual consistency and narrative coherence in the target video. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating a video content reproduction method based on AI information perception, provided in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the structure of a video content reproduction system based on AI information perception provided in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0037] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0039] In this application embodiment, the entity executing the process includes a terminal device. This terminal device includes, but is not limited to, devices capable of executing the methods disclosed in this application, such as servers, computers, smartphones, and tablets. Figure 1 A flowchart illustrating a video content reproduction method based on AI information perception, according to an embodiment of the present invention, is shown below in detail:

[0040] S101 uses a spatiotemporal attention model to perform frame sequence analysis on the original video, and combines audio features to perform multimodal fusion, outputting a set of segmented video clips.

[0041] In this embodiment, the input raw video is parsed frame by frame, decomposing the video into a continuous sequence of image frames, and recording the timestamp information of each frame. For each image frame, a pre-trained convolutional neural network (such as ResNet or EfficientNet) is used to extract spatial features, which include static information such as the shape, color, and texture of objects within the frame. Simultaneously, optical flow or a 3D convolutional network is used to analyze motion changes between adjacent frames, extracting dynamic features in the temporal dimension, such as the trajectory of objects and changes in speed. The spatiotemporal attention model uses a self-attention mechanism to weightedly fuse spatial and temporal features, focusing on regions of significant change (such as human actions and scene transitions) and regions of sustained stability (such as the background environment) in the video, generating a feature map containing spatiotemporal correlation information.

[0042] Simultaneously with video frame processing, audio stream data synchronized with the video is extracted. A short-time Fourier transform is used to convert the audio signal into a time-spectrum graph, and semantic features of the audio, including speech content, ambient sound effects, and musical rhythm, are extracted using a pre-trained audio coding network (such as VGGish or PANNs). To ensure audio-video synchronization, audio features are segmented and aligned according to the timestamps of video frames, ensuring that each video frame corresponds to audio features within a specific time window. Furthermore, a cross-modal attention mechanism is used to analyze the correlation between audio features and video spatiotemporal features, such as recognizing the matching degree between speech and lip movements in dialogue scenarios, and the synchronization between action sound effects and video motion, generating a fused multimodal feature representation.

[0043] The video and audio features output from the spatiotemporal attention model are input into the multimodal fusion module, where a dynamic weighting strategy balances the contributions of visual and auditory information. For example, the weight of audio features is increased in dialogue-intensive scenes, while video dynamic features are emphasized in action-packed scenes. The fused features are then used for temporal modeling via a bidirectional long short-term memory (Bi-LSTM) network to capture long-range dependencies in the video content. Simultaneously, a segmentation prediction branch is introduced, outputting a segmentation probability value for each time point through a fully connected layer. This probability value reflects the likelihood of the current location being a segmentation point. For example, the segmentation probability significantly increases when the video scene changes from indoors to outdoors, or when the audio environment changes from quiet to noisy.

[0044] A dynamic segmentation threshold is set, which adaptively adjusts based on the complexity of the video content. For example, in fast-paced action videos, the threshold is appropriately lowered to capture fine-grained segments; in static interview videos, the threshold is raised to avoid over-segmentation. The segmentation probability sequence is traversed, and when a probability value exceeds the current threshold, that time point is marked as a potential segmentation point. Furthermore, redundant segmentation points are removed using the Non-Maximum Suppression (NMS) algorithm, retaining the position with the highest local probability as the final segmentation point. Finally, based on the segmentation points, the original video is cut into multiple segments, each containing semantically complete video content (such as a dialogue turn or an action scene) and a corresponding audio segment, forming a set of segmented video segments.

[0045] To ensure segmentation accuracy, a dual verification mechanism is employed: First, segmentation accuracy is evaluated using a small-scale, manually annotated test set, calculating the error between the segmentation points and the actual scene transition locations. Second, predefined rules (such as segment duration thresholds and content similarity) are used to automatically filter out abnormal segments (e.g., excessively short segments or repetitive scenes). If verification fails, the parameters of the spatiotemporal attention model (such as attention weights and feature fusion ratios) or the calculation method of the dynamic threshold are adjusted, and the segmentation process is re-executed until the preset accuracy requirements are met. The final output set of video segments retains the semantic integrity of the original video and provides structured input for subsequent keyframe extraction and script generation.

[0046] This embodiment achieves accurate video segmentation based on spatiotemporal attention and audio fusion, effectively solving the problem of scene switching omissions or incorrect segmentation caused by relying solely on visual information in traditional methods, and providing a reliable foundation for video content reproduction.

[0047] S102, based on a set of video clips, adopts a reinforcement learning-driven adaptive keyframe extraction strategy to dynamically select keyframes according to the content complexity, thereby obtaining a keyframe sequence representing the video content.

[0048] In this embodiment, multi-dimensional feature extraction is performed on each segment in the segmented video clip set. For each video clip, a pre-trained spatiotemporal feature extraction network (such as a 3D convolutional neural network or a two-stream network) is used to extract spatial features (such as object shape and scene layout) and temporal features (such as motion amplitude and speed). Simultaneously, optical flow analysis is used to calculate the pixel displacement of adjacent frames within the clip to quantify the intensity of the action; an object detection model is used to count the types and quantities of objects appearing in the clip to assess scene complexity. Furthermore, audio features (such as volume fluctuations and speech rate) and text descriptions (such as automatically generated clip summaries) are combined to comprehensively calculate a content complexity score for each video clip. This score covers three dimensions: motion density, scene diversity, and semantic richness, providing a quantitative basis for subsequent keyframe extraction.

[0049] A reinforcement learning environment is constructed, treating each video segment as an independent learning unit. The state space of the reinforcement learning is defined as the spatiotemporal feature vector of the current segment, its complexity score, and the spatiotemporal distribution of extracted keyframes. For example, the state can be represented as [segment duration, action intensity, number of objects in the scene, and time interval between the most recent keyframes]. The action space is defined as the operation of selecting a frame as a keyframe within a segment, with action values ​​ranging from the indices of all frames within the segment. The reward function is designed as a multi-objective optimization, containing three sub-objectives: first, the similarity between the keyframe and the core content of the segment (calculated using a pre-trained image-text matching model); second, the balance of time intervals between keyframes (avoiding keyframes being too concentrated or sparse); and third, the overall coverage of the segment by the keyframe sequence (ensuring that keyframes cover the main actions and scene changes of the segment). A weighted summation is used to obtain a comprehensive reward value, which guides the agent to learn the optimal extraction strategy.

[0050] A Deep Q-Network (DQN) is used as the reinforcement learning agent. Its input is the state vector of the current segment, and its output is the Q-value for each possible action (i.e., selecting a frame). During training, the agent selects actions using an exploration-exploitation balancing strategy (e.g., ε-greedy): randomly selecting frames as keyframes with probability ε (exploration), and selecting the frame with the highest Q-value with probability 1-ε (exploitation). After each action is executed, an immediate reward is calculated based on the reward function, and the state is transitioned to the next frame, forming an experience tuple (state, action, reward, next state). A large number of experience tuples are stored through an experience replay mechanism, and random sampling is used for batch training to update the DQN parameters to minimize the difference between the predicted Q-value and the target Q-value. During training, the weight parameters of the reward function are dynamically adjusted; for example, the weight of similarity rewards is increased in action-dense segments, and the weight of coverage rewards is increased in static scene segments, allowing the agent to adapt to video content of varying complexity.

[0051] A dynamic thresholding mechanism is introduced to adjust the strictness of keyframe extraction based on the segment complexity score. For high-complexity segments (such as those involving multi-person interaction or rapid action switching), the similarity threshold is lowered and the keyframe time interval is shortened to ensure the capture of fine-grained action changes. For low-complexity segments (such as single-scene dialogue), the similarity threshold is increased and the time interval is extended to avoid redundant keyframes. In the candidate keyframe sequence output by the agent, a non-maximum suppression (NMS) algorithm is further applied: the content similarity between adjacent keyframes is calculated; if the similarity exceeds a preset threshold, the frame with the higher Q-value is retained and the other frame is deleted. Through multiple rounds of iterative optimization, a keyframe sequence that both covers the core content of the segment and avoids redundancy is finally generated.

[0052] To ensure the quality of the keyframe sequence, a dual verification mechanism is employed: First, the content matching degree between keyframes and original segments is evaluated using a manually annotated test set, calculating recall (the proportion of core actions covered) and precision (the proportion of redundant frames); second, abnormal sequences (such as keyframes concentrated at the beginning or end of a segment) are automatically detected using predefined rules (e.g., upper limit on the number of keyframes, uniformity of temporal distribution). If verification fails, the parameters of the reinforcement learning model (e.g., reward function weights, exploration probability ε) or the calculation method of the dynamic threshold are adjusted, and the keyframe extraction process is re-executed. The final output keyframe sequence must meet the following requirements: each segment contains at least one keyframe, the time interval between keyframes does not exceed a preset threshold, and the content of the keyframes covers the main actions, scene changes, and semantic information of the segment.

[0053] This embodiment implements adaptive keyframe extraction based on reinforcement learning, which can dynamically adjust the extraction strategy according to the complexity of video segments and generate a keyframe sequence that is both representative of the video content and logically coherent, providing a high-quality foundation for subsequent script generation and video reproduction.

[0054] S103, based on the keyframe sequence, calls the pre-built video-text-action knowledge graph to generate a script description with causal logic, and obtains a structured script and action description text.

[0055] In this embodiment, the keyframe sequence is preprocessed to unify the resolution and color space of all frames and eliminate noise (such as blurring and flickering) that may be generated during the extraction process. Then, a pre-trained multimodal feature extraction model (such as a hybrid architecture combining ResNet and CLIP) is used to vectorize the features of each frame. This model simultaneously extracts visual features (including object category, scene type, and person's pose) and motion features (calculating motion direction and amplitude using optical flow), and fuses them into a feature vector containing spatiotemporal information. For example, if a "person raising a glass" action appears in a keyframe, its feature vector will include the spatial relationship of "person-hand-glass" and the temporal feature of "upward movement."

[0056] The preprocessed keyframe feature vectors are input into a pre-constructed video-text-action knowledge graph. This graph stores knowledge in triples, containing three types of nodes: visual nodes (e.g., "character running," "sunset scene"), text nodes (e.g., "chase," "farewell scene"), and action nodes (e.g., "jump," "handshake"), connected by semantic association edges (e.g., "cause," "accompany"). The system first matches the node most similar to the keyframe features at the visual node level (e.g., using cosine similarity calculation), then extends along the association edges to text and action nodes. For example, if a keyframe matches the visual node "character raising a glass," the graph might associate it with the text node "celebration" via the "accompany" edge, and then with the action node "clinking glasses" via the "cause" edge. Breadth-first search (BFS) is used to construct node paths from the first frame to the last frame, forming a preliminary narrative chain.

[0057] The initially constructed narrative chain undergoes causal logic verification. The system calls upon a predefined logical rule base in the graph (e.g., "quarrels are usually followed by separation" and "a goal must be achieved before celebration") to check whether adjacent nodes conform to causal relationships. If a logical conflict is found (e.g., "raising a glass" directly relates to the "sadness" text node), the path is adjusted through a backtracking mechanism: high-confidence nodes (e.g., nodes that highly match keyframe features) are retained first, low-confidence nodes are deleted, and alternative paths are searched again. Simultaneously, time constraint rules are introduced (e.g., action intervals must conform to real-world time logic) to ensure the temporal order of the narrative chain is reasonable. For example, if two keyframes are close together but the associated actions are "running" and "sitting still," the system will insert a transitional action node (e.g., "decelerating") to smooth the logic.

[0058] Based on the optimized narrative chain, the system generates structured text in two steps. First, text nodes are arranged chronologically and populated into their corresponding positions in the script template (e.g., scene descriptions, character dialogue). The script template contains standardized fields (e.g., "scene number," "character list," "action sequence") to ensure a consistent output format. Second, detailed action description text is generated for each action node: simple actions (e.g., "handshake") directly reference the standard description from the atlas; complex actions (e.g., "multi-person dance") are broken down into sub-action sequences (e.g., "raise hand - spin - land"), with each sub-action labeled with its participants, duration, and spatial location. For example, if a keyframe sequence is associated with the "wedding ceremony" text node, the system will generate description text containing subdivided actions such as "bride's entrance (walking slowly, holding a bouquet)" and "exchanging rings (hands handed over with both hands, eye contact)."

[0059] To ensure complete consistency between the generated script and action text and the original keyframe content, the system performs multimodal consistency checks. On one hand, it verifies whether the text description matches the visual content of the keyframes using an image-text matching model (such as BLIP). On the other hand, it verifies the accuracy of the action descriptions by calculating action similarity (e.g., comparing the optical flow features of the generated actions with those of the actual actions in the keyframes). If inconsistencies are found (e.g., the text description is "running" but the keyframe shows "walking slowly"), a correction process is triggered: the text description is adjusted first to match the visual content; if the visual content is ambiguous (e.g., blurry frames), reasonable inferences are made by referring to common actions in similar scenes in the atlas. The final output structured script includes scene numbers, a list of characters, storyboard descriptions, and dialogue, while the action description text is annotated with the participants, type, and duration of each action in a timeline format.

[0060] This embodiment achieves efficient conversion from keyframe sequences to structured scripts and action description texts, ensuring that the generated text is faithful to the original video content and has clear causal logic and narrative coherence, providing a reliable foundation for subsequent personalized video generation.

[0061] S104, combining structured scripts and action description texts with a preset user preference model, drives the image diffusion model to generate personalized image sequences that match the original video actions and user style preferences.

[0062] In this embodiment, the structured script and action description text are spatiotemporally aligned. Specifically, a spatiotemporal mapping relationship between text descriptions and keyframe sequences is established based on the scene numbers in the script and the timestamps of the action descriptions. For example, if the description of the third scene in the script is "The protagonist is running in the rain," and the corresponding action description is "The protagonist (ID=1) moves quickly from left to right, with a stride of 0.8 meters / step, lasting 3 seconds," then this text is associated with frames 10-15 in the keyframe sequence (covering a duration of 3 seconds). Subsequently, the script and action descriptions are decoupled by feature extraction: semantic features (such as scene type "rainy day," emotional tone "tense") are extracted from the script, and motion features (such as movement direction "horizontally to the right," speed parameter "0.8 meters / step") are extracted from the action descriptions, and encoded into semantic vectors and motion vectors, respectively. Simultaneously, a preset user preference model is loaded, which generates style preference vectors (such as "cyberpunk style," "low saturation tone") by analyzing user historical behavior data (such as viewing history, style collections).

[0063] The decoupled semantic vector, motion vector, and style preference vector are input into the conditional input layer of the image diffusion model for dynamic fusion. The system employs an attention mechanism to weight the three types of vectors: for motion-related image generation (such as running), the weight of the motion vector is increased; for style-related image generation (such as scene color tone), the weight of the style preference vector is increased. For example, when generating an image of "running in the rain," the model focuses on the "fast movement" feature in the motion vector, while also incorporating the "low saturation" style from the user's preference to adjust the color parameters of the generated image. During the initialization of the diffusion model, pre-trained base weights (such as the general weights of Stable Diffusion) are loaded, and an adaptive conditional encoder is inserted. This encoder can adjust the model parameters in real time according to the dynamic changes of the input vector, ensuring a sensitive response to multimodal conditions during the generation process.

[0064] The image generation process is divided into two stages: coarse-grained generation and fine-grained optimization. In the coarse-grained stage, the model generates an initial image based on the fused conditional vectors, focusing on ensuring the macroscopic consistency of the action. For example, when generating the initial image of "the protagonist running," the model prioritizes ensuring that the character's posture (such as alternating leg extensions) and direction of movement (horizontally to the right) conform to the action description, while ignoring detailed textures. At this point, an action fidelity constraint is introduced: by comparing the spatial displacement of the character's joints (such as the trajectories of the hip and knee joints) in the generated image with those in the keyframes of the original video, an action similarity score is calculated. If the score is lower than a threshold (such as 80%), a feedback mechanism is triggered, adjusting the input parameters of the motion vector (such as increasing the stride length by 0.1 meters / step) and regenerating the image. In the fine-grained stage, the model combines semantic vectors and style preference vectors to optimize image details. For example, based on the description of "rainy day" in the script, raindrop effects are added to the image; based on the user's preference for "cyberpunk style," the neon lighting effects of the building outlines are adjusted. At the same time, a style transfer module ensures global style consistency and avoids local style conflicts (such as inconsistencies between the character's clothing and the background color).

[0065] During the generation process, the system monitors the matching degree between the user preference model and the generated image in real time. A pre-trained style evaluator (such as a CLIP-based contrastive learning model) calculates the similarity score between the generated image and the user preference vector. If the score is lower than a preset value (e.g., 75%), a dynamic style adjustment process is initiated: First, the specific dimensions of the preference mismatch (e.g., hue, composition, element type) are analyzed; for example, the user prefers "low saturation" but the generated image has excessively high saturation. Then, the corresponding parameters of the style preference vector are adjusted accordingly (e.g., reducing the saturation weight by 0.2), and the image is re-input into the model to generate the corrected image. To avoid over-adjustment leading to motion distortion, the system sets an upper limit threshold for style adjustment (e.g., saturation adjustment range ±15%) to ensure that style optimization is based on motion consistency. For example, if the user prefers a "retro filter" but the original video is a modern scene, the model will simulate a retro style by preserving the modern architectural structure and adding a yellowish hue and graininess, rather than directly modifying the architectural form.

[0066] After generating a complete image sequence, the system performs multi-scale quality assessment. At the pixel level, the visual similarity between the generated images and keyframes of the original video is evaluated using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). At the semantic level, the semantic consistency between the images and the script description is evaluated using a pre-trained visual-language model (such as BLIP). At the motion level, the smoothness of motion of characters / objects between consecutive images is calculated using optical flow. If any scale assessment fails (e.g., PSNR < 30dB or semantic consistency score < 0.8), the problematic frame is located and local regeneration is triggered: for frames with insufficient visual similarity, the diffusion steps are increased to improve detail quality; for frames with semantic inconsistency, the input description of the semantic vector is corrected; for frames with non-smooth motion, the motion vector is recalculated and the keypoint trajectory is adjusted. Finally, a temporal smoothing filter is used to eliminate flicker and jitter in the image sequence, outputting a personalized image sequence with coherent motion, consistent style, and user preferences.

[0067] This embodiment achieves a deep integration of structured scripts, action descriptions, and user preferences, driving an image diffusion model to generate high-quality personalized image sequences, providing a solution for video replay technology that combines content accuracy and style customization.

[0068] S105, based on personalized image sequences and motion information from original videos, uses motion fidelity metrics for motion control and optimization to synthesize target videos with high visual consistency and narrative coherence.

[0069] In this embodiment, motion information is extracted from the original video and a motion parameter model is constructed. The system uses optical flow combined with human pose estimation technology (such as OpenPose) to analyze the motion of each frame of the original video and extract key motion parameters, including the spatial coordinates of the character's joints (such as the two-dimensional / three-dimensional positions of the shoulders, elbows, and hips), motion trajectories (such as the path curves of horizontal displacement and vertical jumps), velocity (such as the amount of displacement per unit time), and acceleration (such as the rate of velocity change). Simultaneously, for complex actions (such as rotation and roll), the system generates rotation angle and axial information of the action through three-dimensional reconstruction technology. For example, when analyzing the action of "the protagonist jumping down the stairs," the system records the take-off point (coordinates X1, Y1), the air rotation angle (30 degrees around the Y-axis), the landing point (coordinates X2, Y2), and the velocity at each stage (take-off velocity 2 m / s, air velocity 1.5 m / s). All parameters are sorted by timestamp to construct an original motion parameter library, serving as a benchmark for subsequent optimization.

[0070] The system performs reverse motion parameter parsing on the generated personalized image sequences. It locates people or moving subjects in the images using a target detection model (such as YOLOv8) and extracts their joint coordinates using a keypoint detection algorithm (such as HRNet). Then, it calculates the displacement of joints in adjacent images using inter-frame difference to derive the motion trajectory, velocity, and acceleration. For example, if frames 5 to 10 in the image sequence show a person's shoulder coordinates moving from (100, 200) to (150, 180), the system calculates a horizontal displacement of 50 pixels and a vertical displacement of -20 pixels. Combined with the frame interval (assuming 30 frames per second, with a 5-frame interval of approximately 0.17 seconds), the horizontal velocity is calculated to be approximately 294 pixels per second. Simultaneously, the rotation angle of the motion (such as shoulder rotation during arm swing) is estimated using a rotation matrix. All parsing results are aligned by timestamps to generate an image sequence motion parameter set for comparison with the original motion parameters.

[0071] Based on the original motion parameter library and the image sequence motion parameter set, the system calculates motion fidelity indicators and locates errors. The system defines multi-dimensional fidelity indicators, including spatial fidelity (joint displacement error), temporal fidelity (motion occurrence time deviation), morphological fidelity (motion posture similarity), and physical plausibility (such as whether speed / acceleration conforms to human motion patterns). For example, spatial fidelity is obtained by calculating the Euclidean distance between the joint coordinates in the image sequence and the corresponding coordinates in the original video; temporal fidelity is calculated by comparing the time difference between the start and end frames of the motion. If the jump start time of a character in a frame is 2 frames later than the original video (approximately 0.06 seconds), the temporal fidelity score decreases. The system sets thresholds (e.g., spatial fidelity error > 10 pixels, temporal fidelity deviation > 0.1 seconds). When any indicator exceeds the limit, the frame is marked as an error frame, and the error type (e.g., displacement deviation, time delay) is recorded.

[0072] For positioning error frames, a categorized dynamic compensation strategy is employed. For spatial displacement errors, the joint positions are corrected by adjusting the latent vector of the image generation model. For example, if the right arm of the character in frame 8 is 15 pixels to the right of the original video, the negative weight (-0.3) of the "right arm horizontal displacement" parameter in the model input is increased, causing the right arm to move to the left in the generated image. For temporal errors, the rhythm of the action is adjusted by inserting or deleting transition frames. For example, if a jumping action ends prematurely in the image sequence, two transition frames are inserted after frame 10 to gradually reduce the vertical velocity parameter, making the landing action more natural. For physical inconsistencies (such as excessive speed), a motion constraint model (e.g., a speed limit based on human biomechanics) is introduced to restrict the parameter range of the generated action. For example, if the calculated running speed of the character exceeds 8 m / s (far exceeding human limits), the speed parameter is forcibly adjusted to 5 m / s, and the corresponding frame is regenerated.

[0073] After local error compensation, the system performs global temporal constraint optimization to ensure smooth action and narrative continuity. An Action State Transition Graph is constructed to describe the complete state sequence of an action from start to finish (e.g., "jump → mid-air spin → landing"). The system compares the action state transitions of the image sequence with the transition paths of the original video. If a state is missing (e.g., the "mid-air spin" state is missing) or the sequence is incorrect (e.g., "landing" appears before "jumping"), the system corrects this by adjusting the keyframe generation order or inserting transitional actions. For example, if the "jump" action in the image sequence jumps directly from "standing" to "landing," lacking intermediate states, keyframes for the "jump" and "mid-air" states are inserted, and the motion parameters of the corresponding frames are adjusted. Simultaneously, a temporal smoothing filter (e.g., Gaussian filtering) eliminates abrupt changes in action, ensuring continuous, jump-free changes in joint trajectory and speed. For example, filtering the trajectory of a character's arm swing makes the change in swing amplitude from large to small smoother.

[0074] After optimization, the system performs a multi-dimensional quality assessment. At the action level, fidelity metrics are recalculated to ensure spatial error <5 pixels, temporal error <0.05 seconds, and physical plausibility compliance >95%. At the visual level, SSIM (Structural Similarity Scheme) is used to evaluate the visual consistency between the generated images and keyframes of the original video, requiring a score >0.85. At the narrative level, human evaluators (or a pre-trained narrative understanding model) are invited to judge the rationality of the generated video's narrative logic (e.g., whether actions conform to the script description and whether scene transitions are natural). If any dimension fails to meet the evaluation criteria, the system returns to the corresponding step to adjust parameters (e.g., increasing compensation intensity or re-optimizing temporal constraints). Finally, the optimized personalized image sequence is synthesized chronologically, and the audio track of the original video is added (audio alignment technology ensures audio-visual synchronization), outputting a target video with high visual consistency (image style and color match the original video) and narrative coherence (action logic and narrative rhythm meet expectations).

[0075] This embodiment realizes refined motion control based on motion fidelity index, effectively solving the problems of motion distortion and narrative discontinuity in the prior art, and providing a high-fidelity and highly coherent solution for video reproduction technology.

[0076] In some embodiments, step S101 above, which involves using a spatiotemporal attention model to perform frame sequence analysis on the original video and simultaneously combining audio features for multimodal fusion to output a segmented set of video clips, specifically includes:

[0077] The original video is input into the pre-trained TimeSformer spatiotemporal attention model, and the spatiotemporal features of the video frame sequence are extracted through the spatiotemporal attention mechanism to obtain the video feature sequence.

[0078] Based on the original video, audio features of its audio track are extracted synchronously, and the audio features are aligned with the video feature sequence by timestamps to obtain time-aligned multimodal feature pairs;

[0079] All multimodal feature pairs are fused to obtain a multimodal feature sequence, and the probability distribution of shot switching points is predicted based on the multimodal feature sequence.

[0080] Based on the probability distribution of shot switching points, peak detection is used to determine the shot boundary positions, and the original video is segmented into a set of video segments based on the shot boundary positions.

[0081] In this embodiment, the original video is input into a pre-trained TimeSformer spatiotemporal attention model. This model employs a self-attention mechanism, simultaneously capturing spatial features (such as object shape, texture, and person's posture) and temporal features (such as action continuity and direction of motion) in the video frame sequence. Specifically, the model decomposes the video into a continuous sequence of frames (e.g., 30 frames per second), performs self-attention calculations in the spatial dimension for each frame, and extracts local spatial features (such as facial expressions and object positions); simultaneously, it captures dynamic changes between frames (such as the transition from a person standing to running) through self-attention calculations in the temporal dimension. For example, when processing a video of "a person pushing open a door to enter a room," the model focuses on the change in the door's position (spatial features) and the temporal correlation between the person's hand movements and the door's movement (temporal features). Finally, the model outputs a video feature sequence containing spatiotemporal information. This sequence represents the spatiotemporal features of each frame in the form of feature vectors, with a feature dimension of 512 (example value, actual dimensions depend on the model structure), which is used for subsequent multimodal fusion.

[0082] The audio track is extracted synchronously from the original video, and audio features are obtained using a pre-trained audio feature extraction model (such as VGGish or Wav2Vec2.0). The audio feature extraction process includes: segmenting the audio signal into frames (e.g., each frame is 25ms long, with a 10ms overlap); converting the time-domain signal into frequency-domain features using Short-Time Fourier Transform (STFT); extracting Mel-spectral features using a Mel filter bank; and finally generating a 128-dimensional feature vector for each frame of audio (example value). Subsequently, the audio features are precisely matched with the video feature sequence using timestamp alignment technology. Specifically, using the timestamp of the first frame of the video as a reference, the feature vector at the corresponding time point in the audio feature sequence is paired with the frame feature vector in the video feature sequence to form a multimodal feature pair of "video frame feature - audio frame feature". For example, if the timestamp of the 5th frame of the video is 0.16 seconds, then the audio frame feature with the timestamp closest to 0.16 seconds (such as an audio frame at 0.15 seconds or 0.17 seconds) is found from the audio feature sequence, and this is combined with the feature of the 5th frame of the video to form a multimodal feature pair. In this way, it is ensured that each video frame corresponds to a time-synchronized audio feature, providing a foundation for subsequent fusion.

[0083] All time-aligned multimodal feature pairs are input into the multimodal fusion module. This module fuses video and audio features using either concatenation or weighted summation. For example, concatenation directly concatenates 512-dimensional video frame features with 128-dimensional audio frame features, generating a 640-dimensional multimodal feature vector; weighted summation uses learnable weight parameters (e.g., 0.7 for video features and 0.3 for audio features) to weight the two features and then sum them, generating a 512-dimensional fused feature vector. The fused multimodal feature sequence is then input into a shot transition prediction network (e.g., a fully connected layer + Softmax activation function). This network outputs the probability value of each time point (corresponding to a video frame) being a shot transition point. For example, for a video containing 100 frames, the network outputs 100 probability values ​​(ranging from 0 to 1), representing the likelihood of each frame being a shot transition point. A higher probability value indicates a greater likelihood of a shot transition occurring near that frame.

[0084] Based on the probability distribution of shot transition points, a peak detection algorithm is used to locate shot boundaries. The specific steps are as follows: First, the probability sequence is smoothed (e.g., by moving average filtering) to eliminate noise interference; then, candidate peak points are filtered by setting a threshold (e.g., probability > 0.8); finally, non-maximum suppression (NMS) is applied to the candidate peak points, retaining local maxima as the final shot transition points. For example, if the probability sequence shows peaks with probability values ​​of 0.85, 0.92, and 0.78 at frames 20, 50, and 80 respectively, and the probability value of frame 50 is higher than that of its surrounding frames, then frame 50 is determined to be a shot transition point. Based on the detected shot transition points, the original video is segmented into multiple segments. For example, if the detected shot transition points are located at frames 50 and 100, the original video is segmented into three parts: frames 1-50 are the first segment, frames 51-100 are the second segment, and frame 101 to the end of the video is the third segment. In the final output video clip set, each clip contains complete shot content (such as a complete dialogue scene or an action process), and the switching points between clips are highly consistent with the semantic boundaries of the original video.

[0085] This embodiment achieves accurate video segmentation based on the fusion of spatiotemporal attention model and audio features, effectively solving the problem of false detection of shots caused by traditional methods that rely solely on visual features. It provides a foundation of video segments with clear structure and complete semantics for subsequent tasks such as video reproduction and editing.

[0086] Furthermore, the pre-training steps of the TimeSformer spatiotemporal attention model specifically include:

[0087] A network architecture for constructing the TimeSformer spatiotemporal attention model is provided. The network architecture includes multiple spatiotemporal attention layers, and each spatiotemporal attention layer includes a spatial attention head for analyzing intra-frame spatial relationships and a temporal attention head for analyzing cross-frame temporal relationships.

[0088] Randomly sample a video segment from the prepared training dataset and uniformly sample the video segment to obtain a fixed number of frame sequences;

[0089] Each frame of the frame sequence is divided into multiple image blocks, and each image block is converted into an embedding vector through linear projection. Learnable positional and temporal codes are added to each embedding vector to form an input embedding sequence containing spatiotemporal information.

[0090] The input embedded sequence is fed into the TimeSformer spatiotemporal attention model, and forward propagation is performed through the spatiotemporal attention layer to extract and fuse the spatiotemporal features of the video, and output the fused spatiotemporal features.

[0091] Based on the fusion of spatiotemporal features, a video segment classification task is performed to generate prediction results. The loss value of the model training is calculated based on the prediction results and the preset real labels.

[0092] Based on the loss value, the parameters of the TimeSformer spatiotemporal attention model are updated through backpropagation and gradient descent algorithms, and the model is iteratively trained until it converges.

[0093] In this embodiment, the network structure of the TimeSformer model is designed, with its core consisting of multiple stacked spatiotemporal attention layers. Each spatiotemporal attention layer contains two types of attention heads: spatial attention heads and temporal attention heads. The spatial attention head is used to analyze the spatial relationships between different regions within a single frame of an image. For example, when processing a frame of a video of "a person playing basketball," the spatial attention head focuses on key regions such as the person's hands, the basketball, and the rim, calculating the correlation between these regions (such as the contact relationship between the hands and the basketball) and generating spatial feature representations. The temporal attention head is used to analyze the temporal dynamic relationships between frames. For example, it captures the trajectory of the basketball from being thrown to approaching the rim in consecutive frames, calculates the temporal correlation between frames at different time points, and generates temporal feature representations. The two types of attention heads are combined in parallel or serial manner, enabling the model to simultaneously capture spatial details and temporal evolution information in the video. In addition, the model also includes an embedding layer (used to convert image patches into vectors), a normalization layer (to stabilize the training process), and a feedforward neural network layer (to enhance feature representation capabilities), ultimately outputting a feature vector that fuses spatiotemporal features.

[0094] A video segment is randomly sampled from a pre-prepared training dataset (containing a large number of unlabeled or weakly labeled videos, such as the Kinetics action recognition dataset or the Sports-1M sports video dataset). During sampling, the duration range of the video segment is first determined (e.g., 2 to 10 seconds), and then segments matching this duration are randomly extracted from the video. Subsequently, this segment is uniformly sampled, extracting a fixed number of frame sequences (e.g., 8 frames per second, for a total of 16 frames). The purpose of uniform sampling is to ensure that the frame sequence covers the main content of the video segment while avoiding feature omissions due to uneven sampling. For example, if the video segment contains the action of "a person going from standing to jumping," uniform sampling will extract frames from the start, middle, and end phases of the action, providing complete information for subsequent spatiotemporal feature extraction.

[0095] Each frame in the frame sequence is segmented into multiple non-overlapping image patches (e.g., a 224×224 pixel frame is segmented into 16×16 pixel image patches, totaling 196 patches). Each image patch is converted into a fixed-dimensional embedding vector (e.g., 768-dimensional) through a linear projection layer (a fully connected neural network). This vector represents the initial features of the image patch. To enable the model to perceive the spatial location and temporal order of the image patches, two types of encoding are added to each embedding vector: positional encoding and temporal encoding. Positional encoding uses a pre-defined fixed vector (e.g., a vector generated by a sine / cosine function) to mark the spatial location of the image patch within a single frame (e.g., top left corner, bottom right corner); temporal encoding uses a learnable parameter vector to mark the temporal order of the frames to which the image patch belongs in the sequence (e.g., frame 1, frame 5). By superimposing the embedding vectors, positional encoding, and temporal encoding, an input embedding sequence containing spatiotemporal information is formed, which serves as the input data for the model.

[0096] The input embedding sequence is fed into the TimeSformer model, and propagated forward sequentially through stacked spatiotemporal attention layers. In each spatiotemporal attention layer, the spatial attention head first performs self-attention computation on the image patch embedding vectors within a single frame: using a query, key, and value mechanism, it calculates the similarity of each image patch to other image patches (e.g., dot product operation), generating spatial attention weights. The value vectors are then weighted and summed according to these weights to obtain the spatial feature representation of a single frame. Subsequently, the temporal attention head performs self-attention computation on image patches at the same location across frames (e.g., the first image patch in all frames), capturing dynamic changes in the temporal dimension and generating temporal feature representations. These two types of features are fused through concatenation or addition to form the spatiotemporal feature output of that layer. After multiple layers of stacking and feature fusion, the model finally outputs a feature vector (e.g., 1024-dimensional) that incorporates global spatiotemporal information.

[0097] Based on the fused spatiotemporal features from the model output, a video segment classification task is performed. Specifically, the feature vector is input into the classification head (a fully connected layer + a softmax activation function) to generate probability distributions for video segments belonging to various categories (e.g., the probabilities of action categories "playing basketball," "running," and "jumping"). The labels for the classification task come from pre-defined annotations in the training dataset (e.g., each video segment corresponds to one action category label). The accuracy of the model's predictions is quantified by calculating the cross-entropy loss between the predicted probability distribution and the true label. For example, if the true label is "playing basketball" (probability 1), and the model predicts a probability of 0.8, the loss value is small; if the predicted probability is 0.2, the loss value is large, indicating that the model needs further optimization.

[0098] Based on the calculated loss value, the error is propagated from the output layer to the input layer layer by layer using the backpropagation algorithm, and the gradient of each parameter (such as the gradient of the weight matrix and bias vector) is calculated. Then, a gradient descent algorithm (such as stochastic gradient descent SGD or its variant Adam) is used to update the model parameters in the opposite direction of the gradient, gradually reducing the loss value. During training, a fixed number of iterations (e.g., 100,000) or an early stopping mechanism (stopping training when the loss does not decrease after 10 consecutive iterations) is set to avoid overfitting. At each iteration, video segments are resampled from the training dataset, and steps two through five are repeated until the model converges (the loss value stabilizes within a small range). Finally, the pre-trained TimeSformer model can accurately extract the spatiotemporal features of the video, providing a foundation for subsequent video analysis tasks (such as shot segmentation and action recognition).

[0099] This embodiment achieves efficient pre-training of the TimeSformer spatiotemporal attention model, enabling the model to have a powerful ability to model spatiotemporal information of videos, and providing a reliable feature extraction tool for multimodal video analysis tasks.

[0100] In some embodiments, step S102 above, which involves using a reinforcement learning-driven adaptive keyframe extraction strategy based on a set of video segments to dynamically select keyframes according to content complexity and obtain a keyframe sequence representing the video content, specifically includes:

[0101] For each video segment in the video segment set, its content complexity features are extracted, and a state feature vector is constructed. The content complexity features include spatiotemporal complexity, texture complexity, and semantic richness.

[0102] The state feature vector is input into a pre-trained keyframe extraction decision model, and a keyframe selection action is output through forward propagation.

[0103] Action selection based on keyframes: Candidate keyframes are selected from the original video clips, and a pre-trained video reconstruction network is used to reconstruct the predicted complete video clip based on the candidate keyframes.

[0104] The composite reward signal is calculated based on the similarity between the predicted complete video segment and the original video segment, as well as the number of selected keyframes.

[0105] Using the composite reward signal, the parameter update gradient of the keyframe extraction decision model is calculated through the policy optimization algorithm. The internal parameters of the keyframe extraction decision model are adjusted according to the parameter update gradient until the model performance converges.

[0106] Based on the final decision output by the keyframe extraction decision model, the corresponding frames are selected from the original video clips and assembled into a keyframe sequence representing the video content.

[0107] In this embodiment, for each video segment in the video segment set, multi-dimensional content complexity features are first extracted. Specifically, spatiotemporal complexity is quantified by analyzing the magnitude of motion vector changes and the proportion of motion regions between adjacent frames in the video segment, reflecting the intensity of object movement in the video; texture complexity is measured by calculating the edge density and texture entropy value of the image within the frame, characterizing the richness of image details; semantic richness is evaluated by extracting frame-level semantic features using a pre-trained deep learning model (such as a convolutional neural network) and counting the number of significant semantic categories in the feature vector. After normalizing the features of the above three dimensions, they are concatenated into a state feature vector, which serves as the input state representation for the reinforcement learning model. For example, for a video segment with a duration of 5 seconds and a frame rate of 30fps, the spatiotemporal motion magnitude, texture entropy value, and number of semantic categories of its representative frames per second are extracted, ultimately constructing a 15-dimensional state feature vector.

[0108] The state feature vector is input into a pre-trained keyframe extraction decision model. This model employs a deep reinforcement learning architecture (such as a deep Q-network or a policy gradient network), and its training objective is to output the optimal keyframe selection action given the video content state. The action space is defined as the instructions for selecting specific frames as candidate keyframes from the current video segment, such as discrete actions like "select frame 10" or "skip the next 5 frames." The model calculates the Q-value or policy probability of each action through forward propagation and selects the action with the highest Q-value or the highest probability as the output. For example, given the input state feature vector, the model might output the action "select the current frame as the keyframe" or "skip the next 3 frames and then select the next frame."

[0109] The action is selected based on the output keyframes, and corresponding candidate keyframes are filtered from the original video clip. For example, if the action instruction is "select frame 5, frame 15, and frame 25", then these three frames are used as candidate keyframes. Subsequently, a pre-trained video reconstruction network (such as one based on a generative adversarial network or autoencoder architecture) is used as input to reconstruct the predicted complete video clip. During the reconstruction process, the network learns the spatiotemporal relationships between frames to fill in the missing content between candidate keyframes. For example, after inputting three candidate keyframes, the network can generate a predicted video clip containing 30 frames, with the same frame rate as the original video.

[0110] Based on the reconstructed predicted video clips and the original video clips, a composite reward signal is calculated to guide model training. The reward signal consists of two parts: a similarity reward, which quantifies the reconstruction quality by comparing the frame-by-frame structural similarity index (SSIM) or peak signal-to-noise ratio (PSNR) between the predicted and original clips; and a sparsity reward, which provides negative feedback based on the number of keyframes selected, encouraging the model to achieve higher-quality reconstruction with fewer keyframes. For example, if the predicted clip has an SSIM value of 0.95 and only 3 keyframes are used, the reward signal is 0.95 - 0.1 × 3 = 0.65 (where 0.1 is the sparsity penalty coefficient). The composite reward signal comprehensively reflects the accuracy and efficiency of keyframe selection.

[0111] Using a composite reward signal, the parameter update gradient of the keyframe extraction decision model is calculated through policy optimization algorithms (such as proximal policy optimization or deep deterministic policy gradient). Specifically, the update direction and magnitude of the weights in each layer of the model network are adjusted according to the reward value to maximize the long-term cumulative reward. For example, if a certain action selection results in a low reward value, the probability of that action being selected is reduced; conversely, it is increased. Through multiple rounds of iterative training, the model gradually learns the optimal keyframe selection strategy under different content complexity states until the model performance converges (e.g., the reward value fluctuation is less than 0.01 in 100 consecutive training rounds).

[0112] Based on the converged keyframe extraction decision model, a final decision is made for each video segment in the video clip set. The model outputs the optimal keyframe selection result for each segment, such as "segment 1 selects frames 8 and 20," and "segment 2 selects frames 3, 15, and 28." The keyframes of all segments are assembled in chronological order to form a keyframe sequence representing the entire video content. For example, for a video containing 10 segments, the final generated keyframe sequence may contain 25 frames, covering the main scenes and action changes in the video while maintaining the simplicity of the sequence.

[0113] This embodiment achieves the goal of dynamically and adaptively extracting keyframes based on video content. While ensuring the representativeness of video content, it significantly reduces the number of keyframes and is suitable for application scenarios such as video summary generation and fast browsing.

[0114] In some embodiments, step S103 above, which involves calling a pre-built video-text-action knowledge graph based on the keyframe sequence to generate a script description with causal logic, and obtaining a structured script and action description text, specifically includes:

[0115] Visual semantic encoding is performed on the keyframe sequence to generate a contextual visual feature vector that represents the overall visual content of the segment;

[0116] Based on contextual visual feature vectors, knowledge triples associated with the current visual content are retrieved from a pre-constructed video-text-action knowledge graph to form a related knowledge subgraph;

[0117] By fusing contextual visual feature vectors with textual representations of relevant knowledge subgraphs, a large language model is used to generate structured scripts and action description texts with causal logic.

[0118] In this embodiment, visual features are extracted frame by frame from the input keyframe sequence. A pre-trained visual coding model (such as the visual branch of ResNet, ViT, or CLIP) is used to convert each frame into a high-dimensional feature vector. Subsequently, a temporal attention mechanism is used to aggregate the feature vectors of all frames to generate a visual feature vector representing the contextual information of the entire keyframe sequence. Specifically, the temporal attention mechanism dynamically adjusts the contribution of each frame to the overall features by calculating the association weights between the features of each frame and other frames. For example, if the keyframe sequence contains the action of "a person opens a door - enters a room - sits down", the temporal attention mechanism will give higher weights to the "opening the door" and "sitting down" frames because they are the turning points of the action, while the intermediate frames of the "entering the room" process will have lower weights. The final output contextual visual feature vector has a dimension of 512 and includes the spatial layout of the video clips, object relationships, and action temporal information.

[0119] The generated contextual visual feature vectors are input into the retrieval module of a pre-built video-text-action knowledge graph. This knowledge graph stores video scenes, objects, actions, and their causal relationships (e.g., "person holding a key → opening the door → entering the room") in the form of triples, covering common scenarios such as daily activities, sports, and social interactions. The retrieval module first uses feature similarity matching (e.g., cosine similarity) to filter out the nodes most relevant to the contextual visual features (e.g., "opening the door," "key," "room") from the graph. Then, using these nodes as centers, it expands the retrieval of edges and nodes directly associated with them (e.g., "holding a key → opening the door," "opening the door → entering the room"), forming a knowledge subgraph containing causal logic. For example, if the contextual features point to "indoor scene + person movement," the retrieval module may return the subgraph "person walks towards the door → grasps the doorknob → pushes the door → enters the room," where each node corresponds to a specific action or object, and each edge is labeled with a causal relationship (e.g., "grasping the doorknob" is a prerequisite for "pushing the door").

[0120] The retrieved knowledge subgraphs are converted into textual representations. Specifically, each triple in the subgraph (e.g., "person-holding key-opening door") is described in natural language to generate the short sentence "the person holds the key and opens the door"; simultaneously, the topological structure information of the subgraph (e.g., the order of actions) is preserved. Then, the textual representation is fused with the contextual visual feature vector from step one. The fusion method employs a cross-modal attention mechanism: using the textual representation as the query and the visual feature vector as the key and value, the association weights between the text and visual features are calculated to generate the fused multimodal feature vector. For example, if the textual description "the person sits down" has a high activation value in the visual feature "sofa area," the fused feature will strengthen the association between "sitting down - sofa" and suppress irrelevant information.

[0121] The fused multimodal feature vectors are input into a pre-trained large language model (such as the GPT series or LLaMA), which outputs a structured script and action description text through conditional generation. During the generation process, the model uses the causal logic of the knowledge subgraph as a constraint to ensure the temporal rationality of the script. Specifically, the model first generates a scene title for the script based on the fused features (e.g., "Office Scene: Character Enters"); then, it generates a description sentence by sentence according to the action sequence of the knowledge subgraph (e.g., "1. The character walks towards the door with a document in hand; 2. Turns the doorknob and pushes it open; 3. After entering the room, walks towards the desk"); finally, it adds action description text (e.g., "When pushing the door, one should turn sideways to avoid collision"). The generated text is optimized through beam search, selecting the sequence with the highest probability and logical coherence. For example, if the knowledge subgraph contains "take a cup → pour water → drink water", the model will avoid generating the incorrect sequence "pour water and then take a cup".

[0122] The generated script text undergoes post-processing to extract key information and organize it into a structured format. Post-processing includes: action segmentation (dividing the script into segments based on timestamps or scene transitions), character labeling (identifying the actors performing the actions and tagging their names), and expliciting causal relationships (labeling action logic with conjunctions such as "therefore" and "subsequently"). The final output includes a structured script (divided by scene, character, and action) and action description text (containing causal logic and operational details), suitable for film and television production, game narratives, or educational demonstrations.

[0123] This embodiment achieves efficient generation from keyframe sequences to logical scripts, solving the problems of poor script logic and vague action descriptions in traditional methods, and significantly improving the quality and practicality of automated narrative.

[0124] In some embodiments, step S104 above, which involves combining structured scripts and action description texts with a preset user preference model to drive the image diffusion model to generate personalized image sequences that match the original video actions and user style preferences, specifically includes:

[0125] The structured script and action description text are encoded into text conditional embeddings. Based on the text conditional embeddings and a preset user preference model, the image diffusion model is driven to generate conditional images.

[0126] In the conditional image generation process, semantic content is guided by a cross-attention mechanism based on text conditional embedding, and artistic style is guided by feature modulation based on a user preference model to obtain the initial generated image;

[0127] The motion control signals of the original video are encoded as conditions with the same latent variable dimension as the initially generated image and injected into the UNet network of the image diffusion model;

[0128] By aligning the encoded motion control signals with the initial generated images in the latent space, a personalized image sequence that matches the original video motion and the user's style preferences is generated.

[0129] In this embodiment, the structured script (including scenes, characters, and action sequences) and action description text (including causal logic and operational details) are jointly encoded to generate a text conditional embedding vector. Specifically, a pre-trained text encoding model (such as the CLIP text encoder or the T5 model) is used to extract semantic features from each action paragraph (e.g., "the character pushes open the door and enters the room") and action description (e.g., "when pushing the door, one should turn sideways to avoid collision") in the script. Temporal pooling is then used to aggregate all action features within the same scene into a single text conditional vector. For example, if the script contains three actions: "office scene: character opens the door → walks to the sofa → sits down," the text encoding model will generate a 512-dimensional conditional vector containing semantic information such as "direction of the door opening action," "sofa positional relationship," and "sitting posture." Simultaneously, to enhance the expression of action details, explicit embedding of action labels (e.g., "push," "walk," "sit") is introduced during the encoding process, allowing them to jointly constitute the final text conditional embedding with the natural language description.

[0130] Based on a pre-defined user preference model, the model extracts users' artistic style preference features and converts them into style control vectors that can be used for image diffusion model modulation. The user preference model analyzes users' historical interaction data (such as clicks and favorites on images of different styles) or directly input style descriptions (such as "cyberpunk style" or "ink painting style"), employing style classification networks (such as ResNet variants) or style descriptor encoders (such as CLIP-based text-image alignment models) to generate style feature vectors representing user preferences. For example, if a user prefers "retro oil painting style," the model extracts features such as "rough brushstrokes," "low color saturation," and "brownish-toned shadows," converting them into a 128-dimensional style modulation vector that matches the latent space dimension of the image diffusion model. During image generation, this vector is injected into the UNet network of the diffusion model through a feature modulation layer (such as AdaIN or FiLM) to adjust the mean and variance of intermediate features, ensuring the generated image reflects the user's desired artistic style.

[0131] The generated text conditional embedding vector and style modulation vector are input together into a pre-trained image diffusion model (such as Stable Diffusion or DALL·E 3) to drive conditional image generation. During generation, the text conditional embedding guides semantic content through a cross-attention mechanism: in the UNet network of the diffusion model, the text conditional vector acts as a query, performing attention calculations with the key and value of the current latent space features to strengthen image regions related to the text description (such as the hand and door frame region corresponding to the "pushing the door" action). Simultaneously, the style modulation vector dynamically adjusts the activation values ​​of each layer in the UNet network through the feature modulation layer, ensuring that the image's color, texture, and style are consistent with the user's preferred style. For example, if the text condition points to "a person running in the rain," and the style modulation is "dark watercolor style," the initially generated image will present a dark background, raindrops with a watercolor-like effect, and the outline of the running person.

[0132] Motion control signals (such as human keypoint trajectories and object speeds) are extracted from the original video and encoded into conditional vectors with the same dimension as the latent variables in the image diffusion model. Motion encoding employs a spatiotemporal convolutional network (such as SlowFast or I3D) to perform temporal analysis on human skeleton points and object bounding boxes in video frames, generating motion feature vectors describing the intensity and direction of the motion. For example, if the action of "a person pushing a door" in the original video involves the process of "arm extension → force application → door movement," the motion encoding network will generate a 64-dimensional motion conditional vector containing "arm movement direction" and "door displacement speed." Subsequently, through a latent space alignment module (such as a linear projection layer), this vector is expanded to the same dimension as the latent variables in the image diffusion model (e.g., 512 dimensions). At each denoising step in the diffusion process, this vector is added to the current latent space features, guiding the image to generate motion details that match the original video's motion.

[0133] By jointly optimizing text conditional embedding, user style modulation, and motion control signals, a personalized image sequence is generated that matches the original video actions and user style preferences. Specifically, during the iterative denoising process of the diffusion model, text conditional embedding ensures that the image content is consistent with the script description (e.g., "the character pushes the door" corresponds to the correct hand gesture), user style modulation makes the image present the desired artistic effect (e.g., the brushstrokes of a "retro oil painting"), and motion control signals constrain the motion trajectory of objects in the image (e.g., the displacement direction and speed of the "door"). For example, if the script description is "the character quickly pushes the door open and enters," the user preference is "anime style," and the door moves to the right in the original video, the generated image sequence will present: the first frame shows the character's right hand touching the door (anime style lines), the second frame shows the door opening 30% to the right (bright colors), and the third frame shows the character entering the room (the action speed is consistent with the video). The final output image sequence achieves personalized matching in semantic content, artistic style, and motion dynamics, and is suitable for scenarios such as animation production, game character design, or personalized content generation.

[0134] This embodiment solves the problem of content-style-action disconnect in traditional image generation methods, and realizes efficient transformation from structured scripts to personalized image sequences, significantly improving the controllability and practicality of the generated content.

[0135] In some embodiments, step S105 above, which involves using motion fidelity metrics to perform motion control and optimization based on personalized image sequences and motion information from the original video, to synthesize a target video with high visual consistency and narrative coherence, specifically includes:

[0136] Extract reference motion trajectories from the motion information of the original video, the reference motion trajectories including the main joint angle sequence and pixel-level motion vectors;

[0137] Personalized image sequences are input into the image-generated video model, and an initial output video sequence is generated through temporal processing.

[0138] Calculate the motion fidelity index between the initial output video sequence and the reference motion trajectory, and quantify the motion consistency error using the motion fidelity index;

[0139] Based on the quantized motion consistency error, motion control optimization is performed on the initial output video sequence, resulting in the first output video sequence.

[0140] Temporal filtering is applied to the first output video sequence and script-based transition effects are injected to synthesize a target video with high visual consistency and narrative coherence.

[0141] In this embodiment, subject motion information is extracted from the input raw video to construct a reference trajectory that includes joint angles and pixel-level motion. Specifically, a human pose estimation model (such as OpenPose or AlphaPose) is used to detect key points of the subject in the video frame by frame (such as joints like the shoulder, elbow, and wrist), generating a joint angle sequence and recording the rotation and displacement changes of each joint in three-dimensional space. Simultaneously, an optical flow estimation algorithm (such as Farneback or FlowNet) is used to calculate pixel-level motion vectors between adjacent frames, capturing subtle movements of the object's surface or background (such as clothing fluttering or hair swaying). For example, if the original video is a scene of "a person pushing a door," the pose estimation model will output the joint angle change of "the arm from hanging down to extending," and the optical flow algorithm will record the displacement direction and velocity of the pixels at the edge of the door frame. Finally, the joint angle sequence and pixel-level motion vectors are integrated into a reference motion trajectory, forming continuous motion constraint data on the time axis for subsequent optimization.

[0142] The generated personalized image sequence is input into a pre-trained graph-to-video model (such as AnimateDiff or Gen-2), which generates an initial output video sequence through a temporal attention mechanism. The graph-to-video model uses personalized images as keyframes and combines implicit temporal relationships between images (such as changes in object position and action continuity) to predict intermediate transition frames frame by frame. During generation, the model captures global temporal dependencies in the image sequence through a self-attention layer (such as the coordinated movement of the hand and the door in the "pushing the door" action) and fuses textual conditions (such as the script description "the character quickly pushes the door") and style conditions (such as the user's preference for "anime style") through a cross-attention layer, ensuring that the generated video is consistent with the input in content and style. For example, if the personalized image sequence contains three frames: "character touches the door - door slightly opens - character enters," the graph-to-video model will generate intermediate transition frames: "hand applies force - door opens 15% - door opens 30%," forming a continuous action sequence.

[0143] The motion fidelity index between the initial output video sequence and the extracted reference motion trajectory is calculated to quantify the motion consistency error. Specifically, the motion fidelity index consists of two parts: joint angle matching degree and pixel motion similarity. Joint angle matching degree is calculated by comparing the joint angle sequence of the main subject in the generated video with the joint angle sequence of the reference trajectory, and calculating the root mean square error (RMSE) of the angle deviation. Pixel motion similarity is calculated by comparing the optical flow field of the generated video with the pixel-level motion vector of the reference trajectory, and calculating the cosine similarity of the angle between the optical flow vectors. For example, if the joint angle of "arm extension" in the reference trajectory is 30 degrees, and the corresponding angle in the generated video is 25 degrees, then the joint angle matching degree error is 5 degrees; if the reference optical flow shows "door frame pixels move 10 pixels to the right", and the corresponding pixel in the generated video moves 8 pixels, then the pixel motion similarity is 0.8 (8 / 10). Finally, the two errors are weighted and summed to generate a comprehensive motion fidelity index (range 0-1, higher values ​​indicate stronger motion consistency).

[0144] Based on the quantified motion consistency error, motion control optimization is performed on the initial output video sequence to generate the first output video sequence. The optimization process employs gradient descent, using motion fidelity as the loss function, and backpropagation adjusts the intermediate features of the graph-generated video model. Specifically, in each frame of the generated video, the deviation between the current joint angle and the reference trajectory is calculated. By adjusting the weights of the corresponding temporal layers in the UNet network, the motion trajectories of key objects such as hands and doors are corrected. Simultaneously, based on the pixel motion similarity error, the parameters of the optical flow prediction module are optimized to make the motion of objects in the generated video (such as clothing movement) closer to the original video. For example, if the "door opening speed" in the initial video is 20% slower than the reference trajectory, the optimization process will enhance the activation value of the motion features in the "door region" in the corresponding frame, accelerating the door opening action. The optimized first output video sequence is highly consistent with the original video in both joint motion and pixel-level dynamics.

[0145] The generated first output video sequence is subjected to temporal filtering and script-based transition effects to synthesize the final target video. Temporal filtering employs Gaussian smoothing or Kalman filtering to eliminate inter-frame jitter (such as abrupt hand position changes) in the generated video, improving motion smoothness. For example, if the hand jumps by 5 pixels between frames 5 and 6 when a character pushes open a door in the first output video, Gaussian filtering corrects this jump to a smooth 2-pixel transition by weighted averaging of the hand positions in adjacent frames. Simultaneously, transition effects (such as fade-in / fade-out and zoom-in) are injected into the video based on scene transition descriptions in the structured script (such as "from the office doorway to the interior"). For example, when the script transitions from the "pushing open the door" scene to the "character sitting down" scene, a 1-second fade-out black screen is inserted between the two scenes, accompanied by a "door closing" sound effect, enhancing the continuity of the plot. The final output target video achieves high-quality standards in motion fidelity, visual smoothness, and plot logic, making it suitable for film and television special effects, game animation, or personalized content creation.

[0146] This embodiment realizes the automated synthesis from personalized image sequences to highly consistent target videos, solving the problems of motion distortion and visual jumps in traditional methods, and significantly improving the dynamic realism and narrative coherence of the generated content.

[0147] Reference Figure 2 An embodiment of the present invention provides a video content reproduction system 2 based on AI information perception, wherein the system 2 specifically includes:

[0148] The video segmentation module 201 is used to perform frame sequence analysis on the original video using a spatiotemporal attention model, and to perform multimodal fusion by combining audio features, and output a set of segmented video segments.

[0149] The frame extraction module 202 is used to dynamically select key frames based on the content complexity according to the video segment set and adopt a reinforcement learning-driven adaptive key frame extraction strategy to obtain a key frame sequence representing the video content.

[0150] The knowledge graph module 203 is used to generate a script description with causal logic based on the keyframe sequence by calling a pre-built video-text-action knowledge graph, and obtain a structured script and action description text.

[0151] The image diffusion module 204 is used to combine the structured script and action description text with the preset user preference model to drive the image diffusion model to generate a personalized image sequence that matches the original video action and user style preference.

[0152] The video optimization module 205 is used to perform motion control and optimization based on personalized image sequences and motion information of the original video, using motion fidelity indicators to synthesize target videos with high visual consistency and narrative coherence.

[0153] It is understandable that, such as Figure 1 The content shown in the embodiments of the video content reproduction method based on AI information perception is applicable to the embodiments of the video content reproduction system based on AI information perception. The specific functions implemented in the embodiments of the video content reproduction system based on AI information perception are the same as those shown in the figure. Figure 1 The illustrated method for video content reproduction based on AI information perception is the same as the one shown, and achieves the same beneficial effects. Figure 1 The beneficial effects achieved by the AI-based information perception-based video content reproduction method embodiment shown are also the same.

[0154] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0156] Reference Figure 3 The present invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, it implements the video content reproduction method based on AI information perception as described in any of the above methods.

[0157] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0158] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0160] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video content reproduction method based on AI information perception as described in any of the above methods.

[0161] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0162] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0163] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A video content reproduction method based on AI information perception, characterized in that, The method specifically includes: The original video is analyzed frame by frame using a spatiotemporal attention model, and multimodal fusion is performed by combining audio features to output a set of segmented video clips, specifically including: The original video is input into the pre-trained TimeSformer spatiotemporal attention model, and the spatiotemporal features of the video frame sequence are extracted through the spatiotemporal attention mechanism to obtain the video feature sequence. Based on the original video, audio features of its audio track are extracted synchronously, and the audio features are aligned with the video feature sequence by timestamps to obtain time-aligned multimodal feature pairs; All multimodal feature pairs are fused to obtain a multimodal feature sequence, and the probability distribution of shot switching points is predicted based on the multimodal feature sequence. Based on the probability distribution of shot switching points, the shot boundary position is determined by peak detection, and the original video is segmented into a set of video segments based on the shot boundary position. Based on a set of video clips, an adaptive keyframe extraction strategy driven by reinforcement learning is adopted to dynamically select keyframes according to the content complexity and obtain a keyframe sequence that represents the video content. Based on the keyframe sequence, a pre-built video-text-action knowledge graph is invoked to generate a script description with causal logic, thereby obtaining a structured script and action description text; By combining structured scripts and action description texts with a pre-set user preference model, the image diffusion model is driven to generate personalized image sequences that match the original video actions and user style preferences. Based on personalized image sequences and motion information from original videos, motion control and optimization are performed using motion fidelity metrics to synthesize target videos with high visual consistency and narrative coherence.

2. The method according to claim 1, characterized in that, The pre-training steps of the TimeSformer spatiotemporal attention model specifically include: A network architecture for constructing the TimeSformer spatiotemporal attention model is provided. The network architecture includes multiple spatiotemporal attention layers, and each spatiotemporal attention layer includes a spatial attention head for analyzing intra-frame spatial relationships and a temporal attention head for analyzing cross-frame temporal relationships. Randomly sample a video segment from the prepared training dataset and uniformly sample the video segment to obtain a fixed number of frame sequences; Each frame of the frame sequence is divided into multiple image blocks, and each image block is converted into an embedding vector through linear projection. Learnable positional and temporal codes are added to each embedding vector to form an input embedding sequence containing spatiotemporal information. The input embedded sequence is fed into the TimeSformer spatiotemporal attention model, and forward propagation is performed through the spatiotemporal attention layer to extract and fuse the spatiotemporal features of the video, and output the fused spatiotemporal features. Based on the fusion of spatiotemporal features, a video segment classification task is performed to generate prediction results. The loss value of the model training is calculated based on the prediction results and the preset real labels. Based on the loss value, the parameters of the TimeSformer spatiotemporal attention model are updated through backpropagation and gradient descent algorithms, and the model is iteratively trained until it converges.

3. The method according to claim 1, characterized in that, The method, based on a set of video clips, employs a reinforcement learning-driven adaptive keyframe extraction strategy to dynamically select keyframes according to content complexity, thereby obtaining a keyframe sequence representing the video content. Specifically, this includes: For each video segment in the video segment set, its content complexity features are extracted, and a state feature vector is constructed. The content complexity features include spatiotemporal complexity, texture complexity, and semantic richness. The state feature vector is input into a pre-trained keyframe extraction decision model, and a keyframe selection action is output through forward propagation. Action selection based on keyframes: Candidate keyframes are selected from the original video clips, and a pre-trained video reconstruction network is used to reconstruct the predicted complete video clip based on the candidate keyframes. The composite reward signal is calculated based on the similarity between the predicted complete video segment and the original video segment, as well as the number of selected keyframes. Using the composite reward signal, the parameter update gradient of the keyframe extraction decision model is calculated through the policy optimization algorithm. The internal parameters of the keyframe extraction decision model are adjusted according to the parameter update gradient until the model performance converges. Based on the final decision output by the keyframe extraction decision model, the corresponding frames are selected from the original video clips and assembled into a keyframe sequence representing the video content.

4. The method according to claim 1, characterized in that, The process involves using a pre-built video-text-action knowledge graph based on keyframe sequences to generate a script description with causal logic, thereby obtaining a structured script and action description text. Specifically, this includes: Visual semantic encoding is performed on the keyframe sequence to generate a contextual visual feature vector that represents the overall visual content of the segment; Based on contextual visual feature vectors, knowledge triples associated with the current visual content are retrieved from a pre-constructed video-text-action knowledge graph to form a related knowledge subgraph; By fusing contextual visual feature vectors with textual representations of relevant knowledge subgraphs, a large language model is used to generate structured scripts and action description texts with causal logic.

5. The method according to claim 1, characterized in that, The process of combining structured scripts and action description texts with a pre-defined user preference model to drive an image diffusion model to generate personalized image sequences that match the original video actions and user style preferences includes: The structured script and action description text are encoded into text conditional embeddings. Based on the text conditional embeddings and a preset user preference model, the image diffusion model is driven to generate conditional images. In the conditional image generation process, semantic content is guided by cross-attention mechanism based on text conditional embedding, and artistic style is guided by feature modulation based on user preference model to obtain the initial generated image; The motion control signals of the original video are encoded as conditions with the same latent variable dimension as the initially generated image and injected into the UNet network of the image diffusion model; By aligning the encoded motion control signals with the initial generated images in the latent space, a personalized image sequence that matches the original video motion and the user's style preferences is generated.

6. The method according to any one of claims 1 to 5, characterized in that, The process involves using motion information from personalized image sequences and original videos, employing motion fidelity metrics for motion control and optimization, to synthesize a target video with high visual consistency and narrative coherence. Specifically, this includes: Extract reference motion trajectories from the motion information of the original video, the reference motion trajectories including the main joint angle sequence and pixel-level motion vectors; Personalized image sequences are input into the image-generated video model, and an initial output video sequence is generated through temporal processing. Calculate the motion fidelity index between the initial output video sequence and the reference motion trajectory, and quantify the motion consistency error using the motion fidelity index; Based on the quantized motion consistency error, motion control optimization is performed on the initial output video sequence, resulting in the first output video sequence. Temporal filtering is applied to the first output video sequence and script-based transition effects are injected to synthesize a target video with high visual consistency and narrative coherence.

7. A video content reproduction system based on AI information perception, characterized in that, The system specifically includes: The video segmentation module uses a spatiotemporal attention model to perform frame sequence analysis on the original video, and combines audio features for multimodal fusion, outputting a set of segmented video clips, specifically including: The original video is input into the pre-trained TimeSformer spatiotemporal attention model, and the spatiotemporal features of the video frame sequence are extracted through the spatiotemporal attention mechanism to obtain the video feature sequence. Based on the original video, audio features of its audio track are extracted synchronously, and the audio features are aligned with the video feature sequence by timestamps to obtain time-aligned multimodal feature pairs; All multimodal feature pairs are fused to obtain a multimodal feature sequence, and the probability distribution of shot switching points is predicted based on the multimodal feature sequence. Based on the probability distribution of shot switching points, the shot boundary position is determined by peak detection, and the original video is segmented into a set of video segments based on the shot boundary position. The frame extraction module is used to extract keyframes based on a set of video clips using a reinforcement learning-driven adaptive keyframe extraction strategy. It dynamically selects keyframes according to the complexity of the content to obtain a keyframe sequence that represents the video content. The knowledge graph module is used to generate script descriptions with causal logic based on keyframe sequences by calling pre-built video-text-action knowledge graphs, and obtain structured scripts and action description texts. The image diffusion module is used to combine structured scripts and action description texts with a preset user preference model to drive the image diffusion model to generate personalized image sequences that match the original video actions and user style preferences. The video optimization module is used to perform motion control and optimization based on personalized image sequences and motion information from the original video, using motion fidelity metrics to synthesize target videos with high visual consistency and narrative coherence.

8. A computer device, characterized in that, include: The memory and processor, and the computer program stored in the memory, when the computer program is executed on the processor, implement the video content reproduction method based on AI information perception as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the video content reproduction method based on AI information perception as described in any one of claims 1 to 6.

Images