A multimodal video generation system based on speech temporal constraints and feedback calibration
By performing semantic analysis and duration prediction on the input text, combined with dynamic timeline calibration, the problem of audio-visual asynchrony in multimodal video generation was solved, achieving synchronous and stable generation of multimodal content.
Patent Information
- Application Number
- CN202611303694.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal content generation methods lack accurate modeling of audio playback duration, resulting in audio-visual asynchrony, unnatural rhythm, and the inability to correct timeline offset after generation, thus affecting playback quality.
The semantic processing layer identifies and analyzes the semantic boundaries of the input text, generates structured semantic segments, and predicts the duration of audio playback. The multimodal generation layer generates image, audio, and subtitle content, and the calibration synchronization layer constructs an error feedback model to perform dynamic timeline calibration, ultimately achieving synchronized binding of image, audio, and subtitle.
It achieves synchronized output of images, audio, and subtitles in multimodal video generation, reduces the accumulation of time errors, improves generation stability and automation, and ensures a natural and smooth audio-visual rhythm.
Smart Images

Figure CN123123679A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal video generation technology, and in particular to a multimodal video generation system based on speech temporal constraints and feedback calibration. Background Technology
[0002] With the development of AI-generated technologies, multimodal content generation methods based on text-to-image and audio generation are increasingly being applied in short video production, intelligent dubbing, and digital content production. However, current technologies typically employ fixed time allocation or simple mapping based on text length, lacking precise modeling of audio playback duration. This leads to issues such as audio-visual asynchrony and unnatural pacing in the generated content. Furthermore, most existing methods use a one-way generation process, failing to correct for time errors between audio and visual content after generation, easily resulting in timeline shifts and affecting the overall playback quality. Summary of the Invention
[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multimodal video generation system based on speech timing constraints and feedback calibration.
[0004] The objective of this invention is achieved through the following technical solution: a multimodal video generation system based on speech timing constraints and feedback calibration, comprising a semantic processing layer, a prediction segmentation layer, a multimodal generation layer, a calibration synchronization layer, and a synchronization binding module;
[0005] The semantic processing layer is used to identify semantic boundaries and perform semantic analysis on the input text, generating structured semantic segments;
[0006] The prediction segmentation layer generates image prompts based on semantic segmentation and establishes a speech duration prediction model to predict the speech playback duration of each semantic unit.
[0007] The multimodal generation layer is used to segment based on the prediction results and generate corresponding image, speech and caption content;
[0008] The calibration synchronization layer obtains the actual playback duration of the voice, constructs an error feedback model between the predicted duration and the actual duration, and adjusts the image display time through a dynamic time axis calibration mechanism;
[0009] The synchronization binding module synchronizes images, audio, and subtitles according to a unified timeline and generates the final multimodal video.
[0010] Preferably, the semantic processing layer includes a semantic boundary recognition module, a context analysis and content expansion module, and a speech duration prediction module;
[0011] The semantic boundary recognition module analyzes the overall semantics of the text through a large language model, identifies the boundary positions between different semantics, and generates semantic units for multimodal video generation.
[0012] The context analysis and content expansion module is used to perform contextual semantic analysis on the semantic units output by the semantic boundary recognition module, and expand the content by combining the semantic relationships before and after, extracting information on people, actions, scenes and visual styles, and generating structured image prompts for the image generation model.
[0013] The speech duration prediction module predicts the speech playback duration corresponding to each semantic unit based on the text content of the semantic unit, providing a time benchmark for subsequent sub-display unit division, image display time allocation, and unified timeline construction.
[0014] Preferably, the semantic boundary recognition module performs the following specific processing steps:
[0015] A1: Read the user's input text and process it by removing irrelevant characters, standardizing text format, and correcting punctuation.
[0016] A2: Construct a semantic boundary recognition prompt template, input the input text and semantic segmentation task together into the large language model, guide the model to comprehensively analyze from the perspectives of semantic integrity, contextual coherence, and visual expression requirements. The input process is represented as follows:
[0017] ;
[0018] in, Functions for constructing prompt templates, For input text, This describes the semantic segmentation task.
[0019] A3: The large language model performs semantic reasoning on the text based on the prompt template, generating multiple semantic units;
[0020] A4: Detect the contextual relationships between adjacent semantic units. When a semantic unit is found to be too short, incomplete in semantic expression, or strongly related to the preceding and following content, it is automatically merged. When a semantic unit contains multiple independent events or the corresponding speech is expected to be too long, it is further divided.
[0021] A5: Output semantic unit set ,in, For the first One semantic unit;
[0022] The specific processing steps of the context analysis and content expansion module are as follows:
[0023] B1: Read the semantic units output by the semantic boundary recognition module, analyze each semantic unit separately, and establish semantic relationships by combining the context information before and after the current semantic unit;
[0024] B2: Based on the semantic analysis results, visual information in the semantic units is extracted, and missing visual descriptions are supplemented by combining contextual information;
[0025] B3: After semantic supplementation, the extracted information is structured and organized to form prompt words that meet the input requirements of the image generation model. ,in, For entity information, For action information, For scene information, For visual style information;
[0026] B4: Optimize the consistency of prompt words generated from continuous semantic units;
[0027] B5: Output a structured image prompt sequence And establish a one-to-one correspondence between semantic units and image prompts;
[0028] The specific processing steps of the speech duration prediction module are as follows:
[0029] C1: Read the semantic units output by the semantic boundary recognition module, perform text feature analysis on each semantic unit, and extract relevant features that affect the duration of voice playback;
[0030] C2: Based on the extracted text features, a speech duration prediction model is built to predict the speech playback duration for each semantic unit.
[0031] ;
[0032] in, For the first Predicted speech duration for each semantic unit, For text length, For semantic complexity, As a pause factor, , , All are weighting coefficients. For bias terms;
[0033] C3: For semantic units containing long sentences, parallel structures, or continuous descriptions, the system increases the prediction time accordingly; for semantic units with short content or simple expression, the prediction time is shortened accordingly.
[0034] C4: After completing the prediction of all semantic units, generate the predicted speech duration sequence: ;
[0035] Calculate the total predicted speech duration for the entire text.
[0036] .
[0037] Preferably, the prediction segmentation layer includes a sub-display unit segmentation module, an image prompt word generation module, and a speech / audio generation module;
[0038] The sub-display unit division module is used to perform secondary division of semantic units based on the predicted duration output by the speech duration prediction module;
[0039] The image prompt generation module is used to generate corresponding image prompts based on the sub-display units and convert text semantics into descriptive information that the image generation model can recognize;
[0040] The speech and audio generation module is used to generate corresponding speech based on the text of the sub-display unit and to establish a unified speech resource management mechanism.
[0041] Preferably, the specific processing steps for dividing the sub-display unit into modules are as follows:
[0042] D1: Read the predicted speech duration sequence output by the speech duration prediction module, judge the predicted duration corresponding to each semantic unit, and when the predicted speech duration corresponding to a certain semantic unit exceeds the preset threshold, it is considered that the semantic unit contains a large amount of information and needs to be divided into multiple sub-display units; when the predicted duration does not exceed the threshold, the original semantic unit remains unchanged.
[0043] D2: The semantic units that need to be divided are split according to the principle of semantic integrity;
[0044] D3: The number of sub-display units is determined based on the predicted speech duration.
[0045] ;
[0046] in, The number of sub-display units after division. For the first Predicted speech duration for each semantic unit, The preset target duration for a single display unit;
[0047] D4: After the division is completed, each sub-display unit is renumbered and inherits the context relationship of its semantic unit to form a new sequence of sub-display units. ;
[0048] The specific processing steps of the image prompt generation module are as follows:
[0049] E1: Read the sequence of sub-display units, extract visual elements, and generate corresponding image prompts according to the input format of the image generation model;
[0050] E2: For continuous sub-display units, the consistency of character characteristics, scene environment and overall style is maintained in combination with the context. When the semantics change, the corresponding visual description is automatically adjusted.
[0051] E3: Establish a one-to-one correspondence between sub-display units and image prompts, forming a sequence of image prompts:
[0052] ;
[0053] in, For the first In the semantic unit, the first Image prompts corresponding to each sub-display unit;
[0054] The specific processing steps of the speech and audio generation module are as follows:
[0055] F1: Read the text of each sub-display unit and call the speech synthesis model to generate the corresponding speech;
[0056] F2: Unify the audio encoding and establish the correspondence between voice numbers and sub-display units;
[0057] F3: Writes voice resource information into the database and creates a resource index.
[0058] Preferably, the multimodal generation layer includes an image sequence generation module, a speech sequence generation module, and a duration information acquisition module;
[0059] The image sequence generation module is used to generate corresponding images based on image prompts and organize them into an image sequence according to the order of the sub-display units;
[0060] The speech sequence generation module is used to organize multiple speech segments output by the speech audio generation module into a continuous speech sequence according to the order of the sub-display units, and to establish the correspondence between the speech segments and the sub-display units;
[0061] The duration information acquisition module is used to calculate the actual playback duration of each audio segment and serve as the basis for dynamic timeline calibration.
[0062] Preferably, the calibration synchronization layer includes a dynamic timeline calibration module, an image display time adjustment module, and a unified timeline generation module;
[0063] The dynamic time axis calibration module is used to dynamically correct the image display time based on the error between the predicted speech duration and the actual speech duration.
[0064] The image display time adjustment module adaptively corrects the display time of each image based on the error results output by the dynamic time axis calibration module.
[0065] The unified timeline generation module is used to construct a unified multimodal timeline based on image display time, audio playback time, and subtitle information.
[0066] Preferably, the specific processing steps of the dynamic time axis calibration module are as follows:
[0067] S1: Read the predicted speech duration and the actual speech playback duration output by the speech duration prediction module, and calculate the time error corresponding to each sub-display unit.
[0068] ;
[0069] in, For the first Each sub-display unit corresponds to the actual playback duration of the audio. To predict speech duration, This is due to time error;
[0070] S2: The image display time is dynamically updated through an error feedback mechanism.
[0071] ;
[0072] in, For the first During the nth iteration The image display time corresponding to each sub-display unit For error response coefficients, For local smoothing coefficients, This is the global compensation coefficient;
[0073] Introduce the error convergence criterion:
[0074] ;
[0075] in, A preset error threshold is set. When the time error corresponding to all sub-display units is less than the threshold, the current timeline is considered to have been calibrated and no further time adjustment is performed.
[0076] S3: After completing the time calibration of all sub-display units, the system outputs the calibrated image display time series and retains the corresponding error data.
[0077] Preferably, the adjustment relationship of the image display time adjustment module is as follows:
[0078] ;
[0079] in, For the adjusted image display time, To predict speech duration, This refers to the time error after dynamic time axis calibration.
[0080] The present invention has the following advantages:
[0081] 1. This invention performs semantic analysis and segmentation on the input text to generate structured semantic information. Then, it generates image prompts based on the semantic segments and predicts the playback duration of each segment's audio. Subsequently, it generates corresponding image, audio, and subtitle content based on the predicted duration. After the content generation is completed, it constructs a duration error model based on the actual audio duration and dynamically calibrates the image display time. Finally, it constructs a unified multimodal timeline to achieve synchronous output of images, audio, and subtitles, thus completing the automatic generation of multimodal video.
[0082] 2. This invention introduces time constraints in advance through a speech duration prediction model to achieve reasonable planning of visual content. At the same time, it adopts a dynamic calibration method that combines predicted duration with actual duration to adaptively adjust the multimodal time axis, effectively reducing the accumulation of time errors in the long sequence generation process and improving the stability and automation of multimodal video generation. Attached Figure Description
[0083] Figure 1 This is a schematic diagram of a multimodal video generation system. Detailed Implementation
[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0085] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0086] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.
[0087] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0088] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0089] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0090] In this embodiment, as Figure 1 As shown, a multimodal video generation system based on speech temporal constraints and feedback calibration includes a semantic processing layer, a prediction segmentation layer, a multimodal generation layer, a calibration synchronization layer, and a synchronization binding module.
[0091] The semantic processing layer is used to identify semantic boundaries and perform semantic analysis on the input text, generating structured semantic segments;
[0092] The prediction segmentation layer generates image prompts based on semantic segmentation and establishes a speech duration prediction model to predict the speech playback duration of each semantic unit.
[0093] The multimodal generation layer is used to segment based on the prediction results and generate corresponding image, speech and caption content;
[0094] The calibration synchronization layer obtains the actual playback duration of the voice, constructs an error feedback model between the predicted duration and the actual duration, and adjusts the image display time through a dynamic time axis calibration mechanism;
[0095] The synchronization binding module synchronizes images, audio, and subtitles according to a unified timeline and generates the final multimodal video. This invention performs semantic analysis and segmentation on the input text to generate structured semantic information. Then, based on the semantic segmentation, it generates image prompts and predicts the audio playback duration for each segment. Subsequently, it generates corresponding image, audio, and subtitle content based on the predicted duration. After content generation, it constructs a duration error model using the actual audio duration and dynamically calibrates the image display time. Finally, it constructs a unified multimodal timeline to achieve synchronized output of images, audio, and subtitles, thus completing the automatic generation of multimodal video.
[0096] Furthermore, the semantic processing layer includes a semantic boundary recognition module, a context analysis and content expansion module, and a speech duration prediction module;
[0097] The semantic boundary recognition module analyzes the overall semantics of the text through a large language model, identifies the boundary positions between different semantics, and generates semantic units for multimodal video generation, providing basic data for subsequent context analysis, speech duration prediction and image generation.
[0098] The context analysis and content expansion module performs contextual semantic analysis on the semantic units output by the semantic boundary recognition module, and expands the content by combining the semantic relationships before and after, extracting information on people, actions, scenes and visual styles, and generating structured image prompts for the image generation model; thereby improving the consistency between the subsequent image generation results and the text semantics.
[0099] The speech duration prediction module predicts the speech playback duration corresponding to each semantic unit based on the text content of the semantic unit, providing a time reference for subsequent sub-display unit division, image display time allocation, and unified timeline construction. Furthermore, the semantic boundary recognition module's specific processing steps are as follows:
[0100] A1: Read the user input text and process it by removing irrelevant characters, standardizing text format and punctuation, to ensure data consistency in the subsequent semantic analysis process;
[0101] A2: Construct a semantic boundary recognition prompt template, input the input text and semantic segmentation task together into the large language model, guide the model to conduct a comprehensive analysis based on semantic integrity, contextual coherence, and visual expression needs, rather than segmenting solely based on punctuation marks or a fixed number of characters. The input process is represented as follows:
[0102] ;
[0103] in, Functions for constructing prompt templates, For input text, This describes the semantic segmentation task.
[0104] A3: The large language model performs semantic reasoning on the text based on the prompt template, generating multiple semantic units to ensure that each semantic unit can independently express a complete scene or action, avoiding semantic fragmentation or lack of context.
[0105] A4: Detect the contextual relationship between adjacent semantic units. When a semantic unit is found to be too short, incomplete in semantic expression, or strongly related to the preceding and following content, it is automatically merged. When a semantic unit contains multiple independent events or the corresponding speech is expected to be too long, it is further divided to ensure that each semantic unit has good semantic integrity and visual expression ability.
[0106] A5: Output semantic unit set ,in, For the first Each semantic unit serves as input to the subsequent contextual analysis and content expansion module, as well as the speech duration prediction module, providing a unified semantic foundation for the entire multimodal video generation process.
[0107] The specific processing steps of the context analysis and content expansion module are as follows:
[0108] B1: Read the semantic units output by the semantic boundary recognition module, analyze each semantic unit separately, and combine the context information before and after the current semantic unit to establish semantic relationships, so that the current semantic unit can not only express its own content, but also maintain the continuity with the preceding and following content, avoiding obvious jumps or sudden changes in the video.
[0109] B2: Based on the semantic analysis results, visual information is extracted from the semantic units, and missing visual descriptions are supplemented by contextual information. Specifically, visual information includes people, objects, actions, scenes, environments, time, emotions, etc., and missing visual descriptions are supplemented by contextual information. For example, when the semantic unit does not clearly describe the direction of the person, the ambient lighting, or the weather information, it can be reasonably supplemented based on the contextual semantics to make the visual description more complete.
[0110] B3: After semantic supplementation, the extracted information is structured and organized to form prompt words that meet the input requirements of the image generation model. ,in, For entity information, For action information, For scene information, Visual style information; specifically, the prompts mainly include entity information, action information, scene information, and visual style information.
[0111] B4: To ensure good continuity between adjacent images, the prompts generated by consecutive semantic units are optimized for consistency; when adjacent semantic units belong to the same scene or the same event, the appearance of the characters, the scene environment and the overall style of the picture are kept consistent; when the semantics change, the corresponding visual elements are automatically adjusted according to the new semantic content to make the video content transition naturally.
[0112] B5: Output a structured image prompt sequence Furthermore, a one-to-one correspondence is established between semantic units and image prompts. Specifically, the generated prompt sequence serves as the basic input for subsequent image prompt generation modules and image sequence generation modules, providing unified, complete, and context-continuous visual descriptive information for image content generation.
[0113] The specific processing steps of the speech duration prediction module are as follows:
[0114] C1: Reads the semantic units output by the semantic boundary recognition module, performs text feature analysis on each semantic unit, and extracts relevant features that affect the duration of speech playback. Specifically, the relevant features include text length, semantic complexity, and pause information. Among them, text length reflects the basic playback time of the speech, semantic complexity describes the sentence structure and expression difficulty, and pause information reflects the natural pauses during the speech reading process.
[0115] C2: Based on the extracted text features, a speech duration prediction model is built to predict the speech playback duration for each semantic unit.
[0116] ;
[0117] in, For the first Predicted speech duration for each semantic unit, For text length, For semantic complexity, As a pause factor, , , All are weighting coefficients. For bias terms;
[0118] C3: To improve the stability of prediction results, the system increases the prediction time for semantic units containing long sentences, parallel structures, or continuous descriptions; and shortens the prediction time for semantic units with short content or simple expression, so that the prediction results are more consistent with the actual speech playback rules.
[0119] C4: After completing the prediction of all semantic units, generate the predicted speech duration sequence: ;
[0120] Calculate the total predicted speech duration for the entire text.
[0121] .
[0122] Specifically, the prediction results serve as the initial time reference for subsequent sub-display unit division modules and dynamic time axis calibration modules, providing a reference for the time allocation of visual content and multimodal synchronization.
[0123] In this embodiment, the prediction segmentation layer includes a sub-display unit segmentation module, an image prompt word generation module, and a speech / audio generation module;
[0124] The sub-display unit division module is used to perform secondary division of semantic units based on the predicted duration output by the speech duration prediction module, so that each sub-display unit corresponds to a more balanced speech playback time, thereby improving the time matching accuracy between visual content and speech content.
[0125] The image prompt word generation module is used to generate corresponding image prompt words according to the sub-display unit, and convert the text semantics into descriptive information that the image generation model can recognize;
[0126] The speech / audio generation module is used to generate corresponding speech based on the text of the sub-display units and establish a unified speech resource management mechanism. Further, the specific processing steps of the sub-display unit division module are as follows:
[0127] D1: Read the predicted speech duration sequence output by the speech duration prediction module, judge the predicted duration corresponding to each semantic unit, and when the predicted speech duration corresponding to a certain semantic unit exceeds the preset threshold, it is considered that the semantic unit contains a large amount of information and needs to be divided into multiple sub-display units; when the predicted duration does not exceed the threshold, the original semantic unit remains unchanged.
[0128] D2: The semantic units that need to be divided are split according to the principle of semantic integrity; specifically, the division is prioritized at sentence pauses, phrase boundaries or action transitions to avoid destroying the original semantic structure and ensure that each sub-display unit can still express relatively complete content.
[0129] D3: The number of sub-display units is determined based on the predicted speech duration.
[0130] ;
[0131] in, The number of sub-display units after division. For the first Predicted speech duration for each semantic unit, The preset target duration for a single display unit;
[0132] D4: After the division is completed, each sub-display unit is renumbered and inherits the context relationship of its semantic unit to form a new sequence of sub-display units. Specifically, the final output sub-display unit sequence serves as input for the subsequent image prompt word generation module and speech audio generation module, enabling the subsequently generated images, speech, and subtitles to be organized according to a more reasonable time granularity.
[0133] The specific processing steps of the image prompt generation module are as follows:
[0134] E1: Read the sub-display unit sequence, extract visual elements, and generate corresponding image prompts according to the input format of the image generation model; specifically, visual elements include people, objects, actions, scenes, and styles.
[0135] E2: For continuous sub-display units, the consistency of character characteristics, scene environment and overall style is maintained in combination with the context. When the semantics change, the corresponding visual description is automatically adjusted so that the generated content accurately expresses the semantics of the text.
[0136] E3: Establish a one-to-one correspondence between sub-display units and image prompts, forming a sequence of image prompts:
[0137] ;
[0138] in, For the first In the semantic unit, the first The image prompts corresponding to each sub-display unit are used as input to the image sequence generation module.
[0139] The specific processing steps of the speech and audio generation module are as follows:
[0140] F1: Reads the text of each sub-display unit, calls the speech synthesis model to generate the corresponding speech, and can set parameters such as speaker, speech rate, tone and emotion as needed;
[0141] F2: Unify the audio encoding and establish the correspondence between voice numbers and sub-display units;
[0142] F3: Writes the speech resource information into the database and establishes a resource index, providing speech data for subsequent speech sequence organization, duration statistics, and timeline calibration. Specifically, the speech resource information is shown in Table 1.
[0143] Table 1
[0144] Field Name Data types Information Description Remark audio_id varchar(64) Voice Number Unique Identifier unit_id int Sub-display unit number Corresponding text content audio_path varchar(255) audio file path Audio storage location speaker varchar(64) speaker Optional parameters create_time datetime Generation time Resource Management and Use
[0145] In this embodiment, the multimodal generation layer includes an image sequence generation module, a speech sequence generation module, and a duration information acquisition module;
[0146] The image sequence generation module is used to generate corresponding images based on image prompts and organize them into an image sequence according to the order of the sub-display units;
[0147] The speech sequence generation module is used to organize multiple speech segments output by the speech audio generation module into a continuous speech sequence according to the order of the sub-display units, and to establish the correspondence between the speech segments and the sub-display units;
[0148] The duration information acquisition module is used to calculate the actual playback duration of each audio segment and serve as the basis for dynamic timeline calibration. Specifically, the image sequence generation module works as follows: the system reads image prompts, calls the image generation model to generate corresponding images, and completes the numbering and sorting according to the order of the sub-display units, ensuring that the image content is consistent with the text semantics, forming a continuous visual sequence. The system also establishes image resource management information, the basic fields of which are shown in Table 2.
[0149] Table 2
[0150] Field Name Data types Information Description Remark image_id varchar(64) Image number Unique Identifier unit_id int Sub-display unit number Corresponding text content prompt text Image prompts Image generation basis image_path varchar(255) Image file path Image storage location create_time datetime Generation time Resource Management and Use
[0151] Once the image is generated, the resource information is written to the database, and an image resource index is created to provide visual resources for subsequent timeline construction and multimodal synchronization.
[0152] The speech sequence generation module works as follows: the system arranges each speech segment in the order of the text, and can maintain multiple independent audio formats according to actual application needs, or it can splice them to generate a complete speech file, and finally output a unified speech sequence to provide speech data for subsequent acquisition of duration information and dynamic timeline calibration.
[0153] The specific content of the duration information acquisition module is as follows: Let the first... The actual playback duration of the audio corresponding to each sub-display unit is: The calculation method is as follows:
[0154] ;
[0155] in, and These represent the start and end times of the audio segments, respectively. After the system generates the audio, it automatically calculates the actual playback duration of each segment and writes the results into a duration information table, as shown in Table 3.
[0156] Table 3
[0157] Field Name Data types Information Description Remark unit_id int Sub-display unit number Unique Identifier audio_id varchar(64) Voice Number Corresponding audio resources duration float Actual playback duration (s) Time statistics use update_time datetime Update time Data maintenance and use
[0158] In this embodiment, the calibration synchronization layer includes a dynamic timeline calibration module, an image display time adjustment module, and a unified timeline generation module;
[0159] The dynamic time axis calibration module is used to dynamically correct the image display time based on the error between the predicted speech duration and the actual speech duration, and to build a time axis calibration model based on a feedback mechanism, so that the image playback rhythm can adaptively match the actual speech playback process, thereby improving the audio-visual synchronization accuracy of multimodal content.
[0160] The image display time adjustment module adaptively corrects the display time of each image based on the error results output by the dynamic time axis calibration module.
[0161] The unified timeline generation module is used to construct a unified multimodal timeline based on image display time, audio playback time, and subtitle information, enabling unified management of various resources across the time dimension. Furthermore, the dynamic timeline calibration module's specific processing steps are as follows:
[0162] S1: Read the predicted speech duration and the actual speech playback duration output by the speech duration prediction module, and calculate the time error corresponding to each sub-display unit.
[0163] ;
[0164] in, For the first Each sub-display unit corresponds to the actual playback duration of the audio. To predict speech duration, For time error; when When the actual playback time is longer than the predicted time, the image display time needs to be extended; when If necessary, the image display time should be shortened appropriately;
[0165] S2: To avoid fluctuations in the timeline caused by adjusting solely based on the current error, the image display time is dynamically updated by comprehensively considering the current error, changes in adjacent errors, and cumulative error. This is achieved through an error feedback mechanism.
[0166] ;
[0167] in, For the first During the nth iteration The image display time corresponding to each sub-display unit For error response coefficients, For local smoothing coefficients, This is the global compensation coefficient; specifically, the current error term is used to quickly respond to actual playback changes; the adjacent error difference term is used to reduce local timeline fluctuations; and the cumulative error term is used to correct the overall offset generated during long sequence playback, making the timeline more stable.
[0168] To ensure that the time axis gradually stabilizes, an error convergence criterion is introduced:
[0169] ;
[0170] in, A preset error threshold is set. When the time error corresponding to all sub-display units is less than the threshold, the current timeline is considered to have been calibrated and no further time adjustment is performed.
[0171] S3: After completing the time calibration of all sub-display units, the system outputs the calibrated image display time series and retains the corresponding error data, providing a time basis for subsequent image display time adjustment and unified timeline generation. Specifically, through the above dynamic feedback calibration process, the image display time can be adaptively adjusted according to the actual voice playback situation, effectively reducing the time offset caused by prediction errors, avoiding audio-visual desynchronization during long-term playback, and improving the overall playback effect of multimodal video generation.
[0172] Furthermore, the adjustment relationship of the image display time adjustment module is as follows:
[0173] ;
[0174] in, For the adjusted image display time, To predict speech duration, To compensate for time errors after dynamic timeline calibration, the system updates image playback parameters based on the adjusted display time and transmits the results to the unified timeline generation module. The unified timeline information is stored in the timeline information table, as shown in Table 4.
[0175] Table 4
[0176] Field Name Data types Information Description Remark timeline_id varchar(64) Timeline number Unique Identifier unit_id int Sub-display unit number Data association image_id varchar(64) Image number Image resources audio_id varchar(64) Voice Number Audio resources subtitle text Subtitles Text information start_time float Start time Playback start time end_time float End time Playback End Time
[0177] The system establishes a unified timeline based on the time information of each sub-display unit and completes the time mapping of image, audio and subtitle resources, providing unified time control for multimodal video synthesis.
[0178] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A multimodal video generation system based on speech temporal constraints and feedback calibration, characterized in that: It includes a semantic processing layer, a prediction and segmentation layer, a multimodal generation layer, a calibration and synchronization layer, and a synchronization binding module; The semantic processing layer is used to perform semantic boundary recognition and semantic analysis on the input text, and generate structured semantic segments; Therefore, the prediction segmentation layer generates image prompts based on semantic segmentation and establishes a speech duration prediction model to predict the speech playback duration of each semantic unit. The multimodal generation layer is used to divide the data according to the prediction results and generate corresponding image, audio, and subtitle content. The calibration synchronization layer obtains the actual playback duration of the voice, constructs an error feedback model between the predicted duration and the actual duration, and adjusts the image display time through a dynamic time axis calibration mechanism. The synchronization binding module synchronizes and binds images, audio, and subtitles according to a unified timeline, and generates the final multimodal video.
2. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 1, characterized in that: The semantic processing layer includes a semantic boundary recognition module, a context analysis and content expansion module, and a speech duration prediction module. The semantic boundary recognition module analyzes the overall semantics of the text using a large language model, identifies the boundary positions between different semantics, and generates semantic units for multimodal video generation. The context analysis and content expansion module is used to perform contextual semantic analysis on the semantic units output by the semantic boundary recognition module, and expand the content by combining the semantic relationships before and after, extracting information on people, actions, scenes and visual styles, and generating structured image prompts for the image generation model. The speech duration prediction module predicts the speech playback duration corresponding to each semantic unit based on the text content of the semantic unit, providing a time reference for subsequent sub-display unit division, image display time allocation, and unified timeline construction.
3. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 2, characterized in that: The specific processing steps of the semantic boundary recognition module are as follows: A1: Read the user's input text and process it by removing irrelevant characters, standardizing text format, and correcting punctuation. A2: Construct a semantic boundary recognition prompt template, input the input text and semantic segmentation task together into the large language model, guide the model to comprehensively analyze from the perspectives of semantic integrity, contextual coherence, and visual expression requirements. The input process is represented as follows: ; in, Functions for constructing prompt templates, For input text, This describes the semantic segmentation task. A3: The large language model performs semantic reasoning on the text based on the prompt template, generating multiple semantic units; A4: Detect the contextual relationships between adjacent semantic units. When a semantic unit is found to be too short, incomplete in semantic expression, or strongly related to the preceding and following content, it is automatically merged. When a semantic unit contains multiple independent events or the corresponding speech is expected to be too long, it is further divided. A5: Output semantic unit set ,in, For the first One semantic unit; The specific processing steps of the context analysis and content expansion module are as follows: B1: Read the semantic units output by the semantic boundary recognition module, analyze each semantic unit separately, and establish semantic relationships by combining the context information before and after the current semantic unit; B2: Based on the semantic analysis results, visual information in the semantic units is extracted, and missing visual descriptions are supplemented by combining contextual information; B3: After semantic supplementation, the extracted information is structured and organized to form prompt words that meet the input requirements of the image generation model. ,in, For entity information, For action information, For scene information, For visual style information; B4: Optimize the consistency of prompt words generated from continuous semantic units; B5: Output a structured image prompt sequence And establish a one-to-one correspondence between semantic units and image prompts; The specific processing steps of the speech duration prediction module are as follows: C1: Read the semantic units output by the semantic boundary recognition module, perform text feature analysis on each semantic unit, and extract relevant features that affect the duration of voice playback; C2: Based on the extracted text features, a speech duration prediction model is built to predict the speech playback duration for each semantic unit. ; in, For the first Predicted speech duration for each semantic unit, For text length, For semantic complexity, As a pause factor, , , All are weighting coefficients. For bias terms; C3: For semantic units containing long sentences, parallel structures, or continuous descriptions, the system increases the prediction time accordingly; for semantic units with short content or simple expression, the prediction time is shortened accordingly. C4: After completing the prediction of all semantic units, generate the predicted speech duration sequence: ; Calculate the total predicted speech duration for the entire text. 。 4. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 3, characterized in that: Therefore, the prediction segmentation layer includes a sub-display unit segmentation module, an image prompt word generation module, and a speech / audio generation module; The sub-display unit division module is used to perform secondary division of semantic units based on the predicted duration output by the speech duration prediction module; The image prompt word generation module is used to generate corresponding image prompt words according to the sub-display unit, and convert the text semantics into descriptive information that the image generation model can recognize; The speech and audio generation module is used to generate corresponding speech based on the text of the sub-display unit and to establish a unified speech resource management mechanism.
5. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 4, characterized in that: The specific processing steps for the sub-display unit division module are as follows: D1: Read the predicted speech duration sequence output by the speech duration prediction module, judge the predicted duration corresponding to each semantic unit, and when the predicted speech duration corresponding to a certain semantic unit exceeds the preset threshold, it is considered that the semantic unit contains a large amount of information and needs to be divided into multiple sub-display units; when the predicted duration does not exceed the threshold, the original semantic unit remains unchanged. D2: The semantic units that need to be divided are split according to the principle of semantic integrity; D3: The number of sub-display units is determined based on the predicted speech duration. ; in, The number of sub-display units after division. For the first Predicted speech duration for each semantic unit, The preset target duration for a single display unit; D4: After the division is completed, each sub-display unit is renumbered and inherits the context relationship of its semantic unit to form a new sequence of sub-display units. ; The specific processing steps of the image prompt generation module are as follows: E1: Read the sequence of sub-display units, extract visual elements, and generate corresponding image prompts according to the input format of the image generation model; E2: For continuous sub-display units, the consistency of character characteristics, scene environment and overall style is maintained in combination with the context. When the semantics change, the corresponding visual description is automatically adjusted. E3: Establish a one-to-one correspondence between sub-display units and image prompts, forming a sequence of image prompts: ; in, For the first In the semantic unit, the first Image prompts corresponding to each sub-display unit; The specific processing steps of the speech audio generation module are as follows: F1: Read the text of each sub-display unit and call the speech synthesis model to generate the corresponding speech; F2: Unify the audio encoding and establish the correspondence between voice numbers and sub-display units; F3: Writes voice resource information into the database and creates a resource index.
6. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 5, characterized in that: The multimodal generation layer includes an image sequence generation module, a speech sequence generation module, and a duration information acquisition module; The image sequence generation module is used to generate corresponding images based on image prompts and organize them into an image sequence according to the order of the sub-display units; The speech sequence generation module is used to organize multiple speech segments output by the speech audio generation module into a continuous speech sequence according to the order of the sub-display units, and to establish the correspondence between the speech segments and the sub-display units; The duration information acquisition module is used to calculate the actual playback duration of each audio segment and to serve as the basis for dynamic timeline calibration.
7. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 6, characterized in that: The calibration synchronization layer includes a dynamic timeline calibration module, an image display time adjustment module, and a unified timeline generation module; The dynamic time axis calibration module is used to dynamically correct the image display time based on the error between the predicted speech duration and the actual speech duration. The image display time adjustment module adaptively corrects the display time of each image based on the error results output by the dynamic time axis calibration module. The unified timeline generation module is used to construct a unified multimodal timeline based on image display time, audio playback time, and subtitle information.
8. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 7, characterized in that: The specific processing steps of the dynamic time axis calibration module are as follows: S1: Read the predicted speech duration and the actual speech playback duration output by the speech duration prediction module, and calculate the time error corresponding to each sub-display unit. ; in, For the first Each sub-display unit corresponds to the actual playback duration of the audio. To predict speech duration, This is due to time error; S2: The image display time is dynamically updated through an error feedback mechanism. ; in, For the first During the nth iteration The image display time corresponding to each sub-display unit For error response coefficients, For local smoothing coefficients, This is the global compensation coefficient; Introduce the error convergence criterion: ; in, A preset error threshold is set. When the time error corresponding to all sub-display units is less than the threshold, the current timeline is considered to have been calibrated and no further time adjustment is performed. S3: After completing the time calibration of all sub-display units, the system outputs the calibrated image display time series and retains the corresponding error data.
9. The multimodal video generation system based on speech temporal constraints and feedback calibration according to claim 8, characterized in that: The adjustment relationship of the image display time adjustment module is as follows: ; in, For the adjusted image display time, To predict speech duration, This refers to the time error after dynamic time axis calibration.
