A method and system for automatically generating a video based on multi-modal input

Abstract

The present application relates to the technical field of digital image processing, in particular to a kind of method and system for automatically generating video based on multi-modal input, its method includes: obtaining text description, reference image and background audio, extract text semantic feature, image style feature, audio emotion feature and audio timing feature;Determine the semantic consistency of text and image, the emotion consistency of text and audio and the matching relationship of image and audio, construct multi-modal coordination weight matrix;Then each input and audio timing feature are input into conditional video generation model for weighted fusion, generate target video.The present application can improve the content coordination, style uniformity and rhythm matching of generated video.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, and specifically to a method and system for automatically generating video based on multimodal input. Background Technology

[0002] As animation production and intelligent content creation move towards automation and large-scale production, video generation technology has become a crucial technological direction in digital content processing. Text can provide content intent, images can provide visual style, and audio can provide emotional atmosphere and rhythmic cues. While existing automated generation solutions can shorten production cycles and reduce manual splicing costs, most simply call up or overlay different inputs in parallel, lacking unified processing of content consistency, emotional coordination, and temporal correspondence. This easily leads to problems such as thematic deviations, inconsistent styles, and mismatched rhythms, thus affecting generation quality and processing stability. Summary of the Invention

[0003] This invention provides a method and system for automatically generating videos based on multimodal input, which addresses at least the problem of ensuring consistency between content semantics, visual style, and audio rhythm during the automatic video generation process.

[0004] In a first aspect, the present invention provides a method for automatically generating video based on multimodal input, the method comprising: Obtain text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features; The semantic consistency between the text description and the reference image is determined based on the text semantic features and the image style features, and the emotional consistency between the text description and the background audio is determined based on the text semantic features and the audio emotional features. The matching relationship between the reference image and the background audio is determined based on image style features and audio emotion features, and a multimodal coordination weight matrix is ​​constructed based on semantic consistency, emotion consistency and matching relationship. The text description, reference image, background audio, and audio temporal features are input into the conditional video generation model. The text description, reference image, and background audio are then weighted and fused according to the multimodal coordinated weight matrix to generate the target video corresponding to the background audio temporal sequence.

[0005] In one possible implementation, the extraction steps include: text encoding of the text description to obtain text semantic features; image encoding of the reference image to obtain image style features; and audio encoding of the background audio to obtain audio emotion features and audio temporal features.

[0006] In one possible implementation, the steps for determining semantic consistency include: mapping text semantic features and image style features to a unified semantic representation space; and determining semantic consistency based on similarity in the unified semantic representation space.

[0007] In one possible implementation, the steps for determining emotional consistency include: extracting emotional information from text semantic features; mapping the emotional information and audio emotional features to a unified emotional representation space; and determining emotional consistency based on the degree of difference in the unified emotional representation space.

[0008] In one possible implementation, the steps for determining the matching relationship include: mapping image style features and audio emotion features to a unified representation space; and determining the matching relationship based on the similarity in the unified representation space.

[0009] In one possible implementation, the steps of constructing the multimodal coordination weight matrix include: determining a first weight value between the text description and the reference image, a second weight value between the text description and the background audio, and a third weight value between the reference image and the background audio; and normalizing the first weight value, the second weight value, and the third weight value to obtain the multimodal coordination weight matrix.

[0010] In one possible implementation, the step of weighted fusion of text description, reference image and background audio based on multimodal coordinated weight matrix includes: dividing the audio into multiple time periods based on audio temporal features; determining segmented weight matrices for each time period based on the multimodal coordinated weight matrix; and weighted fusion of text description, reference image and background audio within each time period based on each segmented weight matrix.

[0011] In one possible implementation, the step of weighted fusion of text descriptions, reference images, and background audio within each time period based on the segment weight matrix includes: determining the fusion parameters corresponding to each time period based on the segment weight matrix; generating video segments corresponding to each time period based on the video generation model controlling the conditions according to the fusion parameters; and temporally splicing the video segments to obtain the target video.

[0012] In one possible implementation, the step of generating video segments corresponding to each time period according to the video generation model under the control conditions of each fusion parameter includes: generating keyframes according to each fusion parameter; generating intermediate frames according to adjacent keyframes; and combining the keyframes and intermediate frames into video segments corresponding to each time period.

[0013] Secondly, the present invention provides an automatic video generation system based on multimodal input, for implementing an automatic video generation method based on multimodal input, the system comprising: The input extraction module is used to obtain text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features; The consistency determination module is used to determine the semantic consistency between the text description and the reference image based on the text semantic features and the image style features, and to determine the emotional consistency between the text description and the background audio based on the text semantic features and the audio emotion features. The weight construction module is used to determine the matching relationship between the reference image and the background audio based on image style features and audio emotion features, and to construct a multimodal coordinated weight matrix based on semantic consistency, emotion consistency and matching relationship. The video generation module is used to input text descriptions, reference images, background audio, and audio temporal features into the conditional video generation model, and to perform weighted fusion of text descriptions, reference images, and background audio according to the multimodal coordinated weight matrix to generate a target video corresponding to the temporal sequence of the background audio.

[0014] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: By extracting text, image, and audio features and performing consistency judgment, a pre-verification of the semantic and emotional relationship of the content was achieved; by constructing a multimodal coordination weight matrix, quantitative collaborative control of different input information was achieved; by combining audio temporal features to drive conditional video generation, a stable rhythmic correspondence between the generated video and the background audio was achieved, and the content coordination, style consistency, and generation controllability of the target video were improved. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the execution flow of the method of the present invention; Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation

[0016] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0017] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0018] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0019] Multimodal processing refers to a technical approach that unifies, correlates, analyzes, and collaboratively processes different information carriers such as text, images, and audio. Different modalities carry different types of information: text focuses on expressing thematic content, action relationships, and narrative clues; images focus on expressing visual style, composition, and tone; and audio focuses on expressing emotional changes, rhythmic fluctuations, and temporal sequence. In animation production, the final video is not determined by a single piece of information but is formed by the constraints of multiple pieces of information. Therefore, the key to multimodal processing lies not in simply receiving multiple inputs, but in establishing correspondences, coordination relationships, and fusion rules between different modalities, ensuring that content expression, visual presentation, and temporal rhythm remain unified within the same generation chain. Based on this, this invention proposes an automatic video generation technology solution based on multimodal input, focusing on feature extraction, consistency determination, matching analysis, and collaborative control among multimodal information.

[0020] like Figure 1 As shown, an automatic video generation method based on multimodal input is proposed, which includes: Obtain text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features; In this embodiment, user-inputted text description, reference image, and background audio are first received, and all three types of input are cached in the same processing task. The text description is used to characterize the theme, scene relationships, and emotional tendency of the video to be generated; the reference image provides a visual style benchmark; and the background audio provides emotional atmosphere and temporal rhythm. After input reception is complete, text encoding, image encoding, and audio encoding are performed sequentially to obtain text semantic features, image style features, audio emotional features, and audio temporal features, respectively. Among them, the audio temporal features are used to characterize the beat position, energy change position, or paragraph transition position in the background audio. Through this processing, the raw input can be transformed into structured feature results that can be directly called upon in subsequent consistency judgments and video generation, avoiding instability in the processing chain caused by directly processing the raw data in subsequent stages.

[0021] The extraction steps include: text encoding of the text description to obtain text semantic features; image encoding of the reference image to obtain image style features; and audio encoding of the background audio to obtain audio emotion features and audio temporal features.

[0022] In one embodiment, the feature extraction process is completed using a modal-independent encoding method, and after each encoding is completed, the features are uniformly written into the same task context so that they can be directly read in subsequent processing stages. For text descriptions, basic cleaning is first performed to remove meaningless whitespace characters, repeated delimiters, and abnormal encoded characters, and then the text is segmented by sentences or phrases to obtain a normalized text sequence.

[0023] Subsequently, the normalized text sequence is input into the text encoding network. The text encoding network can employ a pre-trained language encoding model or a coding structure based on word vectors and context aggregation. During the encoding process, the text sequence is first converted into word-level representations, then the semantic representation of each segment of text is calculated in conjunction with contextual relationships. Finally, all semantic representations are aggregated to output fixed-length text semantic features. These text semantic features primarily reflect the subject matter, action relationships, scene information, and emotional tendency in the text description. For the reference image, size unification, color range normalization, and format conversion are performed first to ensure the reference image meets the input requirements of the image encoding network.

[0024] Image coding networks can employ convolutional neural networks or visual coding models. During the encoding process, color distribution, brightness relationships, texture density, edge morphology, and subject composition information are first extracted from the reference image. This information is then fused to obtain image style features. These image style features characterize the overall visual style of the reference image, rather than being directly used as pixel-by-pixel image data in subsequent processing. For background audio, sampling rate unification, channel unification, and amplitude normalization are first performed, followed by frame segmentation for a fixed duration. After framing, energy changes, spectral distribution, and rhythm peaks in the audio segments are extracted, along with corresponding emotional cues. The audio segments are then input into the audio coding network to obtain audio emotional features and audio temporal features. Audio emotional features characterize the background audio's emotional state, such as soothing, tense, cheerful, or somber, while audio temporal features characterize beat points, accent points, energy abrupt changes, and segment boundaries in the background audio.

[0025] In practical implementation, audio temporal features can be further converted into time position data corresponding to the target video frame rate, facilitating direct access during subsequent generation stages when switching shots, changing actions, or adjusting the rhythm of the scene. After completing text encoding, image encoding, and audio encoding, text semantic features, image style features, audio emotion features, and audio temporal features are associated and stored according to the same task identifier, thus forming a basic feature set for subsequent processing stages. Through this method, standardized representation of multimodal data can be achieved during the input stage, eliminating the need for subsequent stages to repeatedly read the original text, images, and audio to continue consistency determination, weight construction, and video generation processing.

[0026] The semantic consistency between the text description and the reference image is determined based on the text semantic features and the image style features, and the emotional consistency between the text description and the background audio is determined based on the text semantic features and the audio emotional features. In this embodiment, after extracting text semantic features, image style features, and audio emotion features, the process proceeds to a consistency determination stage. This stage first uses text semantic features and image style features to determine the degree of matching between the text description and the reference image in terms of subject content, scene expression, and style orientation, obtaining a semantic consistency result. Then, it uses text semantic features and audio emotion features to determine the degree of coordination between the text description and the background audio in terms of emotional tone, obtaining an emotion consistency result. The semantic consistency result reflects whether the visual content matches the text intent, while the emotion consistency result reflects whether the audio atmosphere matches the text expression. By completing these two types of consistency determinations in advance, a clear basis can be provided for constructing multimodal coordination relationships in subsequent stages, avoiding content conflicts or emotional deviations when directly merging different modalities.

[0027] The steps to determine semantic consistency include: mapping text semantic features and image style features to a unified semantic representation space; and determining semantic consistency based on similarity in the unified semantic representation space.

[0028] In one embodiment, semantic consistency is determined using a unified semantic representation space mapping method. The unified semantic representation space is a feature alignment space jointly established for textual semantic features and image style features. Its function is to convert two types of features, originally from different sources and with different expressions, into a unified representation that can be directly compared. Specifically, the textual semantic features are first normalized to ensure they meet the requirements for subsequent comparison in terms of dimensionality and numerical range; then, the image style features are similarly normalized to ensure they maintain consistency with the textual semantic features in terms of representation scale.

[0029] After normalization, the text semantic features and image style features are projected onto the semantic mapping network to obtain text-side semantic vectors and image-side semantic vectors, respectively. The semantic mapping network can adopt a two-way input mapping structure with a shared alignment target. The text-side branch is responsible for preserving the subject objects, action relationships, and scene description information in the text, while the image-side branch is responsible for preserving the subject style, compositional tendency, and visual subject information in the reference image.

[0030] After mapping, the similarity between the text-side semantic vectors and the image-side semantic vectors is compared within a unified semantic representation space. The similarity can be measured by vector angle relationship, vector proximity relationship, or normalized similarity. In engineering implementations, the similarity calculation method with lower computational cost and better stability is preferred. If the similarity reaches a preset standard, the text description and the reference image are considered to have high semantic consistency; if the similarity is lower than the preset standard, the text description and the reference image are considered to have semantic deviation. In actual deployment, the preset standard can be determined based on the statistical results of training samples, or the consistency tolerance range can be configured according to the business scenario.

[0031] To ensure stable results, semantic consistency can be output as continuous numerical values, or it can be divided into three levels: high consistency, medium consistency, and low consistency. Continuous numerical values ​​are suitable for direct participation in subsequent weight calculations, while the level-based results are suitable for subsequent rule judgments. Through this process, the content intent in the text description can establish a clear correspondence with the visual style in the reference image, thus providing an executable basis for judgment in the subsequent video generation stage regarding theme selection and style constraints.

[0032] The steps to determine emotional consistency include: extracting emotional information from text semantic features; mapping the emotional information and audio emotional features to a unified emotional representation space; and determining emotional consistency based on the degree of difference in the unified emotional representation space.

[0033] In one embodiment, emotion consistency is determined by combining text emotion extraction with alignment to a unified emotion representation space. Although text semantic features already contain the overall semantic information of the text description, in the emotion consistency judgment stage, it is necessary to further separate the parts directly related to emotion expression from the text semantic features to avoid thematic information interfering with emotion determination.

[0034] In practice, the emotion parsing unit first performs secondary analysis on the semantic features of the text to extract emotional information reflecting the emotional tendency in the text description. Emotional information mainly includes two aspects: emotional direction and emotional intensity. Emotional direction is used to distinguish whether the text description leans towards calmness, tension, joy, or repression, while emotional intensity reflects the prominence of emotional expression in the text description. After extraction, the emotional information and audio emotional features are input into the emotion mapping network to obtain text-side and audio-side emotion vectors, respectively. The unified emotion representation space is a common alignment space established for text-side emotional information and audio emotional features. Its main function is to ensure that the emotional tendency expressed in the text and the emotional state expressed in the background audio fall within the same comparison framework. During the training phase, the emotion mapping network can establish mapping relationships using manually labeled emotion samples. During the inference phase, the text-side and audio-side emotion vectors are projected into the unified emotion representation space. After projection, the degree of difference between the text-side and audio-side emotion vectors is compared. The smaller the degree of difference, the more harmonious the text description and background audio; the larger the degree of difference, the more likely there is an emotional misalignment between the text description and background audio.

[0035] The degree of difference can be calculated using vector distance, segmented difference statistics, or weighted deviation comparison. In practical applications, the implementation method can be selected based on the device's computing power and response time requirements. For ease of subsequent processing, the emotion consistency result can be output as a continuous consistency score, or it can be converted into three states—consistent, slightly deviating, and significantly deviating—according to a preset threshold. When there are multiple segments of emotion changes in the background audio, the local emotion differences can be calculated first according to time segments, and then the results of all segments can be summarized to obtain an emotion consistency result that better reflects the actual state of the entire background audio segment. Through the above processing, it can be determined whether the text description and the background audio maintain coordination in emotional tone before video generation, thus providing a direct basis for establishing subsequent multimodal coordination relationships and allocating fusion intensity.

[0036] The matching relationship between the reference image and the background audio is determined based on image style features and audio emotion features, and a multimodal coordination weight matrix is ​​constructed based on semantic consistency, emotion consistency and matching relationship. In this embodiment, after determining the semantic consistency between the text description and the reference image, and the emotional consistency between the text description and the background audio, the matching relationship between the reference image and the background audio is further determined based on image style features and audio emotional features. A multimodal coordination weight matrix is ​​then constructed by combining the aforementioned two types of consistency results. The matching relationship reflects whether the visual style expressed by the reference image and the emotional atmosphere expressed by the background audio are coordinated, and the multimodal coordination weight matrix is ​​used to uniformly characterize the correlation strength between the text description, the reference image, and the background audio. Through this processing, a clear basis for modal coordination can be formed before video generation, providing a directly callable weight allocation foundation for the subsequent fusion process.

[0037] The steps to determine the matching relationship include: mapping image style features and audio emotion features to a unified representation space; and determining the matching relationship based on the similarity in the unified representation space.

[0038] In one embodiment, the matching relationship between the reference image and the background audio is obtained through a mapping and comparison process using a unified representation space. The unified representation space is a common comparison space established for image style features and audio emotion features, and its function is to convert visual style information and audio emotion information into directly comparable representations of the same type.

[0039] In practice, the image style features are first normalized to ensure they meet the requirements for subsequent comparison in terms of numerical range and representation length. Then, the audio emotion features are similarly normalized to ensure they maintain the same representation scale as the image style features. After normalization, the image style features are input into the style mapping unit, and the audio emotion features are input into the emotion mapping unit, outputting image-side and audio-side representation results respectively. The style mapping unit preserves the color tendency, brightness distribution, texture density, visual rhythm, and subject composition preferences in the reference image, while the emotion mapping unit preserves the emotional direction, intensity, rhythmic fluctuations, and atmospheric change trends in the background audio.

[0040] After mapping, the image-side representation and the audio-side representation are projected onto the same representation range, and their similarity is compared. A higher similarity indicates a more harmonious visual style between the reference image and the emotional state of the background audio; a lower similarity indicates a greater likelihood of atmospheric conflict between the reference image and the background audio. For example, if the reference image exhibits low saturation, low contrast, and a smooth composition, and the background audio also exhibits a soothing, low-fluctuation emotional state, then their similarity is high; if the background audio exhibits a high-intensity, fast-paced, and impactful emotional state, then their similarity is low. To facilitate subsequent processing, the similarity can be converted into a matching relationship result. The matching relationship result can be represented by a continuous score or divided into three levels—high matching, medium matching, and low matching—according to a preset interval.

[0041] Continuous scores are suitable for direct weighting calculations, while graded results are suitable for rule-based judgments. When there are significant segmentation changes in the background audio, the local matching degree can be calculated separately for each audio time segment, and then the local matching degrees of each time segment can be summarized to obtain the overall matching relationship result. Through the above processing, the relationship between the reference image and the background audio is no longer limited to intuitive judgment, but is transformed into a quantitative basis that can be directly used to construct subsequent multimodal coordination relationships, thereby ensuring that visual style and audio atmosphere can be uniformly processed within the same coordination framework during subsequent video generation.

[0042] The steps for constructing a multimodal coordinated weight matrix include: determining the first weight value between the text description and the reference image, the second weight value between the text description and the background audio, and the third weight value between the reference image and the background audio; normalizing the first weight value, the second weight value, and the third weight value to obtain the multimodal coordinated weight matrix.

[0043] In one embodiment, the multimodal coordination weight matrix is ​​constructed based on semantic consistency, sentiment consistency, and matching relationship. It is used to reflect the strength of the association between the text description, reference image, and background audio, and serves as the direct control basis for the subsequent fusion stage.

[0044] In practice, a first weight value is determined based on the semantic consistency between the text description and the reference image. This first weight value characterizes the degree of content synergy between the text description and the reference image. Higher semantic consistency results in a larger first weight value, indicating a stronger constraint effect of the reference image on the visual representation of the text description during subsequent generation; lower semantic consistency results in a smaller first weight value, indicating that the reference image serves only as a weak style reference during subsequent generation. Next, a second weight value is determined based on the emotional consistency between the text description and the background audio. This second weight value characterizes the degree of atmospheric synergy between the text description and the background audio. Higher emotional consistency results in a larger second weight value, indicating a stronger driving effect of the background audio on the emotional rhythm of the video; lower emotional consistency results in a smaller second weight value, indicating a correspondingly weaker participation of the background audio in emotional control. Finally, a third weight value is determined based on the matching relationship between the reference image and the background audio. This third weight value characterizes the degree of style synergy between the reference image and the background audio. A higher matching relationship results in a larger third weight value, indicating that the visual style and audio atmosphere are more suitable for jointly participating in video generation; a lower matching relationship results in a smaller third weight value, indicating that subsequent fusion should avoid both simultaneously influencing the generated result with high intensity.

[0045] After obtaining the first, second, and third weight values, normalization is performed on them to ensure they fall within a unified weight range. The purpose of normalization is to eliminate differences in units and value ranges between results from different sources, preventing any single weight from excessively influencing subsequent fusion due to an overly large original scale. After normalization, the first, second, and third weight values ​​are written into a matrix structure according to preset positions, forming a multimodal coordination weight matrix. This multimodal coordination weight matrix can be stored as a two-dimensional matrix or as a weight table associated with task identifiers, as long as the three sets of relationships are clearly defined. In the subsequent video generation stage, the fusion strength of text descriptions, reference images, and background audio is directly determined based on the multimodal coordination weight matrix, thus enabling content expression, visual style, and audio atmosphere to work synergistically under the same weight control.

[0046] In the above way, the three relational results obtained from the preceding judgment can be transformed into a unified and executable fusion control basis, ensuring that the entire processing chain has clear intermediate support when transitioning from consistency judgment to video generation.

[0047] The text description, reference image, background audio, and audio temporal features are input into the conditional video generation model. The text description, reference image, and background audio are then weighted and fused according to the multimodal coordinated weight matrix to generate the target video corresponding to the background audio temporal sequence.

[0048] In this embodiment, after constructing semantic consistency, emotional consistency, and the multimodal coordination weight matrix, text descriptions, reference images, background audio, and audio temporal features are jointly input into the conditional video generation model. The conditional video generation model uses text descriptions to provide content constraints, reference images to provide visual style constraints, and background audio and audio temporal features to provide rhythm constraints. It then combines the multimodal coordination weight matrix to weightedly fuse the three types of input information, generating a target video corresponding to the background audio temporal sequence. This process ensures that video content expression, visual style, and audio rhythm are unified and coordinated within the same generation stage, avoiding abrupt shot transitions or rhythmic mismatches caused by generating visuals first and then passively aligning the audio.

[0049] The steps for weighted fusion of text description, reference image and background audio based on multimodal coordinated weight matrix include: dividing the audio into multiple time periods based on audio temporal features; determining segmented weight matrices for each time period based on the multimodal coordinated weight matrix; and weighted fusion of text description, reference image and background audio within each time period based on each segmented weight matrix.

[0050] In one embodiment, the weighted fusion process first divides the entire background audio segment into time segments based on audio temporal features, and then organizes the fusion processing of text description, reference image, and background audio separately within each time segment. The audio temporal features already include temporal information such as beat points, accent points, energy change points, and paragraph boundaries, so the intervals between adjacent beat points, the intervals between adjacent accent points, or the intervals between adjacent paragraph boundaries can be directly used as the basis for dividing the time segments.

[0051] In practice, an initial timeline can be formed based on the overall length of the background audio. Then, key time positions are marked on the timeline according to the audio's temporal characteristics. The intervals between adjacent key time positions are then defined as corresponding time periods. If the intervals between adjacent key time positions are too short, adjacent intervals can be merged to avoid single time periods being too short and detrimental to image generation. If the intervals between adjacent key time positions are too long, the intervals can be further subdivided into fixed-length segments to avoid single time periods being too long and weakening the rhythm control precision. After completing the time division, a segmented weight matrix is ​​determined for each time period based on the multimodal coordination weight matrix.

[0052] The segmented weight matrix is ​​not an independently generated new weight, but rather a time-segment-level weight result obtained by locally adjusting the audio temporal position and emotional state corresponding to the current time segment based on the multimodal coordinated weight matrix. Specifically, if the current time segment corresponds to an accented or emotionally enhanced position in the background audio, the participation weight of the background audio in the current time segment can be appropriately increased; if the current time segment corresponds to a flat position in the background audio, the constraint weight of the text description and reference image in the current time segment can be appropriately increased to make the expression of the visual content more stable.

[0053] For text descriptions, the most suitable visual content for the current time period can be determined based on the theme and action information within the text description. For reference images, the most suitable colors, brightness, and framing for the current time period can be determined based on the visual style and compositional tendencies of the reference images. Thus, each time period does not mechanically apply the same set of weights, but rather forms a segmented weight matrix with time-distinctive characteristics based on a unified weighting. After determining the segmented weight matrix, the text description, reference image, and background audio are called separately within each time period, and weighted fusion is performed according to the corresponding segmented weight matrix. Through this method, video generation no longer treats the entire audio segment as a single processing unit, but rather forms a segment-by-segment control structure along the timeline, thereby ensuring that the generation process maintains global consistency while possessing dynamic adjustment capabilities corresponding to local rhythm changes.

[0054] The steps of weighted fusion of text descriptions, reference images, and background audio within each time period based on the segment weight matrix include: determining the fusion parameters corresponding to each time period based on the segment weight matrix; generating video segments corresponding to each time period based on the video generation model under the control conditions of each fusion parameter; and temporally splicing the video segments to obtain the target video.

[0055] In one embodiment, weighted fusion within each time period is performed using a fusion parameter-driven conditional video generation model, and temporal stitching is executed after all time periods have been processed to obtain the complete target video. Specifically, fusion parameters are first extracted from the segmented weight matrix for each time period. These fusion parameters describe the degree of participation and generation control of the three types of input information within the current time period, including at least content-driven intensity, style-driven intensity, audio-driven intensity, and the video duration of the current time period.

[0056] Content guidance intensity reflects the degree to which the text description constrains the theme, action relationships, and scene organization of the scene within the current time period; style guidance intensity reflects the degree to which the reference image constrains the color distribution, texture representation, and composition within the current time period; audio drive intensity reflects the degree to which background audio drives shot transitions, action rhythm, and visual fluctuations within the current time period; video duration is used to limit the length of the video segment corresponding to the current time period. After the fusion parameters are determined, the text description segment information, reference image style information, background audio segment information, and fusion parameters corresponding to the current time period are jointly input into the conditional video generation model. The conditional video generation model can first establish the generation context of the current time period internally, and then control the influence of various input information in the generation process according to the fusion parameters. For example, when the content guidance intensity is high, the conditional video generation model prioritizes ensuring that the scene content is consistent with the text description; when the style guidance intensity is high, the conditional video generation model prioritizes maintaining the color tone and scene style corresponding to the reference image; when the audio drive intensity is high, the conditional video generation model prioritizes responding to rhythmic and emotional changes in the background audio. In this way, the generation of video clips is not simply a superposition of text, images and audio, but a generation result with differentiated control based on time period under the constraints of fusion parameters.

[0057] After processing each time segment, multiple video clips are obtained in chronological order. Then, these video clips are sequentially spliced ​​together according to their original time segment sequence within the overall background audio. During splicing, priority is given to ensuring the continuity of playback order between adjacent video clips, and maintaining the consistency between clip boundaries and corresponding audio clip boundaries. If there is a significant style shift at the connection point between adjacent video clips, a short transition can be performed at the clip boundary to create a natural connection between the end of the previous clip and the beginning of the next. After splicing, the target video corresponding to the background audio sequence is obtained. This process ensures that the entire video maintains consistency in content, style, and rhythm, and that each scene change has a clear temporal basis.

[0058] The steps for generating video segments corresponding to each time period based on the video generation model under the control conditions of each fusion parameter include: generating keyframes based on each fusion parameter; generating intermediate frames based on adjacent keyframes; and combining the keyframes and intermediate frames into video segments corresponding to each time period.

[0059] In one embodiment, video segments corresponding to each time period are generated using a combination of keyframes and intermediate frames. This method ensures the continuity of the footage while reducing the computational burden of direct frame-by-frame generation and also facilitates the stable transfer of fusion parameters to the complete segment.

[0060] In practice, the conditional video generation model first generates keyframes based on the fusion parameters corresponding to the current time segment. Keyframes represent representative image states within the current time segment, typically corresponding to the start, end, and rhythmic changes of that segment. If the current time segment is short, only start and end keyframes may be generated; if the current time segment is long, or if there are significant audio rhythm changes, one or more keyframes may be added to express the phased changes in image content or visual rhythm. During keyframe generation, text descriptions primarily determine the subject object, action state, and scene layout, while reference images primarily determine the color tone, lighting, and composition style. Background audio and fusion parameters primarily determine the direction of rhythmic changes between keyframes. After keyframe generation, intermediate frames are generated based on adjacent keyframes. These intermediate frames fill in the image changes between adjacent keyframes, ensuring the video clip has the complete frame sequence required for continuous playback.

[0061] When generating intermediate frames, the changes in subject position, camera angle, and color between the previous and subsequent keyframes can be analyzed first. Then, transitional shots are added between the two frames at a preset frame rate. For scenes with significant subject movement, a smooth transition can be generated based on changes in subject position and posture between the preceding and following keyframes. For scenes with significant camera angle changes, a gradual transition can be generated based on differences in shot size and composition between the preceding and following keyframes. For scenes with significant emotional changes, a progressive visual transition can be generated based on changes in hue and brightness between the preceding and following keyframes. After generating intermediate frames, keyframes and intermediate frames are combined in chronological order to form the video segment corresponding to the current time period. Repeating the above process for each time period yields all video segments. The core of this implementation is to first use keyframes to stably express the important visual states within a time period, and then use intermediate frames to supplement the continuous changes. This ensures that the video segments retain the main constraints provided by the text description and reference images while maintaining consistency with the rhythmic changes corresponding to the background audio.

[0062] like Figure 2 As shown, an automatic video generation system based on multimodal input is used to implement the automatic video generation method based on multimodal input. The system includes: The input extraction module is used to acquire text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features. The input extraction module mainly consists of an input interface circuit, a memory, and a feature extraction processing unit. The input interface circuit receives the text descriptions, reference images, and background audio, and may include a network interface, file interface, camera input interface, or audio input interface. The memory is used to cache raw input data and intermediate data. The feature extraction processing unit is typically implemented by a central processing unit in conjunction with a graphics processing unit or neural network accelerator, and is used to perform text encoding, image encoding, and audio encoding respectively, thereby outputting text semantic features, image style features, audio emotion features, and audio temporal features.

[0063] The consistency determination module is used to determine the semantic consistency between the text description and the reference image based on text semantic features and image style features, and to determine the emotional consistency between the text description and the background audio based on text semantic features and audio emotion features. The consistency determination module mainly consists of a feature reading unit, a comparison calculation unit, and a result storage unit. The feature reading unit retrieves text semantic features, image style features, and audio emotion features from memory. The comparison calculation unit, typically implemented by a central processing unit, digital signal processor, or neural network accelerator, performs feature alignment, similarity calculation, or difference calculation to obtain semantic consistency and emotional consistency. The result storage unit stores the consistency determination results for subsequent weight construction.

[0064] The weight construction module determines the matching relationship between the reference image and the background audio based on image style features and audio emotion features, and constructs a multimodal coordination weight matrix based on semantic consistency, emotion consistency, and matching relationship. The weight construction module mainly consists of a relationship analysis unit, a weight calculation unit, and a matrix storage unit. The relationship analysis unit determines the matching relationship between the reference image and the background audio based on image style features and audio emotion features. The weight calculation unit calculates the multimodal coordination weights by combining semantic consistency, emotion consistency, and matching relationship. This part is typically calculated by the central processing unit (CPU), but matrix operations can also be performed by dedicated acceleration circuitry. The matrix storage unit stores the multimodal coordination weight matrix and provides a read channel for subsequent video generation modules.

[0065] The video generation module takes text descriptions, reference images, background audio, and audio temporal features as input to the conditional video generation model. It then weights and fuses these elements according to a multimodal coordination weight matrix to generate a target video corresponding to the background audio temporal sequence. The video generation module mainly consists of a model execution unit, a video memory or cache unit, a video synthesis unit, and an output interface unit. The model execution unit, typically implemented using a graphics processing unit (GPU), a neural network processor, or a high-performance central processing unit (CPU), runs the conditional video generation model. The video memory or cache unit stores model parameters, input features, and frame data generated during the generation process. The video synthesis unit fuses the text descriptions, reference images, and background audio according to the multimodal coordination weights and generates a video frame sequence corresponding to the background audio temporal sequence. The output interface unit writes the target video to local memory or outputs it to an external device via a network interface.

[0066] From an engineering perspective, these four modules can be integrated into a single software and hardware co-location on a server motherboard, or they can be distributed across acquisition terminals, inference devices, and storage devices to work together. The core hardware typically includes a processor, memory, bus, power management circuitry, and necessary input / output interfaces.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0068] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for automatically generating video based on multimodal input, characterized in that, The method includes: Obtain text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features; The semantic consistency between the text description and the reference image is determined based on the text semantic features and the image style features; the emotional consistency between the text description and the background audio is determined based on the text semantic features and the audio emotion features. The matching relationship between the reference image and the background audio is determined based on the image style features and the audio emotion features, and a multimodal coordination weight matrix is ​​constructed based on the semantic consistency, the emotion consistency and the matching relationship. The text description, the reference image, the background audio, and the audio temporal features are input into a conditional video generation model. The text description, the reference image, and the background audio are then weighted and fused according to the multimodal coordinated weight matrix to generate a target video corresponding to the temporal sequence of the background audio.

2. The method according to claim 1, characterized in that, The extraction steps include: The text description is encoded to obtain the text semantic features; The reference image is image encoded to obtain the image style features; The background audio is encoded to obtain the audio emotion features and the audio temporal features.

3. The method according to claim 1, characterized in that, The step of determining the semantic consistency includes: Map the text semantic features and the image style features to a unified semantic representation space; The semantic consistency is determined based on the similarity in the unified semantic representation space.

4. The method according to claim 1, characterized in that, The step of determining the emotional consistency includes: Extract emotional information from the semantic features of the text; Map the emotional information and the audio emotional features to a unified emotion representation space; The emotional consistency is determined based on the degree of difference in the unified emotional representation space.

5. The method according to claim 1, characterized in that, The step of determining the matching relationship includes: Map the image style features and the audio emotion features to a unified representation space; The matching relationship is determined based on the similarity in the unified representation space.

6. The method according to claim 1, characterized in that, The steps for constructing the multimodal coordination weight matrix include: A first weight value between the text description and the reference image, a second weight value between the text description and the background audio, and a third weight value between the reference image and the background audio are determined respectively; The first weight value, the second weight value, and the third weight value are normalized to obtain the multimodal coordination weight matrix.

7. The method according to claim 1, characterized in that, The step of weighted fusion of the text description, the reference image, and the background audio based on the multimodal coordinated weight matrix includes: Divide the audio into multiple time periods based on the aforementioned audio temporal characteristics; Based on the multimodal coordination weight matrix, segmented weight matrices are determined for each of the aforementioned time periods; The text description, the reference image, and the background audio within each time period are weighted and fused according to the segmented weight matrices.

8. The method according to claim 7, characterized in that, The step of weighted fusion of the text description, the reference image, and the background audio within each time period according to each segmented weight matrix includes: The fusion parameters corresponding to each time period are determined based on each segment weight matrix. The conditional video generation model is controlled to generate video segments corresponding to each time period according to the fusion parameters. The target video is obtained by sequentially splicing the video segments.

9. The method according to claim 8, characterized in that, The step of controlling the conditional video generation model to generate video segments corresponding to each time period according to each of the fusion parameters includes: Keyframes are generated based on the fusion parameters described above; Generate intermediate frames based on adjacent keyframes; The keyframes and intermediate frames are combined to form the video segments corresponding to each time period.

10. An automatic video generation system based on multimodal input, used to implement the automatic video generation method based on multimodal input as described in any one of claims 1-9, characterized in that, The system includes: The input extraction module is used to obtain text descriptions, reference images, and background audio, and extract text semantic features, image style features, audio emotion features, and audio temporal features; A consistency determination module is used to determine the semantic consistency between the text description and the reference image based on the text semantic features and the image style features, and to determine the emotional consistency between the text description and the background audio based on the text semantic features and the audio emotion features; The weight construction module is used to determine the matching relationship between the reference image and the background audio based on the image style features and the audio emotion features, and to construct a multimodal coordinated weight matrix based on the semantic consistency, the emotion consistency and the matching relationship; The video generation module is used to input the text description, the reference image, the background audio, and the audio temporal features into a conditional video generation model, and to perform weighted fusion of the text description, the reference image, and the background audio according to the multimodal coordinated weight matrix to generate a target video corresponding to the temporal sequence of the background audio.

Images