A video intelligent synthesis method for multi-modal content conversion

Abstract

The application provides a video intelligent synthesis method for multi-modal content conversion, comprising: acquiring video demand data, performing semantic analysis on the video demand data, generating a video intelligent synthesis data set, performing multi-path material retrieval based on the video intelligent synthesis data set, pruning in combination with a preset dynamic pruning strategy, generating a video synthesis material candidate set and a key frame demand set, generating an intelligent synthesis video segment based on the key frame demand set, performing quality screening on the intelligent synthesis video segment, generating a candidate video synthesis segment set, performing video intelligent synthesis based on beam search guidance according to the candidate video synthesis segment set and the video synthesis material candidate set, generating a candidate intelligent synthesis video set, performing quality evaluation on the candidate intelligent synthesis video set based on a preset video quality evaluation mechanism, and performing secondary synthesis according to the evaluation result to generate an optimal synthesis video, thereby improving the efficiency, accuracy and reliability of automatic video production.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a video intelligent synthesis method for multimodal content conversion. Background Technology

[0002] With the rapid development of artificial intelligence technology, automated video content generation has become an important research direction, especially in fields such as news, education, and business briefings that have high requirements for timeliness and large-scale production.

[0003] Traditional video production processes often rely heavily on the creativity and manual operation of professionals, including script comprehension, material sourcing, editing and compositing, and post-production packaging. This process is time-consuming, labor-intensive, and difficult to scale. Traditional automated video compositing methods mostly lack the ability to perceive and handle uncertainty. Video compositing requirements often contain a large amount of implicit information and multiple possible interpretations. Existing technologies usually output a definite but potentially erroneous intermediate result and pass the error down to downstream processes, causing the final product to deviate from expectations.

[0004] Therefore, there is an urgent need for a video intelligent synthesis method that can deeply integrate multimodal information, perceive and quantify uncertainty, and achieve intelligent collaboration between retrieval and generation, so as to improve the efficiency, accuracy and reliability of automated video production. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a video intelligent synthesis method for multimodal content conversion, the method comprising:

[0006] Acquire video demand data and perform semantic parsing based on uncertainty awareness on the video demand data to generate a video intelligent synthesis dataset;

[0007] Based on the intelligent video synthesis dataset, multi-path material retrieval is performed, and pruning is carried out simultaneously in combination with a preset dynamic pruning strategy to generate a candidate set of video synthesis materials and a key frame requirement set.

[0008] Based on the keyframe requirement set, intelligent synthetic video segments are generated, and the quality of the intelligent synthetic video segments is screened to generate a candidate video synthetic segment set.

[0009] Based on the candidate video synthesis segment set and the candidate video synthesis material set, perform intelligent video synthesis guided by beam search to generate a candidate intelligent synthesized video set;

[0010] The candidate intelligent synthesized video set is evaluated based on a preset video quality evaluation mechanism, and secondary synthesis is performed based on the evaluation results to generate the optimal synthesized video.

[0011] As a further aspect of the present invention, video demand data is acquired, and semantic parsing based on uncertainty awareness is performed on the video demand data to generate a video intelligent synthesis dataset, including:

[0012] The video requirement data includes at least the original descriptive text data and the original video data;

[0013] Perform semantic analysis on the original descriptive text data to obtain a semantic feature dataset;

[0014] Feature extraction is performed on the original image data to generate visual semantic units;

[0015] Based on the semantic feature dataset and visual semantic units, semantic matching and spatiotemporal alignment are performed to obtain semantic matching results;

[0016] The preset rule validator performs rule conflict detection and confidence optimization based on the semantic matching results, and generates an optimized feature set.

[0017] Based on the semantic feature dataset, visual semantic units, and optimized feature set, a video intelligent synthesis dataset is constructed.

[0018] As a further aspect of the present invention, multi-path material retrieval is performed based on a video intelligent synthesis dataset, and pruning is performed simultaneously in conjunction with a preset dynamic pruning strategy to generate a candidate set of video synthesis materials and a keyframe requirement set, including:

[0019] The video intelligent synthesis dataset is parsed to obtain deterministic demand data and exploratory demand data, wherein the deterministic demand data and exploratory demand data at least contain corresponding semantic features and image data;

[0020] Based on the aforementioned deterministic demand data, a first candidate set of video synthesis materials is obtained;

[0021] Based on the aforementioned exploratory demand data, a second set of candidate video synthesis materials is obtained;

[0022] The first and second video synthesis material candidate sets are encapsulated into a video synthesis material candidate set and output.

[0023] As a further aspect of the present invention, based on the deterministic demand data, a first candidate set of video synthesis materials is obtained, including:

[0024] For data with deterministic requirements, the corresponding image data is directly adopted as video synthesis material, and the video synthesis material of all data with deterministic requirements is packaged into the first video synthesis material candidate set.

[0025] As a further aspect of the present invention, based on the exploratory demand data, a second candidate set of video synthesis materials is obtained, including:

[0026] Based on the exploratory demand data, an exploratory demand unit is generated;

[0027] By employing a text-based graph model and simultaneously combining the semantic features corresponding to each exploratory requirement unit, a video material query vector is generated.

[0028] Based on the video material query vector, perform multi-path video material query to obtain the original video synthesis material candidate set for the exploratory demand unit;

[0029] The original video synthesis material candidate set is subjected to quality evaluation and dynamic pruning according to a preset dynamic pruning strategy to construct a second video synthesis material candidate set.

[0030] As a further aspect of the present invention, the method further includes:

[0031] Image frames that do not meet the preset qualification conditions after dynamic pruning are defined as key frames, and a key frame requirement set is generated based on the semantic features corresponding to the key frames.

[0032] As a further aspect of the present invention, intelligent synthetic video segments are generated based on the keyframe requirement set, and the intelligent synthetic video segments are subjected to quality screening to generate a candidate video synthetic segment set, including:

[0033] Based on the keyframe requirement set, a text-based image diffusion model is used to generate keyframes and obtain a keyframe image set.

[0034] The keyframe image set is jointly optimized using a latent diffusion model to generate a primary intelligent synthetic video clip.

[0035] The consistency verification of the primary intelligent synthesized video segment is performed based on a preset fact-checking mechanism to obtain the consistency verification result.

[0036] If the consistency check result is inconsistent with the facts, the parameters of the keyframe generation and intelligent video segment generation stages will be fine-tuned based on the consistency check result.

[0037] After fine-tuning the parameters, the optimized intelligent synthetic video clips are regenerated based on the key frame requirement set until the consistency check result is consistent with the facts.

[0038] All intelligently synthesized video clips are packaged into a candidate video synthesis clip set for output.

[0039] As a further aspect of the present invention, a pre-defined fact-checking mechanism is provided, including:

[0040] The preset fact-checking mechanism is represented as a loop process that includes at least agent A, agent B, and agent C working collaboratively.

[0041] The intelligent agent A is composed of multiple neural networks, which are used to perform logical consistency analysis and obtain the logical consistency analysis results.

[0042] The intelligent agent B is used to perform root cause analysis based on the results of logical consistency analysis, and to generate a set of parameter adjustment instructions based on the results of the root cause analysis.

[0043] The intelligent agent C is used to fine-tune the parameters according to the parameter adjustment instruction set.

[0044] As a further aspect of the present invention, video intelligent synthesis based on beam search guidance is performed according to the candidate video synthesis segment set and the candidate video synthesis material set to generate a candidate intelligent synthesis video set, including:

[0045] A narrative logic backbone is generated based on the video intelligent synthesis dataset, and the first video synthesis material candidate set contained in the video synthesis material candidate set is filled into the narrative logic backbone.

[0046] Based on the candidate video synthesis segment set and the second video synthesis material candidate set contained in the video synthesis material candidate set, the beam search algorithm is run to search for the video synthesis material sequence.

[0047] During the execution of the beam search algorithm, scoring and sequence pruning are performed in conjunction with a preset sequence scoring function.

[0048] The preset sequence scoring function includes at least two dimensions: local matching degree and excessive fluency.

[0049] After the beam search algorithm finishes running, a candidate video synthesis material sequence set is generated, and a candidate intelligent synthesized video set is generated based on the candidate video synthesis material sequence set.

[0050] As a further aspect of the present invention, the candidate intelligent synthetic video set is quality-assessed based on a preset video quality assessment mechanism, and secondary synthesis is performed based on the assessment results to generate the optimal synthetic video, including:

[0051] Based on a preset multidimensional video quality evaluation matrix, a video quality evaluation is performed on each candidate intelligent synthesized video in the candidate intelligent synthesized video set to generate a multidimensional video quality score.

[0052] Based on the actual multi-dimensional video quality evaluation matrix corresponding to each candidate intelligent synthesized video, the deviation source is traced to obtain the deviation source tracing results;

[0053] The multi-dimensional video quality score and deviation tracing results corresponding to each candidate intelligent synthetic video are uploaded to the human-computer interaction interface. Users can make secondary adjustments based on the data on the interface and generate secondary adjustment instructions.

[0054] The candidate intelligent synthetic video set is synthesized a second time based on the secondary adjustment instruction to generate the optimal synthetic video.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] Acquire video demand data and perform semantic parsing based on uncertainty perception on the video demand data to generate a video intelligent synthesis dataset. By performing multimodal joint parsing and uncertainty quantification on the video demand data, a structured synthesis blueprint with confidence assessment is constructed, which helps to understand the demand information more comprehensively in the planning stage, thereby identifying key facts and gaps that need to be filled by creativity, and providing a data foundation for subsequent steps.

[0057] Based on the video intelligent synthesis dataset, multi-path material retrieval is performed, and pruning is carried out simultaneously in combination with a preset dynamic pruning strategy to generate a candidate set of video synthesis materials and a key frame requirement set. Through multi-path retrieval and dynamic pruning strategy, external materials and synthesis material generation requirements are explored and screened, providing data for subsequent material synthesis.

[0058] Intelligent synthetic video segments are generated based on the key frame requirement set, and the quality of the intelligent synthetic video segments is screened to generate a candidate video synthetic segment set. A hierarchical generation strategy is adopted and combined with a multi-agent collaborative evaluation mechanism to improve the reliability and rationality of the generated video segments.

[0059] Based on the candidate video synthesis segment set and the candidate video synthesis material set, intelligent video synthesis guided by beam search is performed to generate a candidate intelligent synthesis video set. The beam search algorithm is used to globally optimize the selection of multi-path materials, which helps to synthesize a better overall candidate video sequence based on comprehensive consideration of segment quality, smoothness of connection and narrative coherence.

[0060] The candidate intelligent synthesized video set is evaluated based on a preset video quality assessment mechanism, and secondary synthesis is performed based on the evaluation results to generate the optimal synthesized video. This ensures that the final product meets expectations in key quality dimensions and provides traceable analytical data for process optimization. Attached Figure Description

[0061] Figure 1 This is a flowchart of the steps of a video intelligent synthesis method for multimodal content conversion according to the present invention;

[0062] Figure 2 This is a flowchart illustrating step S2 in a video intelligent synthesis method for multimodal content conversion according to the present invention. Detailed Implementation

[0063] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating the steps of a video intelligent synthesis method for multimodal content conversion according to the present invention. Figure 2 This is a flowchart illustrating step S2 of a video intelligent synthesis method for multimodal content conversion according to the present invention. The following is a detailed description of this video intelligent synthesis method for multimodal content conversion.

[0064] Step S1: Obtain video requirement data and perform semantic parsing based on uncertainty awareness on the video requirement data to generate a video intelligent synthesis dataset.

[0065] Specifically, the video requirement data includes at least original descriptive text data and original image data. Semantic analysis is performed on the original descriptive text data to obtain a semantic feature dataset, and feature extraction is performed on the original image data to generate visual semantic units.

[0066] Based on the semantic feature dataset and visual semantic units, semantic matching and spatiotemporal alignment are performed to obtain semantic matching results. A preset rule validator performs rule conflict perception and confidence optimization based on the semantic matching results to generate an optimized feature set.

[0067] Based on the semantic feature dataset, visual semantic units, and optimized feature set, a video intelligent synthesis dataset is constructed.

[0068] In one possible embodiment, taking the synthesis of news videos as an example, the video requirement data includes at least original descriptive text data and original video data. The original descriptive text data includes at least the core news text and corresponding supplementary information. The supplementary information includes news categories such as financial markets, sports events, and breaking news; timeliness tags such as breaking news and in-depth reports; regional and audience tags such as domestic, international, and youth groups; and editorial intent information such as highlighting conflict, emphasizing data visualization, and reflecting humanistic care. The original video data refers to the original multimedia materials corresponding to the news text, such as on-site photos taken by reporters, on-site video clips, and historical materials related to the current news event from historical archives.

[0069] For the original descriptive text data, firstly, a Bayesian neural network based on Monte Carlo Dropout is used to process the news article and its supplementary information. Then, a Transformer encoder is used to perform multiple forward propagations and randomly discard some neurons during inference to achieve a probabilistic estimate of the prediction results. Specifically, the text data is semantically segmented into multiple units, such as by sentence or semantic group, and the narrative type label of each unit is predicted. For example, after a news article is segmented, it may correspond to one or more of the narrative type labels such as background description, conflict presentation, data argumentation, opinion citation, and future outlook. Finally, the most likely narrative type label, the corresponding confidence probability, and the probability distribution of all possible labels are output.

[0070] Simultaneously, the Bayesian neural network regresses continuous parameters related to the narrative type label, such as emotional polarity intensity and rhythm coefficient, and outputs their mean and variance, thereby generating a narrative logic trajectory L(t). For example, for the news text "At the press conference, the spokesperson solemnly announced the dismal financial report data for this quarter, and the stock price plummeted accordingly," the model may label it as a narrative type label of conflict presentation with a confidence level of 0.92, and at the same time regress the corresponding continuous parameters and corresponding intensities, obtaining an emotional intensity of -0.8 to represent strong negative emotions and a rhythm coefficient of 0.9 to represent a fast and tense rhythm. At the same time, the data demonstrates that this narrative type label is a low-probability alternative narrative type label.

[0071] Finally, the textual semantic information extracted by the Bayesian neural network is encapsulated into a corresponding semantic feature dataset.

[0072] It should be noted that the Bayesian neural network uses a pre-trained language model as a shared text feature encoder and employs a multi-task output head that includes at least a narrative logic classification head, a continuous value regression head, and a semantic flow peak detection head for data output. Specifically, the narrative logic classification head uses Monte Carlo Dropout to quantify uncertainty through the variance of the prediction results, thereby outputting the probability distribution of each text segment belonging to each narrative type label; the continuous value regression head also uses Monte Carlo Dropout, outputting its mean and variance, where the mean is directly used as the final continuous value prediction result, while the variance serves as a quantification indicator of the prediction uncertainty; the semantic flow peak detection head uses a conditional random field or pointer network to identify which words or phrases in the text are semantic peak points and classify their types.

[0073] The Bayesian neural network uses a large number of news articles annotated by linguistics or journalism professionals according to a predefined narrative grammar as training data. It is trained with the goal of minimizing a preset loss function, which is a weighted sum of cross-entropy loss with Monte Carlo Dropout, mean squared error loss based on sampling, negative log-likelihood loss from conditional random fields, and standard cross-entropy loss.

[0074] Next, a state-space model such as Mamba is used to model the text sequence, thereby capturing the dynamic fluctuations of semantics and outputting a semantic intensity curve S(t) that changes over time. Its peak points correspond to key events or emotional turning points in the text. Similar to narrative logic, the model also outputs the probability distribution of the corresponding intensity and identifies multiple possible peak candidates and their confidence levels. For example, in the above example sentence, "the financial report data was released" and "the stock price plummeted" may correspond to two semantic peaks respectively. Based on the peak candidates and their confidence levels, a corresponding emotional intensity value sequence is generated, that is, each peak candidate corresponds to an emotional fluctuation point, and its corresponding confidence level is the emotional intensity value corresponding to that emotional fluctuation point.

[0075] Next, end-to-end models such as BiLSTM-CRF and BERT are used to perform named entity recognition and relation extraction on the text data. People, organizations, places, times, numbers and events are extracted from the text, and a preliminary relation network between them is constructed. For example, from the above example sentence, entities such as spokesperson (person), financial report data (event / object), and stock price (object) are extracted, as well as the relationship between the spokesperson announcing financial report data and the financial report data showing the stock price plummeting.

[0076] For the raw image data, visual language models such as BLIP-2 and Flamingo are used to generate detailed feature descriptions for each keyframe of the image or video. The feature descriptions include at least the scene, actions, interactions, and emotional atmosphere corresponding to the image data. For example, for a scene image that matches the above example sentence, the visual language model may generate the following description: "A male speaker in a dark suit stands behind the podium with a furrowed brow. The screen behind him displays a downward curve, and the atmosphere in the venue is serious."

[0077] Visual information is extracted from raw image data using a visual understanding model. Specifically, it identifies people or famous landmarks in the image and determines the scene type, such as "press conference", "outdoor protest", or "sports stadium". It also analyzes the overall color tone of the image, whether it is cool or warm, whether the composition is stable, the emotional tone it conveys, and the image quality. Finally, all the visual information extracted from each raw image data is converted into a visual semantic unit. This unit includes at least the timestamp corresponding to the image data, visual description text, a list of detected entities, scene labels, and corresponding emotional features.

[0078] It should be noted that the visual understanding model adopts a Transformer-based multi-task encoder-decoder architecture. This model includes at least the following task heads: an entity recognition head, which is represented as a DETR-based object detection head, used to locate and classify faces, landmarks, and specific objects such as microphones and trophies; a scene and activity classification head, which consists of a multi-layer perceptron, used to output the probability distribution of scene categories such as "press conference" and activity labels such as "speech" and "award ceremony"; a visual sentiment analysis head, which has a similar structure to the scene and activity classification head, used to regress the corresponding visual sentiment information; and a temporal information analysis head, which uses a lightweight 3D-CNN or temporal Transformer module, used to process the continuous frame features extracted by the encoder, and output the probability of shot transition points and global motion intensity estimates.

[0079] The visual understanding model uses a large-scale, multi-labeled visual semantic dataset as its training dataset. For example, datasets such as MS COCO, CelebA, and Google Landmarks can be used to train entity recognition capabilities, while Places365 is used for scene classification. Emotion6 or IAPS-subset is used to train visual sentiment analysis capabilities. Each image or video keyframe in the training dataset must be labeled with at least a bounding box and category, scene category, activity label, and shot transition timestamp. The training objective is to minimize a preset visual training loss function, which consists of entity recognition impairment using a combination of Focal Loss and L1 loss, labeled smoothed cross-entropy loss, Huber loss, and temporal analysis loss. The temporal analysis loss is expressed as follows: for shot transitions, binary cross-entropy loss is used; for motion intensity, mean squared error loss is used.

[0080] By calculating the semantic similarity between the visual description text and the semantic feature datasets corresponding to all original description text data, such as using Sentence-BERT vector cosine similarity to calculate semantic similarity, the semantic feature datasets and visual semantic units are aligned on the timeline. Entities identified from the image and entities extracted from the text are used as strong alignment signals for auxiliary matching. For example, if an image analysis yields a segment with the visual description "crowds gathering on the street, holding signs," and the identified entities are "street A" and "sign B," and the text describes "protesters holding a protest on street A, with sign B clearly visible," the semantic similarity between the two descriptions is calculated. It is found that the two are highly similar and the entities match perfectly, thus aligning the image segment with the text paragraph and assigning it a time position.

[0081] Successfully aligned visual semantic units are marked as fact anchors. These anchors indicate that the real video must be used during a specific time period of the video. For example, if the text mentions "xx signed xx document", and the input data happens to contain a video of xx signing a document in the office, which is successfully aligned, then the video clip will be defined as a fact anchor, and a description vector will be generated synchronously: {At time point T, the video clip ID-XXX must be used, and the content is xx signing xx document}.

[0082] For text sections that are not aligned with real images, i.e., the sections that need to be retrieved or generated, firstly, preset news narrative rules are applied to check whether the predicted narrative logic sequence is reasonable. For example, preset news narrative rules may stipulate that "background description" usually appears at the beginning, and "future outlook" will not immediately follow "conflict presentation". If the preset news narrative rules are violated, a conflict signal is generated. Each preset news narrative rule is accompanied by a basic weight set by journalism experts. When a rule violation is detected, this weight will be used as a penalty factor to punish the confidence of the relevant content. If the rule violation occurs in a key position of the news, such as the beginning, or if multiple rules are violated consecutively, the weight is increased by a preset multiplication factor, but the final value is truncated to 1.0.

[0083] Next, the pre-defined common sense knowledge graph is queried to verify the rationality of entity relationships and event descriptions. For example, if the text mentions "underwater forest fire", this part will be marked as "logical conflict" based on common sense. At this time, the logical probability of entity relationships is obtained by querying the common sense knowledge graph. If the entity relationship is explicitly marked as "impossible", such as "underwater-fire", its corresponding penalty factor is directly set to 1.0. If the co-occurrence probability p is extremely low based on statistics, such as p<0.05, the value of the penalty factor is 1-p, thereby converting the probability into the corresponding conflict intensity.

[0084] Next, it checks whether the predicted sentiment intensity matches the text content. For example, if a paragraph describing "catastrophic loss" is predicted to have "positive sentiment," it will be marked, and the text segment will be independently analyzed using sentiment analysis tools based on sentiment dictionaries or pre-trained models to obtain a sentiment baseline value S. The absolute difference |SP| between S and the model's predicted value P is divided by 2 to obtain a normalized value, which is the intensity of the conflict. This conflict intensity value is also used as the corresponding penalty factor.

[0085] When any conflict is detected, the maximum value of the three penalty factors mentioned above within the current time unit is taken as the final penalty factor value. The confidence of the relevant text content is then calibrated based on the penalty factor, and the calibrated confidence is output as an optimized feature set. For example, the model predicts a paragraph as "future outlook" with a confidence of 0.85, but the preset rule validator finds that the preceding paragraph is "background explanation" and the following paragraph is "data argument". Assuming that the probability of "future outlook" appearing in this position is extremely low in the preset narrative rules, it is determined to violate the preset news narrative rules. Furthermore, the basic weight of the preset news narrative rules is 0.8. The confidence is then calibrated based on this basic weight. After calibration, the narrative logic confidence of the paragraph is reduced to 0.85*(1-0.5*0.8)=0.51, where 0.5 is a preset modulation factor used to balance the value of the penalty factor. At the same time, it is marked as a low-confidence unit, and multiple possible narrative interpretations are considered in the future.

[0086] Finally, based on the semantic feature dataset, visual semantic units, and optimized feature set, a data structure organized along the time axis is generated. This data structure is the corresponding video intelligent synthesis dataset. Specifically, each second or each sentence corresponds to a time slice. Each time slice in the time axis contains at least the following four levels of information: Narrative semantic layer, which stores the corresponding semantic feature dataset; Entity fact layer, which stores the list of entities mentioned in that time period; Resource requirement layer, which stores the corresponding original text fragments and media requirement types. The enumerated values ​​of media requirement types include two categories: using fact anchors and needing to be explored and generated; Metadata layer, which stores global information such as the estimated total duration of the news, the overall sentiment tone, and the rhythm suggestion template.

[0087] Step S2: Based on the video intelligent synthesis dataset, perform multi-path material retrieval, and simultaneously combine the preset dynamic pruning strategy to prune the material, generating a candidate set of video synthesis materials and a keyframe requirement set.

[0088] In this embodiment, step S2 includes:

[0089] Step S2-1: Construct a candidate set of video synthesis materials.

[0090] Specifically, the video intelligent synthesis dataset is parsed to obtain deterministic demand data and exploratory demand data, wherein the deterministic demand data and exploratory demand data at least include corresponding semantic features and image data.

[0091] In one possible embodiment, the values ​​of media demand types corresponding to time slices within the resource demand layer of the video intelligent synthesis dataset are traversed. If the enumerated value of the media demand type is "use fact anchor", then the time slice is marked as a deterministic demand and its corresponding data is used as deterministic demand data; otherwise, the time slice with the enumerated value "needs to be explored and generated" is marked as an exploratory demand and its corresponding data is also used as exploratory demand data.

[0092] In this embodiment, step S2-1 includes:

[0093] Step S2-11: Based on the deterministic demand data, obtain the first candidate set of video synthesis materials.

[0094] Specifically, for data with deterministic requirements, the corresponding image data is directly adopted as video synthesis material, and the video synthesis material of all data with deterministic requirements is packaged into a first video synthesis material candidate set.

[0095] For example, suppose the video intelligent synthesis dataset parses a 30-second time slice U1 with the text description "At the opening ceremony of the 20xx Olympic Games, dazzling fireworks lit up the night sky above the Eiffel Tower." Through cross-modal alignment in step S1, it is found that the video demand data contains a 25-second video of a fireworks display filmed on-site by a reporter, and its visual description highly matches the text. Therefore, the enumeration value of the media demand type corresponding to U1 is "using fact anchors." The ID of this video clip and its precise timecode will be directly loaded into the video synthesis material candidate set without any retrieval or generation operation. Finally, the image data corresponding to all time slices with the enumeration value of "using fact anchors" for all media demand types are directly encapsulated into the first video synthesis material candidate set.

[0096] Step S2-12: Based on the exploratory demand data, obtain the second set of candidate materials for video synthesis.

[0097] Specifically, an exploratory demand unit is generated based on the exploratory demand data. Using a text-to-graph model, the semantic features corresponding to each exploratory demand unit are combined simultaneously to generate a video material query vector.

[0098] Furthermore, based on the video material query vector, a multi-path video material query is performed to obtain the original video synthesis material candidate set corresponding to the exploratory demand unit. The original video synthesis material candidate set is then subjected to quality evaluation and dynamic pruning according to a preset dynamic pruning strategy to construct a second video synthesis material candidate set.

[0099] Finally, the first and second video synthesis material candidate sets are packaged into a video synthesis material candidate set for output.

[0100] In one possible embodiment, each exploratory demand corresponds to a time slice and an exploratory demand unit, which contains all relevant data within the video intelligent synthesis dataset. For each exploratory demand unit, firstly, the calibrated confidence score of the unit's main narrative logic label is compared with a corresponding preset threshold. If the confidence score is higher than the preset threshold, the demand for the main narrative logic label interpreted from its video intelligent synthesis dataset is considered definite. In this case, only a single, clear video material query vector needs to be generated based on the semantic feature dataset corresponding to the main narrative logic label. If the confidence score is lower than the threshold, the demand for the main narrative logic label interpreted from its video intelligent synthesis dataset may be ambiguous. In this case, multiple alternative narrative type labels need to be considered in parallel, i.e., a corresponding video material query vector is generated for each alternative narrative type label. For example, for a unit describing a company's development plan, if the confidence scores of the two labels "data demonstration" and "future outlook" are similar, there may be ambiguity. Therefore, two queries will be generated simultaneously: one focusing on the visual concepts of "charts" and "data visualization" included in the plan; and the other focusing on the visual concepts of "future city" and "smiling crowd" included in the plan.

[0101] It should be noted that when constructing the video material query vector, the text-generated graph model is invoked. Based on the text description and narrative tags of the current unit, 1 to 3 visual concept sketches are quickly generated. The feature vectors of the sketches are extracted using an image encoder, and then weighted and fused with the corresponding semantic feature dataset to form the final video material query vector. For example, for the text "A technology company releases a groundbreaking product", a high-confidence "data-driven" tag will guide the generation of concept maps containing "stock market curves soaring" and "trading screens turning red". Its visual features are integrated into the query vector, making the search more inclined to find news footage containing data visualization charts.

[0102] Subsequently, based on the video material query vector, an approximate nearest neighbor search algorithm is used to perform multi-path material matching within an external real material library to obtain the K candidate video materials that are closest to the query vector in the feature space. Specifically, firstly, the cosine similarity between the query vector and the multimodal feature vectors of the segments in the material library is calculated as a basic score. Next, according to the timeliness and regional requirements of the news, the metadata such as the shooting time and location of the materials are filtered and weighted. For example, for "breaking news", materials from the most recent 24 hours are prioritized for matching, while for "local news", local media materials are prioritized for matching. That is, a reward weight coefficient is given, which is obtained by analyzing historical data, thus obtaining a comprehensive matching score. The K video materials with the highest comprehensive matching scores are selected as the corresponding candidate video materials.

[0103] Next, reverse semantic verification is performed on the K candidate video materials to obtain the corresponding candidate set of original video synthesis materials. Specifically, a visual language model is used to generate a descriptive text for each candidate video material. Then, this text is encoded into a semantic vector, and the semantic similarity between the descriptive text and the semantic vector corresponding to the original text segment in the video intelligent synthesis dataset is calculated using methods such as vector cosine similarity of Sentence-BERT, to obtain a reverse verification score. Then, the reverse verification score is compared with a preset reverse verification threshold. If the comparison result is lower than the threshold, the candidate video material is directly eliminated. For example, a candidate video material meets the query feature of "a strong reaction atmosphere", but the descriptive text generated by the visual language model shows that the content of the material is "fans are celebrating the team's victory", while the original text segment in the video intelligent synthesis dataset is "a technology company released a groundbreaking product, and the company's shareholders are in high spirits on the Internet". Since the two belong to different fields, the reverse verification score corresponding to the candidate material is extremely low, so the candidate video material will be directly eliminated. Finally, the remaining candidate video materials are used as the candidate set of original video synthesis materials.

[0104] Following reverse semantic verification, each video material query vector still contains multiple candidate video materials in its original video synthesis material candidate set. At this point, a preset dynamic pruning strategy is initiated to control the complexity of the decision space. Specifically, firstly, the average reverse verification score of the candidate video materials corresponding to all video material query vectors for each exploratory demand unit is used as the comprehensive material quality score. The N query paths with the highest comprehensive material quality scores are retained, and the remaining paths are pruned. Next, for each retained query path, the comprehensive matching score of its corresponding candidate video materials is checked to see if it exceeds the preset material adoption threshold. If it does, it means that there are usable video synthesis materials for this query path. In this case, the candidate video material with the highest comprehensive matching score among the candidate video materials that exceed the preset material adoption threshold is selected as the final retained video material for the current path. If no candidate video material exceeds the preset material adoption threshold under all retained query paths for an exploratory demand unit, it is determined that there are not enough matching materials in the current material library, and real-time synthesis is required.

[0105] Finally, the final retained video materials under the query paths of all exploratory demand units are packaged into a second video synthesis material candidate set.

[0106] Step S2-2: Generate the keyframe requirement set.

[0107] Specifically, image frames that do not meet the preset qualification conditions after dynamic pruning are defined as keyframes, and a keyframe requirement set is generated based on the semantic features corresponding to the keyframes.

[0108] In one possible embodiment, for an exploratory requirement unit determined to require real-time synthesis during the dynamic pruning process, a corresponding keyframe requirement set is generated. The keyframe requirement set includes at least the original text description and semantic feature dataset corresponding to the exploratory requirement unit, the visual concept sketch corresponding to the video material query vector, technical specification data such as the duration, resolution, frame rate, and aspect ratio of the required video clip, and feature vectors of all key entities involved in the exploratory requirement unit retrieved from a preset fact feature library, such as feature vectors corresponding to the facial features of a specific person or the precise appearance of a building.

[0109] Step S3: Generate intelligent synthesized video segments based on the keyframe requirement set, and perform quality screening on the intelligent synthesized video segments to generate a candidate video synthesis segment set.

[0110] Specifically, based on the keyframe requirement set, a text-based image diffusion model is used to generate keyframes, obtain a keyframe image set, and a latent diffusion model is used to jointly optimize the keyframe image set to generate a primary intelligent synthetic video clip.

[0111] Furthermore, the intelligent synthesized video segments are subjected to consistency verification based on a preset fact-checking mechanism to obtain the consistency verification result. If the consistency verification result is inconsistent with the facts, the parameters of the key frame generation and intelligent synthesized video segment generation stages are fine-tuned according to the consistency verification result.

[0112] It should be noted that the preset fact-checking mechanism is a loop process that includes at least agent A, agent B, and agent C working together; the consistency verification result is that the factual inconsistency means that after agents A and B verify the primary intelligent synthesized video segment, any preset key indicator fails to reach the preset threshold.

[0113] The intelligent agent A is composed of multiple neural networks and is used to perform logical consistency analysis and obtain the logical consistency analysis results. Specifically, the intelligent agent A includes at least an entity recognition module, a visual recognition module, and a scene attribute classification module. The entity recognition module is represented by a face recognition model and a landmark recognition model trained based on ArcFace loss. It is fine-tuned and trained on news face datasets such as VGGFace2-News and landmark datasets. Its training objective is to minimize the cross-entropy loss of identity classification.

[0114] The visual recognition module is represented by a Transformer-based relationship detection model and a 3D-CNN action recognition model. Both are fine-tuned and trained based on news image relationship data and news video action data. The training objectives of the two models are to minimize the cross-entropy loss of relationship and action classification, respectively.

[0115] The scene attribute classification module is represented by a ViT-based scene classification model and an attribute classification model. The scene classification model is pre-trained on Places365 and its parameters are fine-tuned based on news scene data. Similarly, both are trained using cross-entropy loss.

[0116] After a basic intelligent synthesized video clip enters agent A, its three modules perform corresponding logical consistency analysis and output a corresponding logical consistency analysis report. For example, for the generated clip "xx signed xx contract", agent A may report: In frame 25, the facial features of the person and the factual features of the person 'xx' have a matching degree of 0.75, which is lower than the preset threshold, and it is suspected to be distorted. In frames 30 to 35, no clear 'signing' action was detected, and the hand posture is classified as placement rather than writing.

[0117] The agent B is used to perform root cause analysis based on the results of the logical consistency analysis and generate a set of parameter adjustment instructions based on the results of the root cause analysis. Specifically, agent B is represented as a Transformer-based root cause classification model. This model adopts an encoder architecture. Its input is the logical consistency analysis report generated by agent A. By performing root cause analysis on the logical consistency analysis report, retrieving the corresponding instruction template from the preset rule-based instruction template library, and filling in specific parameters such as entity name, frame range, and adjustment amount, a set of parameter adjustment instructions is generated.

[0118] The root cause classification model is trained on manually labeled problem reports and root cause datasets. The training objective is to classify problems into predefined root cause categories such as "insufficient factual feature guidance", "weak dynamic action signal", "vague prompt words", and "unreasonable physical parameters". The model is trained using a cross-entropy loss function based on supervised learning. The rule-based instruction template library contains expert-written instruction templates that map each root cause category to a specific parameter adjustment template, such as "enhance the factual feature weight of entity X to Y" and "adjust the motion coefficient in time period Z".

[0119] The agent C is used to fine-tune the parameters according to the parameter adjustment instruction set. Specifically, agent C parses and executes the parameter adjustment instruction set generated by agent B. If the instruction involves the generation of keyframes, the Wensheng graph model is called again to generate new keyframes. If it involves the generation of primary intelligent synthetic video clips, the corresponding video generation or video repair tool is called for local editing. If it involves post-processing, traditional tools such as color correction and time remapping are called for video optimization, and the corrected optimized intelligent synthetic video clips are output.

[0120] Furthermore, after fine-tuning the parameters, the optimized intelligent synthesized video segments are regenerated based on the keyframe requirement set until the consistency verification result is consistent with the facts. All intelligent synthesized video segments are then packaged into a candidate video synthesis segment set for output.

[0121] In one possible implementation, firstly, the original text descriptions contained in the keyframe requirement set are converted into embedding vectors by the CLIP text encoder and injected into U-Net through a cross-attention mechanism, thereby ensuring that the final generated keyframes conform to the macro-semantics corresponding to the original text descriptions; then, the narrative type labels and sentiment intensity values ​​contained in the semantic feature dataset within the keyframe requirement set are encoded into style vectors and injected into the text-to-image diffusion model through ControlNet, thereby controlling the overall composition, tone, and light and shadow contrast of the keyframes. For example, "conflict" corresponds to a tilted, close-up composition, and "background" corresponds to a stable, wide-angle composition.

[0122] Subsequently, a cross-attention mechanism is used to inject the feature vectors of the key entities contained in the keyframe requirement set into the keyframe generation process. Next, based on image cues such as IP-Adapter, the visual concept sketches contained in the keyframe requirement set are used as soft condition inputs to guide the generated keyframes to stay on track with the initial creative direction. Finally, a set of keyframe images is output. For example, for the requirement "xx signs xx contract in the Oval Office", the generated keyframes may include: KF1 {person: xx sitting face-up, action: holding documents}, KF2 {signing close-up}, KF3 {medium shot of xx communicating with people present}. Each keyframe strictly includes the correct appearance of the person, the typical furnishings in the office, and the composition is dignified.

[0123] After generating keyframes, using the keyframes as strong visual anchors, a video diffusion model conditioned on images is adopted to generate a coherent, smooth, and dynamic video clip that matches the video intelligent synthesis dataset. Specifically, firstly, the generated keyframe sequence is encoded into the latent space of the video diffusion model and used as a constraint condition for the first and last frames or key time points of the video. This ensures that the first and last frames of the generated video clip are highly similar to the keyframes. Furthermore, reasonable interpolation and motion inference are performed while maintaining content consistency, thereby generating more reasonable intermediate frames.

[0124] Next, the semantic intensity curve S(t) contained in the semantic feature dataset within the keyframe requirement set is converted into a time-dependent motion intensity map and camera control signal. For example, at the peak of S(t) corresponding to "announcement" or "outburst", the system will enhance the motion amplitude of the scene near that frame, such as generating faster object movement, more intense camera zoom, and may trigger visual effects such as halos and particles. At the valley of S(t), the system tends to keep the scene stable or slowly pan.

[0125] Next, the last or first frame of the adjacent time slices corresponding to the current keyframe is obtained. If the adjacent time slice is real footage, its boundary frame features are extracted, and the extracted adjacent frames are used as additional condition inputs to guide the smooth transition of the start and end states of the generated video clip, such as color, lighting, and composition, with the last or first frame of the adjacent time slice. Finally, the technical specifications data in the keyframe requirement dataset are combined to generate a primary intelligent synthetic video clip.

[0126] The initial intelligent synthesized video segments are fed into a loop process where agents A, B, and C work together for iterative optimization until the outputs of agents A and B show that preset key indicators such as factual accuracy, main physical rationality, and core narrative consistency are all higher than the corresponding preset thresholds, or the preset maximum number of iterations is reached. The iterative optimization process ends, and all intelligent synthesized video segments that have undergone iterative optimization are packaged into a candidate video synthesis segment set.

[0127] Step S4: Perform intelligent video synthesis based on beam search guidance according to the candidate video synthesis segment set and the candidate video synthesis material set to generate a candidate intelligent synthesized video set.

[0128] Specifically, a narrative logic backbone is generated based on the video intelligent synthesis dataset, and the first video synthesis material candidate set contained in the video synthesis material candidate set is filled into the narrative logic backbone. Based on the candidate video synthesis segment set and the second video synthesis material candidate set contained in the video synthesis material candidate set, a beam search algorithm is run to search for video synthesis material sequences.

[0129] It should be noted that during the execution of the bundle search algorithm, scoring and sequence pruning are performed in conjunction with a preset sequence scoring function, which includes at least two dimensions: local matching degree and over-smoothness.

[0130] Furthermore, after the beam search algorithm finishes running, a candidate video synthesis material sequence set is generated, and a candidate intelligent synthesized video set is generated based on the candidate video synthesis material sequence set.

[0131] In one possible embodiment, video synthesis is modeled as a sequence decision problem, i.e., selecting a segment for each time slice from the start point to the end point to ultimately form a complete path, and running a beam search algorithm to solve the sequence decision problem. Specifically, it is defined at time steps... A state This represents a partial video sequence, specifically from unit 1 to unit 2. Each unit has selected a specific candidate segment; its state is... It includes two attributes: the cumulative score of the sequence and the path history of the selected segment.

[0132] Starting from the first time unit U1, read the video intelligent synthesis dataset, the candidate video synthesis segment set, and the candidate video synthesis material set to obtain all feasible candidate segments of U1. The candidate segments contain the fact anchors that must be used or the corresponding candidate video synthesis segments and video synthesis materials. Take each candidate segment as the starting point and create an initial state. For example, if U1 has 3 candidates, then create 3 initial states for U1.

[0133] For all currently retained "active" states, i.e., the corresponding bundles, perform an expansion operation on the bundles, that is, connect each bundle to the next unit. All feasible candidate fragments are concatenated to generate new candidate states. For example, if there are currently 5 states, If there are 4 candidates, a maximum of 20 new candidate states will be generated. Each newly generated candidate state will be scored based on a preset sequence scoring function to obtain the corresponding sequence score. After all new candidate states have been evaluated, only the M states with the highest total cumulative scores will be retained as the "active" state set to enter the next time unit. The above expansion and evaluation process will be iteratively executed.

[0134] It should be noted that the preset sequence scoring function can be expressed as:

[0135] ;

[0136] in Represented as local matching degree, i.e., the currently selected segment The corresponding comprehensive matching score, if the currently selected segment is the image data corresponding to the fact anchor point, then the value of this item is directly set to 1; This is represented as the smoothness factor, which is the difference between the last frame of the previous segment and the current segment. The cost of smooth transition is calculated based on the differences between the first frames, such as color histogram differences, SIFT feature matching, or optical flow field consistency, and this cost is normalized to the value of the over-smoothness term. and This is an adjustable weighting parameter used to balance segment quality and smoothness of transitions; its value can be determined by analyzing historical data.

[0137] After processing the last time unit, the M remaining states are the M complete video footage sequences from beginning to end. These video footage sequences represent the globally better solution under local optimal decision-making. Subsequently, the specific segment selection path for each sequence is recorded.

[0138] For example, suppose we synthesize a 3-unit short film (U1, U2, U3) with a beamwidth B=2. U1 has 2 candidates (A1, A2); U2 has 3 candidates (B1, B2, B3); and U3 has 2 candidates (C1, C2). First, starting with U1, we create states [A1] and [A2]. Then, we concatenate [A1] with B1, B2, and B3 to obtain [A1, B1], [A1, B2], and [A1, B3]. Similarly, we concatenate [A2] to obtain the corresponding 6 states. After evaluating these 6 states, assuming the highest scores are [A1,B2] and [A2,B1], these two states are retained. Next, [A1,B2] is concatenated with C1 and C2 to obtain [A1,B2,C1] and [A1,B2,C2]. [A2,B1] is concatenated in the same way. After evaluation, the two complete sequences with the highest scores are finally retained, assuming they are [A1,B2,C1] and [A2,B1,C2]. These two sequences are the candidate video synthesis material sequence set output by running the beam search algorithm.

[0139] Finally, based on the video intelligent synthesis dataset, a TTS model is used to generate corresponding intelligent audio for each sequence in the candidate video synthesis material sequence. Based on the language timestamps generated by TTS, standard format subtitle files are automatically generated. Keywords such as named entities, important verbs, and degree adverbs are extracted from the subtitle files through NLP. The semantic intensity curve S(t) in the video intelligent synthesis dataset is associated with the extracted keywords. For example, when the text corresponding to a high S(t) value is played, the subtitle of that keyword will trigger preset dynamic effects such as font enlargement and bolding, color highlighting, addition of pulse halo, or slight displacement animation, thereby generating a corresponding candidate intelligent synthesized video set for each sequence in the candidate video synthesis material sequence.

[0140] Step S5: Based on the preset video quality assessment mechanism, the candidate intelligent synthetic video set is assessed for quality, and secondary synthesis is performed based on the assessment results to generate the optimal synthetic video.

[0141] Specifically, based on a preset multidimensional video quality evaluation matrix, video quality is evaluated for each candidate intelligent synthesized video in the candidate intelligent synthesized video set, generating a multidimensional video quality score.

[0142] Understandably, the multidimensional video quality assessment matrix includes at least three dimensions: narrative rhythm alignment, multimodal synchronization, and fluency. Narrative rhythm alignment is represented by extracting the sequence of shot transition points from each video within the candidate intelligent synthetic video set using shot transition detection technology, calculating the number of shot transitions per unit time, and defining this as rhythm intensity, thereby generating an actual rhythm curve. Based on preset mapping rules, each narrative type label in the narrative logic trajectory L(t) is mapped to an ideal rhythm intensity value, for example, "background" = 0.2, "conflict" = 0.9, and "data" = 0.5, thus forming an ideal rhythm curve. The Spearman rank correlation coefficient between the actual rhythm curve and the ideal rhythm curve is calculated, and this value is defined as the narrative rhythm alignment value of the multidimensional video quality assessment matrix.

[0143] Multimodal synchronization is represented by detecting audio-visual synchronization errors in each video within the candidate intelligent synthetic video set. Specifically, lip-sync detection models such as SyncNet or TalkNet are used to detect lip-syncing characters in the video. Offset analysis is performed on the facial region and audio track to output the average absolute offset of the characters within the video. For scenes that do not require lip-sync, the semantic intensity curve S(t) is directly compared with the trigger log timestamp of the corresponding sound effect, and the root mean square error of the difference is defined as the sound effect synchronization offset. Similarly, the timestamps of subtitle appearance or disappearance are directly compared with the start or end timestamps of the corresponding speech, and the root mean square error of the difference is defined as the subtitle synchronization error value. A comprehensive error value is obtained by weighted fusion of the average absolute offset, sound effect synchronization offset, and subtitle synchronization error value. The composite error value is mapped to [0,1], where e is the composite error value. The preset tolerable error constant is used to obtain the corresponding multimodal synchronization value.

[0144] Fluency is represented by directly using the sequence average of the excessive fluency items recorded in the bundle search log in step S4 as the fluency value.

[0145] Furthermore, the values ​​corresponding to the three dimensions of narrative rhythm alignment, multimodal synchronization, and fluency are weighted and fused to obtain the multidimensional video quality score corresponding to each candidate intelligent synthesized video; deviation source tracing is performed based on the actual multidimensional video quality evaluation matrix corresponding to each candidate intelligent synthesized video to obtain the deviation source tracing results.

[0146] It should be noted that the deviation tracing means that when the score of any dimension of the actual multi-dimensional video quality evaluation matrix calculated based on the corresponding parameters of each video is lower than a preset threshold, the detailed sub-value sequence of the dimension that is lower than the preset threshold is analyzed. For example, if the multimodal synchronization is lower than the preset threshold, the specific time points in all S(t) peak points where the error exceeds the preset threshold are directly listed and defined as abnormal time points. For each abnormal time point, the metadata of the video segment used is queried. For example, if the segment at an abnormal time point has a sound effect delay, it is found through tracing the metadata that the segment is a video material matched from an external real material library, and the video material itself has an audio-visual desynchronization problem. At this time, a deviation tracing result is output: {Abnormal time point: xxx, Abnormal reason: The material itself has an audio-visual desynchronization problem}.

[0147] The multidimensional video quality score and deviation tracing results corresponding to each candidate intelligent synthesized video are uploaded to the human-computer interaction interface. The user makes secondary adjustments based on the data on the interface, generating a secondary adjustment instruction. Based on the secondary adjustment instruction, the candidate intelligent synthesized video set is synthesized again to generate the optimal synthesized video. For example, the user manually repairs the abnormal time points with deviations based on the deviation tracing results, generating repaired time slices. At this time, a slice replacement instruction is generated to "replace the corresponding abnormal time slice with the repaired time slice". The user selects the candidate intelligent synthesized video with the highest multidimensional video quality score as the final synthesis blueprint. Finally, based on the synthesis blueprint, the corresponding slice replacement instruction is executed to generate the optimal synthesized video.

[0148] This embodiment provides an electronic device, which may include: at least one processor, at least one network interface, a user interface, a memory, and at least one communication bus.

[0149] The following is a detailed introduction to the various components of the electronic device:

[0150] The communication bus can be used to enable communication between the various components mentioned above.

[0151] The user interface may include buttons, and optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0152] The network interface may include, but is not limited to, Bluetooth modules, NFC modules, Wi-Fi modules, etc.

[0153] The processor may include one or more processing cores. It connects various parts of the electronic device via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various functions and process data. Optionally, the processor can be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor can integrate one or more of the following: CPU, GPU, and modem, for example, one or more digital signal processors (DSPs) or one or more field-programmable gate arrays (FPGAs). The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0154] The memory may include RAM or ROM. Optionally, the memory may include a non-transitory computer-readable medium. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (e.g., touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor. The memory, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an evaluation application. The processor may be used to call the evaluation application stored in the memory and execute the method steps mentioned in the foregoing embodiments.

[0155] It should be noted that the above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0156] The above embodiments can be implemented, in whole or in part, through software, hardware (such as circuits), firmware, or any other combination thereof.

[0157] When implemented using software, the above embodiments can be implemented in whole or in part as a computer program product, which includes one or more computer instructions or computer programs; when the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part.

[0158] It is understood that the computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device; the computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via transmission methods such as infrared, wireless, or microwave; the computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0159] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0160] It should be understood that, in the embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video intelligent synthesis method for multimodal content conversion, characterized in that, It includes the following steps: Acquire video demand data and perform semantic parsing based on uncertainty awareness on the video demand data to generate a video intelligent synthesis dataset; The video requirement data includes at least the original descriptive text data and the original video data; Perform semantic analysis on the original descriptive text data to obtain a semantic feature dataset; Feature extraction is performed on the original image data to generate visual semantic units; Based on the semantic feature dataset and visual semantic units, semantic matching and spatiotemporal alignment are performed to obtain semantic matching results; The preset rule validator detects rule conflicts based on the semantic matching results, obtains the penalty factor corresponding to the rule conflict, calibrates the confidence of the relevant text content based on the penalty factor, and outputs the calibrated confidence as an optimized feature set. Based on the semantic feature dataset, visual semantic units, and optimized feature set, a video intelligent synthesis dataset is constructed. Based on the intelligent video synthesis dataset, multi-path material retrieval is performed, and pruning is carried out simultaneously in combination with a preset dynamic pruning strategy to generate a candidate set of video synthesis materials and a key frame requirement set. The generation of the video synthesis material candidate set is represented by parsing the video intelligent synthesis dataset to obtain deterministic demand data and exploratory demand data. The deterministic demand data and exploratory demand data at least include corresponding semantic features and image data. The acquisition of deterministic and exploratory demand data is represented by iterating through the values ​​of media demand types corresponding to time slices within the resource demand layer of the video intelligent synthesis dataset. If the enumerated value of the media demand type is "use fact anchors", then the time slice is marked as a deterministic demand and its corresponding data is used as deterministic demand data; otherwise, the time slice with the enumerated value "needs to be generated" is marked as an exploratory demand and its corresponding data is used as exploratory demand data. Based on the aforementioned deterministic demand data, a first candidate set of video synthesis materials is obtained; Based on the aforementioned exploratory demand data, a second set of candidate video synthesis materials is obtained; The first and second video synthesis material candidate sets are encapsulated into a video synthesis material candidate set and output. Image frames that do not meet the preset qualification conditions after dynamic pruning are defined as key frames, and a key frame requirement set is generated based on the semantic features corresponding to the key frames. Based on the keyframe requirement set, intelligent synthetic video segments are generated, and the quality of the intelligent synthetic video segments is screened to generate a candidate video synthetic segment set. Based on the candidate video synthesis segment set and the candidate video synthesis material set, perform intelligent video synthesis guided by beam search to generate a candidate intelligent synthesized video set; The candidate intelligent synthesized video set is evaluated based on a preset video quality assessment mechanism, and the candidate intelligent synthesized video set is then synthesized a second time according to the user-generated secondary adjustment instructions to generate the optimal synthesized video.

2. The video intelligent synthesis method for multimodal content conversion according to claim 1, characterized in that, Based on the aforementioned deterministic demand data, a first candidate set of video synthesis materials is obtained, including: For data with deterministic requirements, the corresponding image data is directly adopted as video synthesis material, and the video synthesis material of all data with deterministic requirements is packaged into the first video synthesis material candidate set.

3. The video intelligent synthesis method for multimodal content conversion according to claim 1, characterized in that, Based on the aforementioned exploratory demand data, a second candidate set of video synthesis materials is obtained, including: Based on the exploratory demand data, an exploratory demand unit is generated; By employing a text-based graph model and simultaneously combining the semantic features corresponding to each exploratory requirement unit, a video material query vector is generated. Based on the video material query vector, perform multi-path video material query to obtain the original video synthesis material candidate set for the exploratory demand unit; The original video synthesis material candidate set is subjected to quality evaluation and dynamic pruning according to a preset dynamic pruning strategy to construct a second video synthesis material candidate set.

4. The video intelligent synthesis method for multimodal content conversion according to claim 1, characterized in that, Based on the keyframe requirement set, intelligent synthesized video segments are generated, and the quality of the intelligent synthesized video segments is screened to generate a candidate video synthesis segment set, including: Based on the keyframe requirement set, a text-based image diffusion model is used to generate keyframes and obtain a keyframe image set. The keyframe image set is jointly optimized using a latent diffusion model to generate a primary intelligent synthetic video clip. The consistency verification of the primary intelligent synthesized video segment is performed based on a preset fact-checking mechanism to obtain the consistency verification result. If the consistency check result is inconsistent with the facts, the parameters of the keyframe generation and intelligent video segment generation stages will be fine-tuned based on the consistency check result. After fine-tuning the parameters, the optimized intelligent synthetic video clips are regenerated based on the key frame requirement set until the consistency check result is consistent with the facts. All intelligently synthesized video clips are packaged into a candidate video synthesis clip set for output.

5. The video intelligent synthesis method for multimodal content conversion according to claim 4, characterized in that, The pre-set fact-checking mechanism includes: The preset fact-checking mechanism is represented as a loop process that includes at least agent A, agent B, and agent C working collaboratively. The intelligent agent A is composed of multiple neural networks, which are used to perform logical consistency analysis and obtain the logical consistency analysis results. The intelligent agent B is used to perform root cause analysis based on the results of logical consistency analysis, and to generate a set of parameter adjustment instructions based on the results of the root cause analysis. The intelligent agent C is used to fine-tune the parameters according to the parameter adjustment instruction set.

6. The video intelligent synthesis method for multimodal content conversion according to claim 1, characterized in that, Based on the candidate video synthesis segment set and the candidate video synthesis material set, beam search-guided intelligent video synthesis is performed to generate a candidate intelligent synthesized video set, including: A narrative logic backbone is generated based on the video intelligent synthesis dataset, and the first video synthesis material candidate set contained in the video synthesis material candidate set is filled into the narrative logic backbone. Based on the candidate video synthesis segment set and the second video synthesis material candidate set contained in the video synthesis material candidate set, the beam search algorithm is run to search for the video synthesis material sequence. During the execution of the beam search algorithm, scoring and sequence pruning are performed in conjunction with a preset sequence scoring function. The preset sequence scoring function includes at least two dimensions: local matching degree and excessive fluency. After the beam search algorithm finishes running, a candidate video synthesis material sequence set is generated, and a candidate intelligent synthesized video set is generated based on the candidate video synthesis material sequence set.

7. The video intelligent synthesis method for multimodal content conversion according to claim 1, characterized in that, The candidate intelligent synthesized video set is evaluated based on a preset video quality assessment mechanism, and then a secondary synthesis is performed on the candidate intelligent synthesized video set according to the user-generated secondary adjustment instructions to generate the optimal synthesized video, including: The generation of the optimal synthesized video is represented by evaluating the video quality of each candidate intelligent synthesized video in the candidate intelligent synthesized video set based on a preset multi-dimensional video quality evaluation matrix, and generating a multi-dimensional video quality score. Based on the actual multi-dimensional video quality evaluation matrix corresponding to each candidate intelligent synthesized video, the source of deviation is traced to obtain the source of deviation; The multi-dimensional video quality score and deviation tracing results corresponding to each candidate intelligent synthetic video are uploaded to the human-computer interaction interface. Users can make secondary adjustments based on the data on the interface and generate secondary adjustment instructions. The candidate intelligent synthetic video set is synthesized a second time based on the secondary adjustment instruction to generate the optimal synthetic video.

Images