Conversational multimedia content generation method and system

Abstract

The invention provides a dialogue multimedia content generation method and system, and the method comprises the steps: carrying out the semantic understanding and intention analysis of a standard text through a pre-trained large language model, so as to judge whether an input instruction is a retrieval instruction or an editing instruction; if the input instruction is a retrieval instruction, inputting the text unit into a preset video retrieval model, so that the video retrieval model feeds back a target video matched with the text unit; and if the input instruction is an editing instruction, performing similarity matching on the standard text to obtain an editing mode, and sequentially performing editing processing of video matting, video reconciling, video segmentation, video restoration, audio generation, mouth shape generation and the like on a to-be-edited video pre-selected by a user by adopting the editing mode to obtain multimedia generation content. Therefore, the multimedia content editing becomes intelligent, the threshold of multimedia editing can be greatly reduced, and more convenient and efficient editing experience is provided for users.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and more particularly to a method and system for generating multimedia content. Background Art

[0002] Existing media editing platforms or software have the problem of being object-oriented and monotonous. They tend to only provide editing functions for a single media type, that is, one of the media types such as video, audio or pictures, but are unable to effectively integrate different types of media materials together. This leads to limited diversity of editing content, difficulty in creating cross-media content, and limits users' creative expression.

[0003] Secondly, the editing service functions provided by existing platforms are relatively limited. For example, the video editing platform mainly provides editing services such as video splicing, playback settings, special effects settings, etc.; the image editing platform mainly provides beautification services such as filter settings and portrait beauty; the audio editing platform mainly provides audio extraction, merging and other services. The above services cannot meet users' complex needs for video / image content and audio production.

[0004] At the same time, the existing platforms and software usually have complex functional modules to use. Beginners need to spend a lot of time learning and familiarizing themselves with the operation. The learning curve is steep and it is difficult to get started quickly, and the threshold for use is high. Operations usually need to be completed manually by users. This manual operation method not only increases the user's operating burden, but also limits the flexibility and efficiency of editing. Users need to edit frame by frame and adjust parameters repeatedly, which is not only time-consuming and labor-intensive, but also prone to operational errors and unstable editing.

[0005] Therefore, there is an urgent need for a conversational multimedia content generation method and system that can realize the retrieval of text to multimedia materials and the call of text to the AIGC tool library, and realize the fusion editing of multimodal content, so that multimedia content editing becomes intelligent, greatly lowers the threshold of multimedia editing, and provides users with a more convenient and efficient editing experience. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a method and system for generating conversational multimedia content, so as to solve the problems that the common functional modules of existing platforms and software are relatively complicated to use, beginners need to spend a lot of time learning and familiarizing themselves with the operation to make or change multimedia content, the learning curve is steep and it is difficult to get started quickly, and the threshold for use is high.

[0007] The present invention provides a method for generating interactive multimedia content, which comprises:

[0008] Convert the input instructions in the front-end user dialog box into standard text through the background;

[0009] The standard text is semantically understood and analyzed through a pre-trained large language model to determine whether the input instruction is a search instruction or an edit instruction; wherein,

[0010] If the input instruction is a search instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text unit;

[0011] If the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, and the editing mode is used to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

[0012] Preferably, the input instruction in the front user dialog box is converted into a standard text by the background, including:

[0013] Capturing input instructions in the front-end user dialog box through a preset front-end application program;

[0014] Preprocessing the input instructions to obtain key instructions;

[0015] The key instructions are converted into standard text in a standard format, and the standard text is transmitted to a back-end server via a network protocol based on SSL or TLS encryption technology; wherein the standard format is JSON format or XML format.

[0016] Preferably, the process of pre-training a large language model includes:

[0017] Build a basic language model based on the Transformer architecture;

[0018] Pre-training the language basic model through a pre-collected text set to form an initial language model with a basic understanding of the language;

[0019] The initial language model is repeatedly fine-tuned and refined to form a large language model.

[0020] Preferably, the process of pre-training the language base model includes:

[0021] Convert the input text in the given text set into a text sequence, convert the vocabulary in the text sequence into a vector representation, and retain the position information of each vocabulary in the sequence through position encoding; obtain the optimized word vector x=x1...x in the text sequence based on the window size of the language basic model and the historical words about each vocabulary in the pre-training process n The maximum likelihood estimate L pt :

[0022] L pt (x) = ∑ i logP(x i |x i-k , ..., x i-1 ;θ)

[0023] Where k represents the language model window size; θ represents the parameters of the neural network model; P represents the probability distribution function; x = x1...x n Represents a text sequence of length n;

[0024] The process of fine-tuning and refining the initial language model includes:

[0025] Input the text sequence into the initial language model, and extract the hidden layer output corresponding to the last word in the last layer of the initial language model The predicted label is obtained by transforming the hidden layer output through a preset fully connected layer; the cross entropy loss function is applied to calculate the difference between the predicted label and the actual label until the cross entropy loss function reaches a preset cross entropy threshold, then fine-tuning is stopped; wherein the cross entropy loss function is:

[0026] L(y|x1,...,x n )=softmax(h [L] W y ), where W y represents the word vector matrix; y represents the actual label of the text sequence;

[0027] The downstream tasks of the initial language model are fine-tuned by optimizing the target optimization loss function, minimizing the target optimization loss function, and stopping the fine-tuning until the target optimization loss function is less than a preset optimization threshold; wherein the target optimization loss function is:

[0028] L FT (C) = ∑logP(y|x1, ..., x n );

[0029] Where C represents the labeled data of the downstream task; x = x1...x nrepresents a text sequence of length n; y represents the actual label of the text sequence.

[0030] Preferably, the process of performing 5W1H analysis on the standard text to obtain text units, and inputting the text units into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text units includes:

[0031] Performing 5W1H analysis on the standard text to generate a 5W1H structured text template;

[0032] Inputting the 5W1H structured text template into a preset video retrieval model, and comparing the 5W1H structured text template with the video material metadata in the video library of the video retrieval model to obtain a preset data amount of pending videos;

[0033] The similarity comparison module in the video retrieval model compares the similarities between the preset number of pending videos and the 5W1H structured text template to obtain similarity values;

[0034] The pending videos corresponding to the similarity values ​​are arranged from large to small according to the similarity values, and the top-ranked pending videos are used as a preset number of target videos.

[0035] Preferably, the process of obtaining similarity values ​​by comparing the similarities between the preset number of pending videos and the 5W1H structured text template respectively through the similarity comparison module in the video retrieval model includes:

[0036] Adjusting the frame image of the pending video to a preset size so as to be input into the similarity comparison module as a standard video;

[0037] The TimeSformer layer in the similarity comparison module performs feature extraction on the standard video to obtain the image feature V of each frame image in the standard video. sst , aggregate the image features to obtain a visual embedding sequence The BERT model of the Transformer in the similarity comparison module performs feature extraction on the 5W1H structured text template to obtain text feature L SST , and add position information to the text features to obtain a text embedding sequence

[0038] A similarity value is obtained based on the visual embedding sequence and the text embedding sequence; wherein the similarity value is:

[0039]

[0040] Among them, gv (·) and g t (·) is the default linear projection.

[0041] Preferably, if the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, including:

[0042] Conducting 5W1H analysis on the standard text to obtain text units;

[0043] Performing feature extraction on the text unit to obtain comparative text features;

[0044] The comparison text feature is compared with the edit text feature in the comparison library for similarity to obtain the edit text with the highest similarity value ranking; and the edit mode corresponding to the edit text with the highest similarity value ranking is obtained.

[0045] Preferably, if the editing mode is video matting, the video to be edited is sent to a preset video matting module, so that the video matting module performs video matting; wherein,

[0046] The video matting module includes an encoder module, a loop decoder module, a depth-guided filter module and a segmentation module;

[0047] A feature extraction operation is performed on a single video frame of the video to be edited by the encoder module to obtain multi-scale features; wherein the encoder module uses MobileNetV3-Large as a backbone network, and uses an LR-ASPP module behind the backbone network for a semantic segmentation task; and a dilated convolution without a downsampling step is used in the last layer of MobileNetV3;

[0048] The recursive decoder module adopts a loop mechanism to autonomously learn the video to be edited based on the multi-scale features to obtain effective information, so as to determine the matting information and the time coherence information, and obtains the hidden features of the video to be edited through a preset decoder;

[0049] Performing high-resolution upsampling on the video to be edited based on the hidden features and the multi-scale features by the depth-guided filter module to obtain high- and low-resolution sampling frames;

[0050] The segmentation module generates high-resolution transparency and foreground based on the high- and low-resolution sampling frames, and segmentation is performed according to the high-resolution transparency and foreground to obtain a matte video; wherein the matte video includes a single-channel transparency mask prediction video, a three-channel foreground prediction video and a single-channel segmentation prediction video.

[0051] Preferably, if the editing mode is lip generation, the video to be edited is sent to a pre-trained lip matching module, so that the lip matching module matches the lip shape of the audio in the video to be edited based on the trained SyncNet model and residual jump connection; wherein,

[0052] The synchronization probability of an audio-video pair is evaluated by computing the dot product between the matched lip shape and speech embeddings:

[0053]

[0054] Among them, v and s represent the lip-sync video in the video to be edited and the speech embedding vector in the video to be edited respectively, ∈ is a preset parameter and is a positive number; the output value of the synchronization probability is between [0,1], indicating the probability of synchronization between the audio and video in the video to be edited.

[0055] The present invention also provides a conversational multimedia content generation system, which implements the conversational multimedia content generation method as described above, comprising:

[0056] The front user dialog box is used to receive the input instructions input by the user and enable the background to convert the input instructions into standard text;

[0057] A large language model, used for performing semantic understanding and intent analysis on the standard text to determine whether the input instruction is a search instruction or an edit instruction;

[0058] A video retrieval model, for performing a 5W1H analysis on the standard text to obtain a text unit if the input instruction is a retrieval instruction, and feeding back a target video that matches the text unit;

[0059] The video editing model is used to perform similarity matching on the standard text to obtain an editing mode if the input instruction is an editing instruction, and use the editing mode to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

[0060] It can be seen from the above technical scheme that the conversational multimedia content generation method provided by the present invention first pre-places the input instruction input value in the user dialog box, so that the background converts the input instruction into a standard text, and then uses a pre-trained large language model to perform semantic understanding and intent analysis on the standard text to determine whether the input instruction is a retrieval instruction or an editing instruction; if the input instruction is a retrieval instruction, a 5W1 H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model, so that the video retrieval model feeds back a target video that is compatible with the text unit; if the input instruction is an editing instruction, a similarity match is performed on the standard text to obtain an editing mode, and the editing mode is used to perform video matting, video blending, video segmentation, video repair, audio generation, lip shape generation and other types of editing processing on the video to be edited pre-selected by the user in turn to obtain multimedia generated content. In this way, based on an advanced modern large language model, through 5W1 H analysis, a 5W1 H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model, so that the video retrieval model feeds back a target video that is compatible with the text unit; if the input instruction is an editing instruction, a similarity match is performed on the standard text to obtain an editing mode, and the editing mode is used to perform video matting, video blending, video segmentation, video repair, audio generation, lip shape generation and other types of editing processing on the video to be edited pre-selected by the user in turn to obtain multimedia generated content. H factor extraction technology can accurately extract semantics and analyze the intent of user input commands, so that users only need to enter text commands to achieve corresponding retrieval or editing needs. At the same time, it introduces the most advanced machine learning and deep learning technologies, and trains models to identify key multi-dimensional information in each modal material, thereby realizing the retrieval of text to multimedia materials and the call of text to tool libraries, and realizing the fusion editing of multimodal content. This method and system use multiple rounds of iterative operations in the form of dialogue, making multimedia content editing intelligent, greatly reducing the threshold of multimedia editing, and providing users with a more convenient and efficient editing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings:

[0062] Figure 1 is a flow chart of a method for generating conversational multimedia content according to an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of executing a method for generating conversational multimedia content according to an embodiment of the present invention;

[0064] Figure 3 4 is a system block diagram of a conversational multimedia content generation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The existing platforms and software usually have complex function modules. Beginners need to spend a lot of time to learn and familiarize themselves with the operation. The learning curve is steep and it is difficult to get started quickly, and the threshold for use is high. Operations usually need to be completed manually by users. This manual operation method not only increases the user's operating burden, but also limits the flexibility and efficiency of editing. Users need to edit frame by frame and adjust parameters repeatedly, which is not only time-consuming and labor-intensive, but also prone to operational errors and unstable editing.

[0066] In view of the above problems, the present invention provides a method and system for generating interactive multimedia content. The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] In order to illustrate the method and system for generating interactive multimedia content provided by the present invention, Figure 1-Figure 3 The embodiments of the present invention are exemplarily indicated.

[0068] The following description of exemplary embodiments is merely illustrative in nature and is not intended to limit the present invention and its application or use. Technologies and devices known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technologies and devices should be considered as part of the specification.

[0069] like Figure 1 , Figure 2 As shown in the figure, the present invention provides a method for generating interactive multimedia content, comprising:

[0070] S1: Convert the input instructions in the front-end user dialog box into standard text through the background;

[0071] S2: Perform semantic understanding and intent analysis on the standard text through a pre-trained large language model to determine whether the input instruction is a search instruction or an edit instruction; wherein,

[0072] A: If the input instruction is a search instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text unit;

[0073] B: If the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, and the editing mode is used to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

[0074] In short, it is to use the powerful natural language processing capabilities of the large language model to realize the 5W1H element extraction, precise semantic understanding and intent analysis of user input commands, and complete the retrieval or editing needs of materials. The system architecture consists of three main functions: command analysis, multimedia resource retrieval, and multimedia resource editing. The command analysis function is responsible for parsing the text input by the user, and using the large language model to understand whether the user's needs are resource retrieval or resource editing. If the input command is for resource retrieval, the input text is further analyzed by 5W1H. If the input command is for resource editing, the editing method analysis is performed. Multimedia resource retrieval is responsible for calling the retrieval algorithm, matching the 5W1H elements identified and extracted by the large language model in the multimedia resource library, and retrieving the most suitable resources to feedback to the user; multimedia resource editing is responsible for calling the corresponding algorithm in the AI ​​tool library and processing multimedia resources according to the editing method analyzed by the large language model.

[0075] In this embodiment, the input instruction is input into the front user dialog box, and the background converts the input instruction into a standard text, including:

[0076] S11: Capturing input instructions in the front-end user dialog box through a preset front-end application;

[0077] S12: Preprocessing the input instruction to obtain key instructions;

[0078] S13: Convert the key instructions into a standard text in a standard format, and transmit the standard text to a backend server through a network protocol based on SSL or TLS encryption technology; wherein the standard format is JSON format or XML format.

[0079] Specifically, users enter questions or instructions through the front-end user dialog box of the front-end interface. These text inputs are captured by the front-end application, pre-processed and converted into standard text formats, such as JSON or XML. Then, the text data is transmitted to the back-end server through the network protocol. During the process, SSL / TLS encryption technology is used to protect the security and integrity of the data. After the back-end server receives the text data, it processes the request through a series of middleware and services. The user's needs are passed to the large language model to parse the text and extract the intent and key information.

[0080] Step S2 is a process of performing semantic understanding and intent analysis on the standard text through a pre-trained large language model to determine whether the input instruction is a retrieval instruction or an editing instruction.

[0081] In this embodiment, the large language model is pre-trained in advance, wherein the process of pre-training the large language model includes:

[0082] S21: Build a basic language model based on the Transformer architecture;

[0083] S22: pre-training the language basic model using a pre-collected text set to form an initial language model with a basic understanding of the language;

[0084] S23: repeatedly fine-tuning and refining the initial language model to form a large language model.

[0085] Specifically, the process of pre-training the basic language model in step S22 includes:

[0086] S221: Convert the given input text in the text set into a text sequence, convert the vocabulary in the text sequence into a vector representation, and retain the position information of each vocabulary in the sequence through position encoding; obtain the optimized word vector x=x1...x in the text sequence based on the window size of the language basic model and the historical words about each vocabulary in the pre-training process n The maximum likelihood estimate L pt :

[0087] L pt (x) = ∑ i logP(x i |x i-k , ..., x i-1 ;θ)

[0088] Where k represents the language model window size; θ represents the parameters of the neural network model; P represents the probability distribution function; x = x1...x n Represents a text sequence of length n;

[0089] The process of fine-tuning and fine-tuning the initial language model in step S23 includes:

[0090] S231: Input the text sequence into the initial language model, and extract the hidden layer output corresponding to the last word of the last layer in the initial language model The predicted label is obtained by transforming the hidden layer output through a preset fully connected layer; the cross entropy loss function is applied to calculate the difference between the predicted label and the actual label until the cross entropy loss function reaches a preset cross entropy threshold, then fine-tuning is stopped; wherein the cross entropy loss function is:

[0091] L(y|x1,...,x n )=softmax(h [L] W y ), where W y represents the word vector matrix; y represents the actual label of the text sequence;

[0092] S232: fine-tuning the downstream tasks of the initial language model by optimizing the target optimization loss function, minimizing the target optimization loss function, and stopping the fine-tuning until the target optimization loss function is less than a preset optimization threshold; wherein the target optimization loss function is:

[0093] L FT (C) = ∑logP(y|x1, ..., x n );

[0094] Where C represents the labeled data of the downstream task; x = x1...x n represents a text sequence of length n; y represents the actual label of the text sequence.

[0095] In a specific embodiment, the architecture of the instruction analysis module includes three submodules: instruction classification, 5W1H analysis and editing method analysis. The result of instruction classification determines the next analysis step. When performing instruction analysis, the large language model is first used to perform semantic understanding and intent analysis on the user input instruction. The main purpose is to determine whether the user needs to perform a retrieval operation or an editing operation. If the judgment result is to perform a retrieval operation, the 5W1H elements in the user instruction are identified and extracted through the large language model, namely: Who, What, When, Where, Why and How, and provided to the retrieval algorithm as a retrieval reference; if it is an editing operation, the user's needs are matched with the editing algorithm in the AI ​​tool library for similarity, and the editing method that is most likely to meet the user's requirements is determined.

[0096] Specifically, in this specific embodiment, a large language model based on the Transformer architecture is used. In order to capture the wide distribution of implicit tasks in the text and enhance the versatility and transfer learning ability of the model, it is necessary to use a large amount of text as a text set for pre-training to provide a basic understanding of the language. In the pre-training stage, for a given text sequence, each word contained in it is converted into a vector representation, and their position information in the sequence is retained through position encoding. The word vector x=x1...x is optimized using the conventional language modeling method. n The maximum likelihood estimate L pt :

[0097] L pt (x) = ∑ i logP(x i |x i-k , ..., x i-1 ;θ)

[0098] L pt(x) provides the model with an understanding of the basic structure and semantics of the language. pt (pre-training) represents the function used in the pre-training stage, which enables the model to have good initial performance when facing different tasks. Pre-training improves the few-shot performance of the model, so that when facing new tasks, even if only a small amount of task-specific data is provided, the model can also perform reasoning and generation better.

[0099] Here, k represents the language model window size, and the context window refers to the size range of the previous token or text fragment considered by the language model when predicting or generating text. The two k here actually represent the same meaning, that is, based on k historical words x i-k …x i-1 Predict the word x at the current moment i ; θ represents the parameters of the neural network model; P represents the probability distribution function; x=x1...x n Represents a text sequence of length n.

[0100] In the fine-tuning stage, in this specific embodiment, the pre-trained model is used to further optimize its performance on a specific task. If the labeled data is C, the input of each sample is x=x1...x n The length of the text sequence is n, and the corresponding label is y, W y The word vector matrix is ​​used to input the text sequence into the pre-trained model, and the hidden layer output corresponding to the last word in the last layer is extracted. The hidden layer output is then transformed through a fully connected layer to predict the final label, and a cross entropy loss function is applied to calculate the difference between the model prediction and the actual label:

[0101] L(y|x1,...,x n )=softmax(h [L] W y ), where L also represents the probability distribution function, which is used to predict the probability of label y appearing when a given input text sequence x1,…,xn is given.

[0102] And fine-tune the downstream tasks by optimizing the following loss function (target optimization loss function):

[0103] L FT (C) = ∑logP(y|x1, ..., x n )

[0104] L ft(fine-tuning) represents the function used in the fine-tuning phase (target optimization loss function); by further optimizing the model parameters, the model can be more accurate and effective in specific downstream tasks. FT (C) The model is able to learn task-specific features, thereby improving the generation quality and accuracy on that task.

[0105] In this way, through the large language model, the system will determine whether the user input instruction is a search instruction or an editing instruction. If the judgment result is a search instruction, it will enter the multimedia resource retrieval module. The retrieval algorithm will combine the 5W1H information in the user input instruction analyzed by the large language model, and generate a structured text template based on 5W1H on this basis to enrich the semantic information, and match the extracted and processed information with the metadata of the media material content (including title, description, tags, etc.) to achieve the purpose of retrieval. In terms of technical implementation, the algorithm encodes the 5W1H text information and calculates the similarity with the visual encoding of the video in the database, and finally outputs the video content with the highest similarity as the search result to the user. This process ensures the efficiency and accuracy of the retrieval and meets the complex needs of users for multimedia material retrieval.

[0106] In case A, if the input instruction is a search instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text unit.

[0107] The process of performing 5W1H analysis on the standard text to obtain a text unit, and inputting the text unit into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text unit includes:

[0108] SA1: Perform 5W1H analysis on the standard text to generate a 5W1H structured text template;

[0109] SA2: Input the 5W1H structured text template into a preset video retrieval model, and compare the 5W1H structured text template with the video material metadata in the video library of the video retrieval model to obtain a preset amount of pending videos;

[0110] SA3: using a similarity comparison module in the video retrieval model to compare the similarities between the preset number of pending videos and the 5W1H structured text template to obtain similarity values;

[0111] SA4: Arrange the pending videos corresponding to the similarity values ​​from large to small according to the similarity values, and use the top-ranked pending videos as a preset number of target videos.

[0112] In a specific embodiment, in order to process visual information, the frame image sampled in the video is resized to a preset size. In this specific embodiment, the preset size is 224, and it is used as input. A 12-layer TimeSformer-224 is used to extract visual features. Similar to ViT, TimeSformer divides each image into N non-overlapping image blocks, and introduces learnable position encoding in each image block to retain the position information of each area in the image. After the image block is flattened, a series of image block tokens containing the feature information of the image and the position embedding are generated through a linear projection layer. Next, through the self-attention mechanism of Transformer, the similarity comparison module in the video retrieval model can capture important visual information in the frame image. In TimeSformer, special attention is paid to performing the self-attention mechanism along the time dimension. After the above operations, the features of each image are obtained. Where d is the feature dimension; finally, mean pooling is used to aggregate all image features into visual features as input; finally, we get a series of visual embeddings: in Represents the embedding of the video [CLS] token; for the feature extraction of text content, the BERT model based on a 6-layer Transformer is used to obtain the text features in the 5W1H information. The template-constructed body-state text is input into BERT. Each layer has a self-attention mechanism and a forward neural network. The output of these layers is an embedding sequence containing the given K embedding vectors. Similar to the video encoder, BERT also needs to take into account the position information of the tokens in the input text. For this purpose, position embedding is added to the embedding vector of the text token to help the model understand the positional relationship of the token in the text sequence. The final encoder outputs the embedding sequence in d is the feature dimension, [CLS] token for embedded text.

[0113] Through the above processing, we get the visual and text [CLS] tags, which are called visual embedding sequence and text embedding sequence Then the similarity function between vision and text can be expressed as:

[0114]

[0115] where g v (·) and gt (·) is a linear projection, which maps [CLS] embedding to a low-dimensional space (e.g., 256-d) under a unified standard. Based on this formula, the similarity score between text and video can be obtained.

[0116] The system returns a list of videos with the highest similarity scores based on the calculation results. In this specific embodiment, the number of videos returned is 6. If satisfied, the user directly selects the video he needs by clicking, and the system returns the video selected by the user; if not satisfied, the user can enter "unsatisfied, change a batch", and the system will search again and repeat the above steps SA1-SA4 until the user is satisfied.

[0117] In addition, after the user selects the video they need, they can enter the video editing command. If the large language model determines that the result is an editing command, the multimedia resource editing module will be entered; or the user can directly upload a short video to be edited, and then directly edit it by entering the command.

[0118] The specific editing instruction process is as described in Part B. If the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, and the editing mode is used to edit the user's pre-selected videos to be edited in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

[0119] In one embodiment, if the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, including:

[0120] Conducting 5W1H analysis on the standard text to obtain text units;

[0121] Performing feature extraction on the text unit to obtain comparative text features;

[0122] The comparison text feature is compared with the edit text feature in the comparison library for similarity to obtain the edit text with the highest similarity value ranking; and the edit mode corresponding to the edit text with the highest similarity value ranking is obtained.

[0123] In a more specific embodiment, if a video matting operation is performed, a video matting module in a preset tool library is used to separate the person in the video screen from the background and place him naturally in other environments. This part of the architecture mainly includes an encoder module, a loop decoder module, and a depth-guided filter module.

[0124] The encoder module is based on the most advanced semantic segmentation network, aiming to accurately separate human bodies in videos. The module uses MobileNetV3-Large as the backbone network, followed by the LR-ASPP module for semantic segmentation tasks. The last layer of MobileNetV3 uses a dilated convolution without downsampling step. The encoder module operates on a single video frame and extracts features at different scales for use by the recursive decoder.

[0125] Recursive decoder module, the loop mechanism can autonomously learn what information to retain and forget in the continuous video stream, improving the quality of matting and temporal coherence. This module uses a multi-scale ConvGRU to effectively aggregate temporal information. The form of ConvGRU is defined as:

[0126] z t =σ(w zx *x t +w zh *h t-1 +b z )

[0127] r t =σ(w rx *x t +w rh *h t-1 +b r )

[0128] o t =tanh(w ox *x t +w oh *(r t ⊙h t-1 )+b o )

[0129] h t =z t ⊙h t-1 +(1-z t )⊙o t

[0130] The operators * and ⊙ represent convolution and element-wise product respectively; tanh and σ represent the hyperbolic tangent function and sigmoid function respectively. w and b are the convolution kernel and bias term. The hidden state h t is used as the output and loop state for the next time step, i.e. h t-1 The initial loop state h0 is an all-zero tensor.

[0131] The decoder consists of a bottleneck block, an upsampling block, and an output block. The bottleneck block is located after the LR-ASPP module and operates at a feature scale of 1 / 16. Through splitting and connection, a ConvGRU layer only operates on half of the channels. The upsampling block is repeatedly performed at feature scales of 1 / 8, 1 / 4, and 1 / 2. It merges the upsampled output of the previous step, the feature map of the encoder at the corresponding scale, and the input image repeatedly reduced by 2×2 average pooling, and then uses batch normalization and ReLU activation after convolution for feature merging and channel reduction. The final output block optimizes the result through regular convolution, which merges the input image and the bilinear upsampled output of the previous module. Then, two repeated convolutions, batch normalization, and ReLU activation are used to generate the final hidden features.

[0132] The deep guided filter module is used for high-resolution upsampling. When processing high-resolution videos such as 4K and high-definition (HD), we downsample the input frame by a scale factor s and then pass it through the encoder-decoder network. Next, the low-resolution transparency, foreground, final hidden features, and high-resolution input frames are provided to the DGF module to generate high-resolution transparency and foreground. The entire network is trained in an end-to-end training manner. If the video resolution to be processed is low, the encoder-decoder network can run independently. Input a single video frame or a video sequence (usually an RGB image), the output includes a single-channel transparency mask prediction, a three-channel foreground prediction, and a single-channel segmentation prediction, where the segmentation output is used to train and evaluate the accuracy of the model.

[0133] In this specific embodiment, if the editing mode of video blending is processed, a pre-set video blending model is used to blend the video, that is, to adjust the color, brightness, contrast and other parameters in the synthesized video to improve the video quality and achieve a more realistic visual effect. For example, a blending model based on a white box method can be used, which combines a neural network for filter parameter prediction and a white box filter for image coordination. The model mainly consists of a backbone encoder ε (EfficientNet-B0), a regressor R and a set of image filters F = {F1, ..., Fk}. These filters are based on the processing methods that statistical engineers will use when color grading, such as brightness, illumination, color, etc. If a video to be edited is given, the video to be edited is first split into composite images I′0 and a foreground mask M corresponding to each composite image. First, each composite image is downsampled to a resolution of 256×256 and input into ε to extract image features Z (containing 160 channels):

[0134] Z=ε(I′0,M)

[0135] Then Z is processed globally and the filter parameters θ are derived from it using R. Considering the possible correlation between filters, we introduce a cascade regressor to regress the parameters θ i conditioned using the feature vector of the previous filter parameters:

[0136] θ={θ1,...,θ k =R(Z)},θ i ∈[-1, 1], i=1,...k.

[0137] θ1=R1(Z),

[0138] θ i =R i (Z|θ i-1 )=R i (Z|θ i- 1, ..., θ1)

[0139] Using parameters θ, the filter performs k filters on I′0 in sequence:

[0140] I′ i =F i (I′ i-1 θ i ), i = 1, ..., k.

[0141] Finally, the harmonized image I′ is obtained:

[0142] I′=MI′ k +(1-M)I′0

[0143] That is, the composite image I′0 and the foreground mask M are input, and the blended image I′ is output, and then the blended images are combined into a blended video.

[0144] If it is the video segmentation editing mode, the pre-trained video segmentation model is used to accurately segment the specific object in the video sequence of the video to be edited from the background. The video segmentation model mainly adopts the language-driven semantic segmentation method. When pre-training the video segmentation model, a large amount of training is performed on visual data. First, the text encoder embeds N possible label sets into a continuous vector space R c In the process, N vectors T1...T are generated. n ∈R c As output, the number N varies freely, and the set of output vectors is invariant to the order of the input labels.

[0145] The image encoder uses a dense prediction transformer (DPT) as the underlying architecture to generate an embedding vector for each input pixel. Assume that H×W is the input image size and s is a user-defined downsampling factor (s=2 in our implementation). We define:

[0146]

[0147] Its output is a dense embedding Due to memory constraints, the image encoder predicts pixel embeddings at a lower resolution than the input image resolution. We use an additional post-processing module that spatially regularizes the predictions and restores the predictions at the original resolution using bilinear interpolation via two functions that achieve this property: DepthwiseBlock and BottleneckBlock.

[0148] After embedding the image and label, we associate them through the inner product to create a s ×W s ×N tensor, defined as:

[0149] f ij =I ij ·T k

[0150] Among them I ij is the embedding of pixel (i, j), f ij is the inner product of the embedding of pixel (i, j) and the kth word.

[0151] The N-dimensional vector of the inner product of the embedding of pixel (i, j) and all N words is F ij =(f ij1 , f ij2 , ..., f ijk ) T , define a pixelated softmax target:

[0152]

[0153] Among them, t is a custom temperature parameter, which is set to 0.07. After the normalization of the softmax function, the inner product of each pixel's embedding and all labels becomes a probability distribution, that is, the matching degree of each pixel in all labels can be represented by a probability, which is obtained by maximizing the ground truth label k=y of pixel i, j. ij The corresponding entry f ijk The goal of the dot product is to assign each pixel to the label that is most relevant to it, and the temperature parameter is used to control the smoothness of the probability distribution.

[0154] If the editing mode is video restoration, the pre-trained video restoration model is used to fill in the "damaged" area in the segmented video screen and make the reasonably coherent content run through the entire video. The temporal and spatial information is used, combined with the internal image and context to ensure that the restored video sequence remains visually coherent and natural. In this specific embodiment, an end-to-end video restoration framework based on optical flow guidance can be used as the framework of the video restoration model. That is, given a damaged video sequence {X t ∈R H×W×3 |t=1,...,T}, sequence length T and corresponding frame binary mask {M t ∈R H×W×1 |t=1,...,T}. First, before optical flow prediction, a context encoder is used to encode the original corrupted frame X at 1 / 4 resolution. t Downsampling is performed, and the downsampling frame is recorded as The optical flow prediction between adjacent frames i and j is calculated by the optical flow estimation network F:

[0155]

[0156] Use the L1 loss function to restore bidirectional flow:

[0157]

[0158] F t→t+1 and F t→t-1 are the ground truth forward flow and backward flow, respectively, computed from the original uncorrupted video.

[0159] is the local temporal neighborhood feature extracted from the context encoder, where T l Represents the length of the local neighborhood frame. Using the completed optical flow Guide the feature propagation from frame t to frame t+1:

[0160]

[0161] Where W(·) represents the spatial deformation operation based on optical flow, is the backward propagation feature of the tth time step, and the propagation function P b (·) represents two convolutional layers with LeakyReLU activation.

[0162] Modulated deformable convolution is used to further index and weight the candidate feature points and calculate the weight mask W t→t+1 and offset △F t→t+1 :

[0163]

[0164] Use deformable convolutional layers to deform backward features And update the back-propagation features

[0165] A multi-layer temporal focal transformer is used to achieve content generation by combining the propagated local domain features with the non-local reference features. Finally, a decoder is used to amplify the padded features and reconstruct them into the final video sequence {Y t ∈R H×W×3 |t=1,...,T}. By combining the reconstructed video frames, the final restored video is obtained.

[0166] If an audio generation editing operation is performed, a pre-trained audio generation model can be used to convert text into audio with specific character voice characteristics, and natural and fluent voice content can be generated by controlling its intonation, emotion, and speech speed. In this specific embodiment, an end-to-end speech synthesis model based on conditional variational autoencoder and adversarial learning is used as an audio generation model. The conditional variational autoencoder (VAE) framework aims to improve the naturalness and efficiency of speech synthesis by maximizing the variational lower bound (ELBO).

[0167] First, the VITS method builds a model through variational inference, aiming to maximize the variational lower bound (ELBO) of the unsolvable marginal log-likelihood of the data:

[0168]

[0169] Among them, p θ (z|c) is the prior distribution of the latent variable z, p θ (x|z) is the likelihood function of data point x, q φ (z|x) is the approximate posterior distribution. Maximizing ELBO is a trade-off between reconstruction loss and KL divergence, where the reconstruction loss encourages the generated data to be similar to the real data, while the KL divergence pushes the posterior distribution to be close to the prior distribution.

[0170] In terms of reconstruction loss, we use mel spectrogram as the target data point. Through the decoder, the latent variable z is upsampled to the waveform domain and then converted to a mel spectrogram. The L1 loss is used to calculate the difference between the predicted and target mel spectrograms:

[0171]

[0172] In order to improve the expressiveness of the generative model, a regularization flow is applied to the conditional prior distribution to transform the simple distribution into a more complex distribution:

[0173]

[0174] where f θ represents the regularized flow, allowing the model to capture richer data distribution characteristics.

[0175] In terms of alignment estimation, a monotone alignment search (MAS) based on dynamic programming is used to estimate the alignment between the input text and the target speech. A random duration predictor is also introduced to generate speech with different rhythms. This predictor is a flow-based generative model that processes discrete duration data through variational dequantization and variational data augmentation techniques. In this way, the model is able to learn the duration distribution of a given phoneme and generate speech rhythms with diversity during the synthesis process.

[0176] In terms of adversarial training, a discriminator is added to distinguish between generated speech and real speech to improve the quality of synthesized speech. The loss function of the discriminator includes least squares loss and feature matching loss, which work together on the generator to encourage it to generate outputs closer to real speech. Finally, the total loss function is a combination of VAE loss and GAN loss.

[0177] In addition, if the editing mode of lip generation is performed, that is, if the editing mode is lip generation, the video to be edited is sent to a pre-trained lip matching module, so that the lip matching module matches the lip shape of the audio in the video to be edited based on the trained SyncNet model and the residual jump connection; wherein,

[0178] The synchronization probability of an audio-video pair is evaluated by computing the dot product between the matched lip shape and speech embeddings:

[0179]

[0180] Among them, v and s represent the lip-sync video in the video to be edited and the speech embedding vector in the video to be edited respectively, ∈ is a preset parameter and is a positive number; the output value of the synchronization probability is between [0,1], indicating the probability of synchronization between the audio and video in the video to be edited.

[0181] In a specific embodiment, a pre-trained lip matching module is used to generate lip movements that match the specified audio on a video of a specific person. In this specific embodiment, a pre-trained model is used to enforce accurate and natural lip movement generation. The model is based on the SyncNet model. In order to adapt to the lip shape generation task, the SyncNet model is modified to enable it to process color images, and residual jump connections are introduced to deepen the model structure. In addition, cosine similarity and binary cross entropy loss are used as new loss functions to evaluate the synchronization probability of audio-video pairs by calculating the dot product between video and speech embeddings:

[0182]

[0183] Where v and s represent video and speech embeddings, respectively, and ∈ is a small positive number to avoid division by zero. The output value of this formula is between [0,1], indicating the probability of synchronization of the input audio-video pair.

[0184] In addition, the lip shape is generated by a pre-trained generator in the lip matching module, where a lip sync discriminator is used to penalize inaccurate lip shape generation when training the generator. The generator G consists of three main parts: an identity encoder, a speech encoder, and a face decoder. The identity encoder encodes a random reference frame R through a stack of residual convolutional layers and concatenates it along the channel axis with the pose prior P masked by the lower half of the target face. The speech encoder is also a stack of 2D convolutions that encodes the input speech segment S and then concatenates it with the facial representation. The decoder is a stack of convolutional layers containing a transposed convolutional layer for upsampling. The training goal of the generator is to minimize the L1 reconstruction loss between the generated frame Ld and the real frame LG, calculated as:

[0185]

[0186] At the same time, the generator also needs to minimize the lip sync loss E sync :

[0187]

[0188] Through training, the loss is reduced, and finally lip shapes that are visually realistic and synchronized with speech are generated.

[0189] To sum up, in short, after the user enters the input command, the semantic extraction and intent analysis are used to accurately understand the user's needs, and the material library is searched and screened to find the video materials that meet the requirements. Subsequently, the system presents the video materials with high matching degree to the user, and the user can independently select the video returned by the system, click the icon to select the most satisfactory result to perform subsequent operations, and the subsequent operations can be performed by entering the input command again. In this way, the prediction model can identify it as an editing command, and the video selected by the user can be edited by types such as video matting, video blending, video segmentation, video repair, audio generation and lip generation. Of course, the user can also spontaneously upload a video to be edited, and directly perform editing operations by entering the input command to generate multimedia content.

[0190] As described above, the conversational multimedia content generation method provided by the present embodiment first places the input instruction input value in front of the user dialog box, so that the background converts the input instruction into a standard text, and then uses a pre-trained large language model to perform semantic understanding and intent analysis on the standard text to determine whether the input instruction is a retrieval instruction or an editing instruction; if the input instruction is a retrieval instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model, so that the video retrieval model feeds back a target video that is compatible with the text unit; if the input instruction is an editing instruction, a similarity match is performed on the standard text to obtain an editing mode, and the editing mode is used to perform video matting, video blending, video segmentation, video repair, and Audio generation and lip-sync generation and other types of editing processing are used to obtain multimedia generated content. In this way, based on the advanced modern large language model, through the 5W1H element extraction technology, the user input instructions can be accurately extracted and analyzed for intent, so that users only need to enter text instructions to achieve the corresponding retrieval or editing needs. At the same time, the most advanced machine learning and deep learning technologies are introduced to train models to identify key multi-dimensional information in each modal material, thereby realizing the retrieval of text to multimedia materials and the call of text to tool libraries, and realizing the fusion editing of multimodal content. This method and system use multiple rounds of iterative operation in the form of dialogue, making multimedia content editing intelligent, which can greatly reduce the threshold of multimedia editing and provide users with a more convenient and efficient editing experience.

[0191] like Figure 3 As shown, the present invention also provides a conversational multimedia content generation system 100, which implements the conversational multimedia content generation method as described above, including:

[0192] The front user dialog box 101 is used to receive the input instruction input by the user and enable the background to convert the input instruction into a standard text;

[0193] A large language model 102, used for performing semantic understanding and intention analysis on the standard text to determine whether the input instruction is a search instruction or an edit instruction;

[0194] The video retrieval model 103 is used for performing 5W1H analysis on the standard text to obtain a text unit if the input instruction is a retrieval instruction, and feeding back a target video adapted to the text unit;

[0195] The video editing model 104 is used to perform similarity matching on the standard text to obtain an editing mode if the input instruction is an editing instruction, and use the editing mode to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

[0196] The specific implementation method is referred to the embodiment of the above-mentioned method for generating interactive multimedia content, which will not be described in detail here.

[0197] As described above, the conversational multimedia content generation system provided by the present invention first places the input instruction input value in front of the user dialog box 101, so that the background converts the input instruction into a standard text, and then uses the pre-trained large language model 102 to perform semantic understanding and intent analysis on the standard text to determine whether the input instruction is a retrieval instruction or an editing instruction; if the input instruction is a retrieval instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model 103, so that the video retrieval model feeds back a target video that is compatible with the text unit; if the input instruction is an editing instruction, the video to be edited is obtained through the video editing model 104, and the standard text is matched for similarity to obtain an editing mode, and the editing mode is used to perform video matting, video modulation, and the like on the video to be edited in sequence. And, video segmentation, video repair, audio generation and lip generation and other types of editing processing to obtain multimedia generated content. In this way, based on the advanced modern large language model, through the 5W1H element extraction technology, it can accurately extract the semantics and analyze the intention of user input instructions, so that users only need to enter text instructions to achieve the corresponding retrieval or editing needs. At the same time, the most advanced machine learning and deep learning technologies are introduced to train models to identify the key multi-dimensional information in each modal material, so as to realize the retrieval of text to multimedia materials and the call of text to tool library, and realize the fusion editing of multimodal content. The method and system use the form of dialogue to perform multiple rounds of iterative operation, which makes multimedia content editing intelligent, can greatly reduce the threshold of multimedia editing, and provide users with a more convenient and efficient editing experience.

[0198] As described above, the method and system for generating interactive multimedia content according to the present invention are described by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various improvements can be made to the method and system for generating interactive multimedia content according to the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A method for generating interactive multimedia content, characterized in that: include: Convert the input instructions in the front-end user dialog box into standard text through the background; The standard text is semantically understood and analyzed through a pre-trained large language model to determine whether the input instruction is a search instruction or an edit instruction; wherein, If the input instruction is a search instruction, a 5W1H analysis is performed on the standard text to obtain a text unit, and the text unit is input into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text unit; If the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, and the editing mode is used to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

2. The method for generating interactive multimedia content according to claim 1, wherein: The input instructions in the front-end user dialog box are converted into standard text through the background, including: Capturing input instructions in the front-end user dialog box through a preset front-end application program; Preprocessing the input instructions to obtain key instructions; The key instructions are converted into standard text in a standard format, and the standard text is transmitted to a back-end server via a network protocol based on SSL or TLS encryption technology; wherein the standard format is JSON format or XML format.

3. The method for generating interactive multimedia content according to claim 2, wherein: The process of pre-training a large language model includes: Build a basic language model based on the Transformer architecture; Pre-training the language basic model through a pre-collected text set to form an initial language model with a basic understanding of the language; The initial language model is repeatedly fine-tuned and refined to form a large language model.

4. The method for generating interactive multimedia content according to claim 3, wherein: The process of pre-training the language base model includes: Convert the input text in the given text set into a text sequence, convert the vocabulary in the text sequence into a vector representation, and retain the position information of each vocabulary in the sequence through position encoding; obtain the optimized word vector x=x1...x in the text sequence based on the window size of the language basic model and the historical words about each vocabulary in the pre-training process n The maximum likelihood estimate L pt : L Pt (x)=Σ i logP(x i [x i-k ,…,x i-1 ;0) Where k represents the language model window size; θ represents the parameters of the neural network model; P represents the probability distribution function; x = x1...x n Represents a text sequence of length n; The process of fine-tuning and refining the initial language model includes: Input the text sequence into the initial language model, and extract the hidden layer output corresponding to the last word in the last layer of the initial language model The predicted label is obtained by transforming the hidden layer output through a preset fully connected layer; the cross entropy loss function is applied to calculate the difference between the predicted label and the actual label until the cross entropy loss function reaches a preset cross entropy threshold, then fine-tuning is stopped; wherein the cross entropy loss function is: L(y|x1,…,x n )=softmax(h [L] W y ), where Wy represents the word vector matrix; h [L] is the hidden layer output; y represents the actual label of the text sequence; The downstream tasks of the initial language model are fine-tuned by optimizing the target optimization loss function, minimizing the target optimization loss function, and stopping the fine-tuning until the target optimization loss function is less than a preset optimization threshold; wherein the target optimization loss function is: L FT (C)=∑logP(y|x1,…,x n ); Where C represents the labeled data of the downstream task; x = x1...x n represents a text sequence of length n; y represents the actual label of the text sequence.

5. The method for generating interactive multimedia content according to claim 4, wherein: The process of performing 5W1H analysis on the standard text to obtain text units, and inputting the text units into a preset video retrieval model so that the video retrieval model feeds back a target video that matches the text units includes: Performing 5W1H analysis on the standard text to generate a 5W1H structured text template; Inputting the 5W1H structured text template into a preset video retrieval model, and comparing the 5W1H structured text template with the video material metadata in the video library of the video retrieval model to obtain a preset data amount of pending videos; The similarity comparison module in the video retrieval model compares the similarity between the preset number of pending videos and the 5W1H structured text template to obtain similarity values; The pending videos corresponding to the similarity values ​​are arranged from large to small according to the similarity values, and the top-ranked pending videos are used as a preset number of target videos.

6. The method for generating interactive multimedia content according to claim 5, wherein: The process of obtaining similarity values ​​by comparing the similarities between the preset number of pending videos and the 5W1H structured text template respectively through the similarity comparison module in the video retrieval model includes: Adjusting the frame image of the pending video to a preset size so as to be input into the similarity comparison module as a standard video; The TimeSformer layer in the similarity comparison module performs feature extraction on the standard video to obtain the image feature V of each frame image in the standard video. sst , aggregate the image features to obtain a visual embedding sequence The BERT model of the Transformer in the similarity comparison module extracts features from the 5W1H structured text template to obtain text features L SST , and add position information to the text features to obtain a text embedding sequence A similarity value is obtained based on the visual embedding sequence and the text embedding sequence; wherein the similarity value is: Among them, g v (·) and g t (·) is the default linear projection.

7. The method for generating interactive multimedia content according to claim 6, wherein: If the input instruction is an editing instruction, similarity matching is performed on the standard text to obtain an editing mode, including: Conducting 5W1H analysis on the standard text to obtain text units; Performing feature extraction on the text unit to obtain comparative text features; The comparison text feature is compared with the edit text feature in the comparison library for similarity to obtain the edit text with the highest similarity value ranking; and the edit mode corresponding to the edit text with the highest similarity value ranking is obtained.

8. The method for generating interactive multimedia content according to claim 7, wherein: If the editing mode is video matting, the video to be edited is sent to a preset video matting module, so that the video matting module performs video matting; wherein, The video matting module includes an encoder module, a loop decoder module, a depth-guided filter module and a segmentation module; A feature extraction operation is performed on a single video frame of the video to be edited by the encoder module to obtain multi-scale features; wherein the encoder module uses MobileNetV3-Large as a backbone network, and uses an LR-ASPP module behind the backbone network for a semantic segmentation task; and a dilated convolution without a downsampling step is used in the last layer of MobileNetV3; The recursive decoder module adopts a loop mechanism to autonomously learn the video to be edited based on the multi-scale features to obtain effective information, so as to determine the matting information and the time coherence information, and obtains the hidden features of the video to be edited through a preset decoder; Performing high-resolution upsampling on the video to be edited based on the hidden features and the multi-scale features by the depth-guided filter module to obtain high- and low-resolution sampling frames; The segmentation module generates high-resolution transparency and foreground based on the high- and low-resolution sampling frames, and segmentation is performed according to the high-resolution transparency and foreground to obtain a matte video; wherein the matte video includes a single-channel transparency mask prediction video, a three-channel foreground prediction video and a single-channel segmentation prediction video.

9. The method for generating interactive multimedia content according to claim 8, wherein: If the editing mode is lip generation, the video to be edited is sent to a pre-trained lip matching module, so that the lip matching module matches the lip shape of the audio in the video to be edited based on the trained SyncNet model and residual jump connection; wherein, The synchronization probability of an audio-video pair is evaluated by computing the dot product between the matched lip shape and speech embeddings: Among them, v and s represent the lip-sync video in the video to be edited and the speech embedding vector in the video to be edited respectively, ∈ is a preset parameter and is a positive number; the output value of the synchronization probability is between [0,1], indicating the probability of synchronization between the audio and video in the video to be edited.

10. A conversational multimedia content generation system, characterized in that Implementing the method for generating interactive multimedia content as claimed in any one of claims 1 to 9, comprising: The front user dialog box is used to receive the input instructions input by the user and enable the background to convert the input instructions into standard text; A large language model, used for performing semantic understanding and intent analysis on the standard text to determine whether the input instruction is a search instruction or an edit instruction; A video retrieval model, for performing a 5W1H analysis on the standard text to obtain a text unit if the input instruction is a retrieval instruction, and feeding back a target video that is compatible with the text unit; The video editing model is used to perform similarity matching on the standard text to obtain an editing mode if the input instruction is an editing instruction, and use the editing mode to edit the videos to be edited pre-selected by the user in turn to obtain multimedia generated content; wherein the editing mode includes video matting, video blending, video segmentation, video repair, audio generation and lip-sync generation.

Images