A method for providing personalized replies for users based on multi-modal large models

Abstract

The application provides a method for providing personalized replies for users based on a multimodal large model, comprising: obtaining a first instruction request of a user; determining at least one expected mode from a plurality of candidate modes according to the first instruction request of the user, wherein the plurality of candidate modes comprise: generating text according to text, generating pictures according to text, generating videos according to text, and generating sound according to text; obtaining a first text request and a second instruction request of the user, determining at least one first running model from a plurality of candidate models pre-set in the expected mode according to the second instruction request, and responding to the first text request; and outputting a first personalized reply corresponding to the first text request through the running model. The application can effectively improve the user interaction experience and realize high-quality, diversified and personalized reply generation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to a method for providing personalized replies for users based on a multi-modal large model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, especially large model technology, generative artificial intelligence is gradually widely used in the fields of text, image, video, and sound generation. AI generation technology based on deep learning algorithms can simulate complex generation processes through neural network training and has achieved preliminary success in the market.

[0003] However, the existing artificial intelligence generation technology still has some limitations. At present, most of the technologies on the market mainly generate pictures from text or convert picture styles, and cannot realize more complex generation tasks such as text-to-video and text-to-sound, making it difficult to fully meet the needs of users. In addition, due to insufficient model depth or insufficient algorithm optimization, the existing generation technology has many defects in detail processing, especially when generating human faces, hands, clothing details, and various complex objects, which are prone to distortion, detail errors, or splicing traces. These problems affect the accuracy of the generated images, indicating that the existing technology needs to be further improved and optimized. SUMMARY

[0004] To solve at least one of the above problems, the present application provides a method for providing personalized replies for users based on a multi-modal large model.

[0005] According to one aspect of the present application, a method for providing personalized replies for users based on a multi-modal large model is provided, comprising: obtaining a first instruction request of a user; determining at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, the plurality of candidate modes including: generating text from text, generating pictures from text, generating videos from text, and generating sounds from text; obtaining a first text request and a second instruction request of the user, determining at least one first running model from a plurality of candidate models pre-set in the desired mode to respond to the first text request of the user according to the second instruction request; and outputting a first personalized reply corresponding to the first text request through the running model.

[0006] Preferably, it further comprises: obtaining a second text request and a third instruction request of the user, determining at least one second running model from the plurality of candidate models to respond to the second text request and the first personalized reply of the user according to the third instruction request; and outputting a second personalized reply corresponding to the second text request and the first personalized reply through the running model.

[0007] Preferably, when the desired mode is to generate a video from a text, the first operation mode comprises: extracting a text vector from the first text request, the text vector comprising a sentence vector and a word vector, the sentence vector representing overall semantic information of the first text request, and the word vector being used to represent semantic information of individual words of the first text request; sampling a random noise vector from a normal distribution; splicing the sentence vector and the noise vector to generate a latent variable, the latent variable being used as an input of a generator; generating a sequence of time-continuous video frames from the latent variable by the generator using a first convolution filter and a second convolution filter, wherein the sequence of video frames is consistent with the overall semantic information; optimizing local regions of the video frames using the word vector, wherein local details of the video frames match the semantic information of the individual words; and outputting a video generated from the first text.

[0008] Preferably, the consistency of the sequence of video frames with the overall semantic information is evaluated by: setting a first discriminator for evaluating the consistency of the video as a whole with the overall semantic information.

[0009] Preferably, the consistency of the sequence of video frames with the overall semantic information is evaluated by: setting a second discriminator for analyzing the video frames frame by frame to evaluate the authenticity of each frame and the matching degree with the semantic information of the individual words.

[0010] Preferably, the video generated from the first text is optimized by: calculating a generation loss of the generator, the generation loss of the generator being based on a determination result of a second discriminator on the authenticity of the generated video, and the goal of the generator being to minimize the generation loss to generate a realistic video; calculating a global consistency loss, the global consistency loss ensuring that the generated video is consistent with the first text request at a global level by comparing differences between global features of the generated video and the overall semantic information of the first text request; calculating a local consistency loss, the local consistency loss ensuring that the generated video is consistent with the first text request in details by comparing local region features of the generated video with the semantic information of the individual words; and training the generator, the first discriminator and the second discriminator by jointly the generation loss, the global consistency loss and the local consistency loss, and minimizing a joint loss to optimize the visual quality and semantic consistency of the generated video.

[0011] Preferably, when the desired mode is to generate a video according to the text, before outputting the first personalized reply corresponding to the first text request by the running model, the method further comprises the step of confirming that the generated video does not contain abnormal frames, wherein the abnormal frames are abnormal events that do not conform to normal patterns or expected behaviors.

[0012] Preferably, the abnormal frames are confirmed by the following steps: feature extraction of the generated video by a self-encoder, wherein the self-encoder comprises an encoder for extracting channel features of the generated video by convolution operation and spatial structure features of the generated video by self-attention mechanism; a decoder for reconstructing video frames according to the extracted features; clustering of the extracted features by a dual-scale feature clustering module, wherein the clustering module simultaneously compresses the channel features and the spatial structure features of the generated video to increase the boundary between normal data and abnormal data; detection of the occurrence of the abnormal events by calculating the average frame abnormality score of the video segment in the task of reconstructing the video frames; and confirmation that the generated video does not contain the abnormal frames by setting a threshold and comparing the reconstruction errors of the video segments.

[0013] According to another aspect of the present application, a system for providing personalized replies to users based on multi-modal large models is provided, comprising: a first acquisition module for acquiring a first instruction request of a user; a mode selection module for determining at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, wherein the plurality of candidate modes include: generating text according to text, generating picture according to text, generating video according to text, and generating sound according to text; a second acquisition module for acquiring a first text request and a second instruction request of the user; a first model determination module for determining at least one first running model from a plurality of candidate models pre-set in the desired mode according to the second instruction request, wherein the first running model responds to the first text request of the user; and a first output module for outputting a first personalized reply corresponding to the first text request by the running model.

[0014] Preferably, the system further comprises: a third acquisition module for acquiring a second text request and a third instruction request of the user; a second model determination module for determining at least one second running model from the plurality of candidate models according to the third instruction request, wherein the second running model responds to the second text request of the user and the first personalized reply; and a second output module for outputting a second personalized reply corresponding to the second text request and the first personalized reply by the running model.

[0015] Preferably, when the desired mode is to generate a video according to a text, the first running model comprises: a text vector extraction module for extracting a text vector from the first text request, the text vector comprising a sentence vector and a word vector, the sentence vector representing overall semantic information of the first text request, and the word vector representing semantic information of each word of the first text request; a random noise generation module for sampling a random noise vector from a normal distribution; a latent variable generation module for splicing the sentence vector and the noise vector to generate a latent variable, the latent variable serving as an input of a generator; a video generation module, the generator using a first convolutional filter and a second convolutional filter to generate a time-continuous video frame sequence from the latent variable, the video frame sequence being consistent with the overall semantic information; a video optimization module for optimizing local regions of the video frame using the word vector, so that local details of the video frame match the semantic information of the individual word; and a third output module for outputting a video generated according to the first text.

[0016] Preferably, further comprising: a first discriminator for evaluating consistency of the video as a whole with the overall semantic information.

[0017] Preferably, further comprising: a second discriminator for analyzing the video frame by frame to evaluate authenticity of each frame and matching degree with the semantic information of the individual word.

[0018] Preferably, further comprising: a generator loss calculation module for calculating a generation loss of the generator, the generation loss of the generator being based on a determination result of the second discriminator on authenticity of the generated video, the target of the generator being to minimize the generation loss to generate a realistic video; a global consistency loss calculation module for calculating a global consistency loss, the global consistency loss ensuring that the generated video is consistent with the first text request on a global level by comparing differences between global features of the generated video and the overall semantic information of the first text request; a local consistency loss calculation module for calculating a local consistency loss, the local consistency loss ensuring that the generated video is consistent with the first text request in details by comparing local region features of the generated video with the semantic information of the individual word; and a joint loss optimization module for training the generator, the first discriminator and the second discriminator jointly on the generation loss, the global consistency loss and the local consistency loss to minimize a joint loss and optimize visual quality and semantic consistency of the generated video.

[0019] Preferably, when the desired mode is to generate a video according to the text, further comprising: an anomaly detection module, configured to confirm that the generated video does not contain abnormal frames before outputting the first personalized reply corresponding to the first text request, wherein the abnormal frames are abnormal events that do not conform to normal patterns or expected behaviors.

[0020] Preferably, the anomaly detection module comprises: a self-encoder module, configured to perform feature extraction on the generated video, wherein the self-encoder comprises: an encoder, configured to extract channel features of the generated video through convolution operation, and extract spatial structure features of the generated video through self-attention mechanism; a decoder, configured to reconstruct video frames according to the extracted features; a dual-scale feature clustering module, configured to cluster the extracted features, wherein the clustering module simultaneously compresses the channel features and the spatial structure features of the generated video to increase the boundary between normal data and abnormal data; an anomaly scoring module, configured to detect the occurrence of the abnormal event by calculating the average frame anomaly score of a video segment in the task of reconstructing video frames; and a threshold determination module, configured to confirm that the generated video does not contain the abnormal frames by setting a threshold and comparing the reconstruction errors of the video segments.

[0021] The present application provides a method and system for providing personalized replies to users based on multi-modal large models, which has the following advantages:

[0022] Flexibility and personalization of multi-modal interaction: The present application supports the generation of text, pictures, videos, sounds and other modes according to text, and can select the most suitable candidate mode according to the specific needs of the user, realizing flexible multi-modal interaction. Users can not only obtain text-based answers, but also generate personalized multimedia content such as pictures, videos and sounds, greatly improving the personalization and richness of user experience.

[0023] High quality and consistency of video generation: In the process of generating a video according to the text, the present application generates a time-continuous video frame sequence by combining the text vector and the random noise vector using a convolution filter, and optimizes the local details through the word vector, ensuring that the generated video frames not only maintain global consistency, but also reflect specific semantic information in the text request in detail. This method ensures that the video is more realistic in visual effect and highly consistent with the user's text request in semantic.

[0024] Abnormality detection guarantees the reliability of the video: The application also introduces an abnormality detection module, which uses a self-encoder to extract features from the generated video, compresses the channel features and spatial structure features of the video through a double-scale feature clustering module, and further increases the boundary between normal data and abnormal data. Before outputting the personalized video reply, the average frame abnormality score of the video segment is calculated, and a threshold is set to compare the reconstruction error, ensuring that the generated video does not contain abnormal frames. This mechanism effectively prevents abnormal phenomena in the generated video, ensuring the logical rationality and visual consistency of the video content.

[0025] Optimization and discrimination of generation quality: The application uses a joint optimization mechanism of the generator, the first discriminator and the second discriminator to evaluate and optimize the generated video at the global and local levels respectively. By calculating the generation loss, global consistency loss and local consistency loss, it ensures that the generated video not only matches the text request as a whole, but also matches the word semantic information in detail, thereby improving the visual quality and semantic accuracy of the generated video.

[0026] Multiple interactions improve user satisfaction: The application supports further optimization of the first personalized reply based on the user's second text request. By obtaining more user feedback information, the system can select the most suitable model for multiple interactive optimization, thereby continuously improving the quality and accuracy of the personalized reply and increasing user satisfaction.

[0027] In summary, the application can effectively improve the user interaction experience, realize high-quality, diversified and personalized reply generation, and guarantee the accuracy and consistency of the video content through a perfect abnormality detection and optimization mechanism, significantly enhancing the practicality and reliability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the application, and form a part of the application. The illustrative embodiments of the application and their descriptions serve to explain the application without forming an improper limitation on the application. In the drawings:

[0029] Figure 1 A flowchart is provided according to an embodiment of the application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below with reference to specific embodiments and the accompanying drawings. It should be noted that the following embodiments do not limit the scope of protection required by the application, but are only illustrative embodiments.

[0031] In one embodiment of the present application, a method for providing personalized replies for users based on a multi-modal large model is provided, which comprises: obtaining a first instruction request of a user; determining at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, the plurality of candidate modes comprising: generating text according to text, generating picture according to text, generating video according to text, and generating sound according to text; obtaining a first text request and a second instruction request of the user, determining at least one first running model from a plurality of candidate models pre-set in the desired mode according to the second instruction request, the first running model responding to the first text request of the user; and outputting a first personalized reply corresponding to the first text request through the running model.

[0032] The multi-modal large model can process and understand large-scale artificial intelligence models from multiple modalities (such as text, image, audio, video, etc.) data. It can not only process a single type of data (such as only processing images or only processing text), but also can comprehensively analyze and generate multiple modalities of data. The multi-modal large model provided by the present application can receive text input and generate corresponding video, or generate images or even more complex outputs by combining text descriptions.

[0033] For example, by inputting the instruction "generate video according to text" through the interface, the system selects "generate video according to text" from "generate text according to text / generate picture according to text / generate video according to text / generate sound according to text" according to the user's instruction. In this embodiment, the system analyzes the user's request content and determines that the user wants to obtain a video. The system further obtains the text request of the user, such as the specific question input by the user: "provide a video that tells the main events of Chinese modern history, the time is not more than 1 minute, and at least includes 1 war event and 1 cultural event", and the user then inputs the instruction request, requiring "stable-diffusion-3-medium", which becomes the running model for responding to the user's text request subsequently.

[0034] The system provides the generated video to the user in the form of a personalized reply, and the user can see the corresponding video demonstration. This system based on a multi-modal large model can flexibly generate personalized replies of different modalities (such as text, image, video or audio) according to different user needs.

[0035] According to another embodiment of the present application, it further comprises obtaining a second text request and a third instruction request of the user, determining at least one second running model from a plurality of candidate models according to the third instruction request, the second running model responding to the second text request and the first personalized reply of the user; and outputting a second personalized reply corresponding to the second text request and the first personalized reply through the running model.

[0036] For example, when the user is not satisfied with the first generated personalized reply or wants to continue operation based on the first generated personalized reply, the system can further implement the user's needs. As the specific question input by the user: "Based on the above reply, continue to provide a video telling the main events of Chinese modern history, with a time not exceeding 2 minutes, including at least 2 war events and 2 cultural events, and at least one war event and at least one cultural event need to be consistent with the above reply", the user then inputs the instruction request, requiring "stable-diffusion-4-medium", which becomes the running model for subsequent response to the user text request and the first personalized reply.

[0037] The system can further operate on the first generated personalized reply according to the user's specific needs to meet the user's customized requirements. In addition, the system can effectively associate different modalities of content (such as historical event text and video content) to ensure that the generated content is consistent in semantics and theme. This inter-modal association helps to improve user experience and avoid fragmentation or incoherence between generated content.

[0038] According to another embodiment of the present application, when the desired mode is to generate a video according to the text, the first running mode comprises: extracting a text vector from the first text request, the text vector comprising a sentence vector and a word vector, the sentence vector representing the overall semantic information of the first text request, and the word vector being used to represent the semantic information of each word of the first text request; sampling a random noise vector from a normal distribution; concatenating the sentence vector with the noise vector to generate a latent variable, the latent variable being used as an input of the generator; generating a time-continuous video frame sequence from the latent variable by the generator using a first convolutional filter and a second convolutional filter, wherein the video frame sequence is consistent with the overall semantic information; optimizing a local region of the video frame using the word vector, wherein the local details of the video frame match the semantic information of the individual word; and outputting a video generated according to the first text.

[0039] In this embodiment, the random noise vector is sampled from a normal distribution, comprising the following steps:

[0040] 1. Determine the parameters of the normal distribution: The normal distribution is defined by two main parameters: mean μ: representing the center position of the distribution, set to 0. Standard deviation σ: representing the width or dispersion of the distribution, set to 1. The random noise vector of the normal distribution is sampled from the standard normal distribution (mean 0, standard deviation 1).

[0041] 2. Define the dimension of the vector: The dimension n of the noise vector is pre-set, which determines the size of the sampled random noise vector. In this embodiment, the dimension of the noise vector is 100, which is used to provide sufficient randomness to the generator.

[0042] 3. Sample noise vector from normal distribution: Using Python (with NumPy library), call np.random.normal function to generate random numbers from a normal distribution with given mean (0) and standard deviation (1). The generated noise vector has a size of 100 dimensions.

[0043] 4. Input noise vector into generator: Once the noise vector is sampled from the normal distribution, it will be passed as input to the generator. The generator generates output data (video frames) based on this random noise vector and other inputs.

[0044] 5. Update in training: During the training of the generative adversarial network, the sampling of the noise vector is a constantly repeated process. Each time the generator samples a new noise vector from the normal distribution, this ensures that the generated samples have diversity. The weights of the generator are constantly updated through training, and eventually can generate realistic outputs.

[0045] The first convolutional filter used in this embodiment can be a 3D convolutional filter and deconvolution. The specific implementation method of generating a series of video frame sequences from the latent variable through the first convolutional filter involves a generative model in deep learning for video generation tasks. The key to this step is to use 3D convolution to process time series information, combined with deconvolution operation (also known as transpose convolution), to generate time-continuous and spatial-consistent video frame sequences. Here is the specific implementation method of this process:

[0046] 1. Input latent variable: First, the latent variable (combination of text embedding and noise vector) obtained from the previous step is taken as the input of the generator network. The latent variable z is a high-dimensional random vector that carries the initial information for generating videos.

[0047] 2. 3D convolutional filter: 3D convolution can operate simultaneously in time and spatial dimensions, which is crucial for generating time-continuous video frame sequences. The convolution operation of the 3D convolution kernel not only convolves in the width and height of each frame image, but also along the time dimension, capturing the relationship between each frame and the previous and next frames in the video.

[0048] Core operation of 3D convolution: The size of the 3D convolution kernel is (kd, kh, kw), where kd is the kernel size in the time dimension, and kh and kw are the convolution kernel sizes in the spatial dimension. The convolution operation generates feature maps containing time information, thus generating a time-continuous frame sequence layer by layer.

[0049] 3. Deconvolution (Transposed Convolution) Operation: Deconvolution (also known as transposed convolution) is used to upsample feature maps and expand them to a larger spatial resolution, thereby generating a higher-resolution video frame sequence. In video generation, deconvolution operations are used to gradually expand the size of video frames until the target resolution is reached.

[0050] The core operation of deconvolution: The deconvolution layer (ConvTranspose3d) is used to gradually upsample low-resolution feature maps to generate larger frames. Deconvolution expands low-dimensional features into high-resolution video frames by upsampling the input.

[0051] 4. Generating a video frame sequence: After several rounds of 3D convolution and deconvolution operations, the generator gradually generates a temporally continuous sequence of video frames from the latent variables. These frames not only contain spatial features (the visual information of each frame), but also capture the temporal dependencies between frames through 3D convolution. The format of the video frame is (batch_size, channels, depth, height, width), where depth represents the temporal dimension of the frame, that is, the number of frames in the video, height and width represent the resolution of each frame, and channels is usually 3, representing the RGB channels.

[0052] 5. Gradual upsampling and generation of high-resolution frames: To generate high-resolution video frame sequences, 3D convolution and deconvolution operations are usually performed layer by layer. Through layer-by-layer deconvolution operations, the feature maps are expanded from low resolution to high resolution.

[0053] 6. Output of video frame sequence: Ultimately, the generated video frame sequence consists of multiple temporally consecutive frames, each containing visual information at the specified resolution and number of channels. The video sequence can be further output to a video file or used for other application scenarios (analysis).

[0054] Using 3D convolutional filters to capture temporal information, combined with deconvolution operations to progressively upsample feature maps, we can generate a temporally continuous sequence of video frames from latent variables. 3D convolution extracts features across both temporal and spatial dimensions, while deconvolution upsamples layer by layer to produce high-resolution video. By applying multiple layers of 3D convolution and deconvolution, we can generate realistic and temporally continuous video sequences.

[0055] The core idea of ​​using word vectors to optimize local areas of video frames is to match and align the semantic information of words in text descriptions with local areas of video frames to ensure that the local details of the video are consistent with the semantics of the words. This is achieved through the following steps:

[0056] 1. Preprocessing of input text and video frames

[0057] a. Extract word vectors: Extract the vector representation of each word from the input text description. These word vectors can be extracted using a pre-trained language model, and each word vector represents the semantic information of that word.

[0058] b. Extract local features of video frames: Take video frames as input and use a convolutional neural network to extract local region features for each frame. Feature extraction can be achieved through a deep convolutional network such as ResNet, VGG, etc., and each local region feature represents the visual information of that region.

[0059] 2. Calculate the similarity between word vectors and local features of video frames

[0060] a. Alignment of word vectors and local features

[0061] Align the extracted word vectors with the local features of the video frames. For each word (such as "red", "car", etc.), we match its vector with the features of the corresponding local region of the video frame.

[0062] b. Similarity calculation

[0063] Use cosine similarity or other similarity measurement methods to calculate the similarity between word vectors and local region features to determine the matching degree of word semantics and local visual features. Cosine similarity is used to measure the similarity between vectors.

[0064] 3. Optimize local regions of video frames based on similarity

[0065] a. Weighted fusion

[0066] According to the calculated similarity, adjust the features of the local regions of the video frames to make them more consistent with the semantics of the word vectors. The similarity can be used as a weighting coefficient to fuse the local region features with the word vectors to generate more detailed details that match the text description.

[0067] Example: If the word vector represents "red", the region (such as a car) that matches this word vector will enhance the visual features related to red.

[0068] b. Generate optimized local regions

[0069] Use the optimized local features to generate specific details of the video frames, ensuring that these regions are consistent with the input word semantics. For example, the generated red car matches the word vector "red".

[0070] 4. Generation of optimized video frames

[0071] The optimized local region features are integrated into the video frames, and the generated frames not only match the text description as a whole, but also have more consistent local details (such as "red car", "road", etc.) with the word semantics.

[0072] 5. Output of the video frames

[0073] Finally, the optimized feature maps are restored into high-resolution video frames through deconvolution or other generation operations. The optimized local regions are visually consistent with the corresponding word semantics, so that the generated video is consistent with the text description in local details.

[0074] According to another embodiment of the present application, a method for evaluating the consistency of a video frame sequence with overall semantic information is also provided: a first discriminator is set to evaluate the consistency of the video as a whole with the overall semantic information. A second discriminator is also set to analyze the video frames frame by frame to evaluate the authenticity of each frame and the matching degree with the semantic information of individual words.

[0075] The video generated according to the first text is optimized by the following steps to make the generated video visually realistic and semantically consistent with the description requested by the first text: the generation loss of the generator is calculated, the generation loss of the generator is based on the determination result of the second discriminator on the authenticity of the generated video, and the target of the generator is to minimize the generation loss to generate realistic video; the global consistency loss is calculated, the global consistency loss ensures that the generated video is consistent with the first text request at the global level by comparing the global features of the generated video with the overall semantic information requested by the first text; the local consistency loss is calculated, the local consistency loss ensures that the generated video is consistent with the first text request in details by comparing the local region features of the generated video with the semantic information of individual words; the generator, the first discriminator and the second discriminator are trained by jointly the generation loss, the global consistency loss and the local consistency loss, and the joint loss is minimized to optimize the visual quality and semantic consistency of the generated video.

[0076] In this embodiment, the task of the discriminator is to distinguish between real videos and generated videos, and to judge whether the generated video is semantically matched with the text description. The target of the discriminator is to ensure that the generator cannot successfully "deceive" it.

[0077] Input: The discriminator receives two types of input:

[0078] Real video: a sequence of video frames that actually exist, used as a contrast benchmark.

[0079] Generated video: a sequence of video frames generated by the generator.

[0080] Output: The discriminator gives a probability value indicating whether the input video is real or generated. Additionally, the discriminator assesses whether the generated video aligns with the given textual description.

[0081] The discriminator typically consists of two sub-networks:

[0082] First discriminator: Evaluates the authenticity of the entire video sequence and its semantic consistency with the textual description. Second discriminator: Analyzes each frame of the video individually, checking its authenticity and temporal consistency between frames.

[0083] The adversarial training between the generator and discriminator is a continuous optimization process, with both the generator and discriminator updating their weights alternately, aiming to improve performance in a competitive manner.

[0084] Step 1: Generator generates a video

[0085] The generator receives the textual vector and noise vector, and generates a sequence of video frames that match the input textual description. The generator attempts to generate a video that appears sufficiently realistic to "fool" the discriminator.

[0086] Step 2: Discriminator evaluates the video

[0087] The discriminator receives both real videos and videos generated by the generator, and evaluates them separately.

[0088] The goal of the discriminator is to identify which videos are generated and which are real. The discriminator also needs to assess the semantic consistency of the generated video with the textual description. The discriminator outputs a probability value indicating whether the input video is generated or real, and calculates a loss value based on the evaluation results.

[0089] Step 3: Calculation of loss function

[0090] The generator and discriminator each have their own independent loss functions.

[0091] Generator's loss: The goal of the generator is to minimize the probability that the discriminator judges its generated video as "fake". The generator's loss function is calculated based on the discriminator's evaluation of the generated video. If the generator generates a video that is identified as real by the discriminator, its loss is smaller, and vice versa.

[0092] Discriminator's loss: The discriminator's loss function is calculated based on its evaluation of both real and generated videos. The discriminator needs to minimize its incorrect judgments of real videos and incorrect judgments of generated videos.

[0093] Step 4: Parameter update

[0094] Generator Update: The generator updates its parameters based on the gradient of the loss function through the backpropagation algorithm. The generator optimizes its weights to make the generated video more realistic and match the textual description in order to deceive the discriminator.

[0095] Discriminator Update: The discriminator also updates its parameters based on the gradient of the loss function through the backpropagation algorithm, aiming to improve its ability to recognize generated videos while ensuring accurate semantic consistency between videos and textual descriptions.

[0096] Step 5: Loop Training

[0097] The training of the generator and discriminator is alternately conducted, with the generator attempting to generate more realistic videos and the discriminator constantly improving its recognition ability. After multiple iterations, the generator gradually generates increasingly realistic videos, while the discriminator becomes more accurate.

[0098] 4. Introducing Adversarial Loss and Consistency Loss

[0099] In addition to the adversarial loss, this embodiment also introduces a consistency loss to ensure that the generated video is not only visually realistic but also highly consistent with the input textual description in terms of semantics. These losses help the generator generate video content that is more semantically consistent.

[0100] The adversarial training between the generator and the discriminator is achieved through mutual competition, with the generator attempting to generate as realistic videos as possible, while the discriminator constantly improves its ability to distinguish between generated and real videos. The generator optimizes its output by minimizing the loss of the discriminator, while the discriminator constantly improves by maximizing its ability to recognize generated videos. Ultimately, the generator can generate high-quality, semantically consistent realistic videos, and the discriminator can accurately distinguish between generated and real videos.

[0101] In this embodiment, the input is a sentence ∫ and a noise vector z (z ∈ R 100 ) sampled from a normal distribution, and we design a generator network G to generate a series of frames:

[0102] {R d∫,dz}→R dc×dl×dh×dw}

[0103] where dc, dl, dh, and dw represent the number of channels, sequence length, frame height, and frame width, respectively. To capture spatial and temporal information of the video, 3D convolution filters and deconvolution operations are used, while 2D convolution filters are used to synthesize spatial information and ensure temporal coherence between adjacent frames. First, a fully connected layer is used to learn a unified embedding m, which is obtained by concatenating the text embedding e and the noise variable. The unified embedding is converted through a feature transformation, as shown in the following equation:

[0104] m = W e [z.e] ∈ R dm+dz

[0105] where W e ∈ R dm+dz is the transformation matrix. Then, the generator G takes this latent variable as input and generates the corresponding video, as shown in the following equation:

[0106] Q = G(m) ∈ R dc×dl×dh×dw

[0107] Here, Q is the generated video, Fi is the i-th frame of the generated video, where F i ∈ R dc×dh×dw .

[0108] 3.3 Discriminator

[0109] To ensure that the generated video is realistic and maintains temporal coherence between adjacent frames, we use a two-layer discriminator setup: the first discriminator and the second discriminator.

[0110] 3.3.1 First Discriminator D1(Q ^ ,e)

[0111] The discriminator D1 receives two inputs: the video tensor Q^ and the text embedding e. First, it processes the video input through a 3D convolutional layer, generating a video-level tensor T Q^ ∈ R dc×dl×dh×dw . This tensor represents high-level features extracted from the video, covering different spatiotemporal positions. Next, the video-level tensor is augmented with the text embedding e, incorporating semantic information, and then processed through a dense layer with a Softmax activation function to distinguish whether the input video is from real data (real video) or generated data (synthetic video), and whether it maintains semantic consistency with the given text description. This process is shown in the following equation:

[0112] D1(Q^, e) → [0, 1]

[0113] Unlike traditional discriminators, the additional requirement here is to maintain the semantic relationship between the generated video and the corresponding caption. Therefore, a conditional discriminator is needed, which not only evaluates the authenticity of the video but also evaluates whether the video is consistent with the given text description. The loss function of the discriminator is shown in the following equation:

[0114]

[0115] 3.3.2 Second Discriminator D2(Q^, e)

[0116] To further enhance the realism of the frames and their semantic consistency with the given text, while maintaining temporal coherence, we use a second discriminator. This discriminator is actually composed of two sub-networks: a frame discriminator (Dframe) and a motion discriminator (Dmotion). The frame discriminator is responsible for judging the realism of each frame of the video and assessing whether it maintains semantic consistency with the given text description. On the other hand, the motion discriminator is responsible for checking the temporal coherence in the video, assessing the smoothness and natural flow of motion between adjacent frames.

[0117] Frame discriminator: D frame (F i ,e)

[0118] First, a 2D convolutional model is used to extract a frame-level tensor T F ∈R dc×dh×dw for each frame. This frame-level tensor is then combined with the text embedding e and input into the frame discriminator D frame to distinguish the realism of the frame and its semantic consistency with the caption. Its loss function is as follows:

[0119]

[0120] Motion discriminator: D motion

[0121] To maintain temporal coherence between adjacent frames, the Euclidean distance of the inter-frame tensor is used to calculate the similarity between adjacent frames. The size of the motion tensor is as follows:

[0122]

[0123] The temporal coherence adversarial loss function is as follows:

[0124]

[0125] 3.4 Optimization

[0126] The overall optimization of the discriminator can be achieved by minimizing the comprehensive loss at the video level and the frame level, while considering the temporal coherence loss, as follows:

[0127]

[0128] By minimizing the above loss, the discriminator can identify the realism of the video and its frames, as well as whether the video is semantically consistent with the description text, while also identifying the temporal changes between frames. For the generator network, the adversarial loss at the video level and the frame level is as follows:

[0129]

[0130] In addition, the VTM loss LVTML is introduced to further strengthen the requirement of fine-grained consistency between the video and the description text. The objective function of the final generator is as follows:

[0131]

[0132] The generator objective function enables the generator to generate a real and consistent video with the description text.

[0133] According to another embodiment of the present application, when the desired mode is to generate a video according to the text, before running the model to output the first personalized reply corresponding to the first text request, the step of confirming that the generated video does not contain abnormal frames is further included, the abnormal frames being abnormal events that do not conform to the normal mode or expected behavior. The step of confirming that the generated video does not contain abnormal frames is confirmed by the following steps:

[0134] The generated video is feature-extracted by a self-encoder, which includes an encoder for extracting channel features of the generated video through convolution operation and spatial structure features of the generated video through self-attention mechanism, and a decoder for reconstructing video frames according to the extracted features. The extracted features are clustered by a double-scale feature clustering module, which simultaneously compresses the channel features and the spatial structure features of the generated video to increase the boundary between normal data and abnormal data. In the task of reconstructing video frames, the occurrence of abnormal events is detected by calculating the average frame abnormal score of the video segment. By setting a threshold, the reconstruction error of the video segment is compared to confirm that the generated video does not contain abnormal frames.

[0135] In this embodiment, the abnormal frames in the video refer to frames that do not conform to the normal mode or expected behavior, which may be caused by errors or abnormal behavior during the generation process. When generating a personalized video, the model generates a video segment corresponding to the input first text request. In the process of generating the video segment, in order to ensure that the generated video meets the user's expectations and does not contain abnormal frames, the present application adds an abnormality detection step before outputting the video.

[0136] Feature extraction of the generated video by a self-encoder

[0137] After the video is generated, in order to detect whether there are abnormal frames, the system first extracts features of the generated video by using a self-encoder. The self-encoder is an unsupervised neural network architecture including an encoder and a decoder, which can learn high-level features of the input data and reconstruct the input.

[0138] Encoder: The encoder part is used to extract features from the video segment. Through two mechanisms, the encoder extracts the channel features and the spatial structure features of the video:

[0139] Channel Feature Extraction: Each pixel channel of the video frame is processed through a convolution operation (e.g., Convolutional Neural Network, CNN) to extract features. These channel features contain information such as color, texture, and edges in the video frame. Through multiple layers of convolution operations, each video frame generates multiple feature channels, each reflecting different attributes in the video.

[0140] Spatial Structure Feature Extraction: Spatial relationships and structural features of the video are extracted through a self-attention mechanism (e.g., Swin Transformer framework). The self-attention mechanism analyzes the interdependence between different pixels in the video frame to capture local and global spatial layout information in the video frame. Through local window and moving window self-attention operations, the model can better understand the layout and motion of objects in the video.

[0141] Reconstruction of Video Frames by Decoder

[0142] The role of the decoder is to reconstruct the video frame based on the features extracted by the encoder. The decoder gradually restores the spatial and channel information of the video through deconvolution operations, mapping the features back to the original image dimension, thereby generating a reconstructed frame similar to the input frame.

[0143] Reconstruction Process: The reconstruction process of the autoencoder attempts to restore the extracted features to the original video frame. For normal video data, the encoder and decoder can well extract and reconstruct the features, so the reconstruction error is small; for data containing abnormal events, the model cannot accurately reconstruct due to not having seen abnormal data, resulting in a larger reconstruction error.

[0144] Feature Clustering by Dual-Scale Feature Clustering Module

[0145] To further enhance the ability to distinguish between normal and abnormal data, the channel features and spatial structure features extracted by the autoencoder are processed by a dual-scale feature clustering module.

[0146] Clustering of Channel Features: Channel features represent different attributes of the video frame, such as color, texture, etc. Through K-Means clustering, the clustering module compresses the channel features and finds the cluster center representing the normal mode. Compression of channel features can remove redundant information and only retain the most representative features, which helps to increase the difference between normal and abnormal data.

[0147] Clustering of Spatial Structure Features: Spatial structure features describe the spatial relationships between pixel points in the video frame. Through clustering of these spatial features, the module can further compress the spatial patterns of normal data, making normal data more compact in the feature space. The spatial features of abnormal data often do not conform to this pattern and are difficult to be clustered, resulting in a larger error during reconstruction.

[0148] Through double-scale clustering, the clustering module effectively compresses the features of the video and increases the boundary between normal data and abnormal data, making the abnormal data more obvious in the feature space.

[0149] Calculating average frame anomaly score in reconstruction task

[0150] To detect the occurrence of abnormal events, the system calculates the average frame anomaly score of the video segment in the reconstruction task of the video frame to evaluate the degree of abnormality.

[0151] Calculation of anomaly score: For each reconstructed frame of a video segment, the system calculates the reconstruction error of each frame, i.e., the difference between the original frame and the reconstructed frame. In this embodiment, the Euclidean distance is used to measure this difference. To improve the stability of detection, the system not only calculates the reconstruction error of a single frame, but also averages the anomaly score of each frame in the video segment to obtain the average anomaly score of the video segment.

[0152] Significance of anomaly score: The reconstruction error of normal video segments is usually low, so the average anomaly score is also low; while for video segments containing abnormal events, the reconstruction error is larger, resulting in a higher average anomaly score. By calculating the average frame anomaly score, the system can more accurately judge the occurrence of abnormal events.

[0153] Comparing reconstruction error by setting threshold

[0154] After calculating the reconstruction error or average anomaly score of the video segment, the system determines whether there is an abnormal frame by setting a pre-set threshold.

[0155] Setting threshold: The threshold is determined according to the reconstruction error of normal data. The system will count the range of reconstruction error of normal data in the training stage and select a suitable error value as the threshold. Generally, the reconstruction error of normal data will not exceed this threshold.

[0156] Judging abnormal frame: If the average frame anomaly score or reconstruction error of a video segment exceeds the threshold, the video segment contains an abnormal frame; if the reconstruction error is lower than the threshold, it is considered that the video segment has no abnormal frame. In this way, the system can effectively screen out abnormal frames in the generated video and process them.

[0157] Confirming that the generated video does not contain abnormal frames

[0158] Finally, the system performs anomaly detection on the generated video through the above steps to confirm whether there are abnormal frames in the video segment. After confirming that there are no abnormal frames in the video, the system outputs the video segment as the final generated result. If abnormal frames are detected, the system can regenerate the video or correct the abnormal frames.

[0159] The method can effectively detect abnormal frames in the generated video by feature extraction and reconstruction of the video through the autoencoder and combining the double-scale feature clustering module. By setting a threshold to compare the reconstruction error and calculating the average frame anomaly score, the system can accurately determine the occurrence of abnormal events and confirm the quality of the generated video. This method effectively improves the accuracy of video generation and avoids the generation of unexpected abnormal behavior during the generation process.

[0160] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for providing personalized replies for a user based on a multi-modal large model, characterized in that, The method comprises the following steps: obtaining a first instruction request of a user; determining at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, wherein the plurality of candidate modes comprise: generating text according to text, generating pictures according to text, generating videos according to text, and generating sound according to text; obtaining a first text request and a second instruction request of the user, and determining at least one first running model from a plurality of candidate models pre-set in the desired mode according to the second instruction request, wherein the first running model responds to the first text request of the user; outputting a first personalized reply corresponding to the first text request through the running model; when the desired mode is generating videos according to text, the first running model comprises the following steps: extracting a text vector from the first text request, wherein the text vector comprises a sentence vector and a word vector, the sentence vector represents overall semantic information of the first text request, and the word vector is used to represent semantic information of each word of the first text request; sampling a random noise vector from a normal distribution; splicing the sentence vector and the noise vector to generate a latent variable, wherein the latent variable is used as an input of a generator; generating a time-continuous video frame sequence from the latent variable through the generator using a first convolution filter and a second convolution filter, wherein the video frame sequence is consistent with the overall semantic information, a first discriminator and a second discriminator are set to ensure that the generated video is real and maintains time continuity between adjacent frames; the first discriminator is used to evaluate the consistency of the overall video with the overall semantic information; the second discriminator is set to analyze the video frame by frame to evaluate the authenticity of each frame and the matching degree with the semantic information of a single word; optimizing a local area of the video frame using the word vector, wherein local details of the video frame are matched with the semantic information of the single word, aligning the word vector with the local of the video frame, calculating the similarity of the word vector and the features of the local of the video frame, adjusting the features of the local of the video frame according to the similarity, and integrating the optimized features of the local of the video frame into the video frame; outputting a video generated according to the first text.

2. The method of claim 1, wherein, The method further comprises the following steps: obtaining a second text request and a third instruction request of the user, and determining at least one second running model from the plurality of candidate models according to the third instruction request, wherein the second running model responds to the second text request of the user and the first personalized reply; outputting a second personalized reply corresponding to the second text request and the first personalized reply through the running model.

3. The method of claim 2, wherein, The video generated according to the first text is optimized by the following steps to make the generated video visually realistic and semantically consistent with the description of the first text request: calculating a generation loss of the generator, wherein the generation loss of the generator is based on the determination result of the second discriminator on the authenticity of the generated video, and the target of the generator is to minimize the generation loss to generate a realistic video; calculating a global consistency loss that ensures the generated video is consistent with the first text request at a global level by comparing a global feature of the generated video with overall semantic information of the first text request; calculating a local consistency loss that ensures the generated video is consistent with the first text request in details by comparing a local region feature of the generated video with semantic information of the single word; training the generator, the first discriminator and the second discriminator by jointly the generation loss, the global consistency loss and the local consistency loss, minimizing the joint loss to optimize the visual quality and semantic consistency of the generated video.

4. The method according to any one of claims 1 to 3, characterized in that, when the expected mode is generating a video according to a text, before outputting a first personalized reply corresponding to the first text request by the running model, further comprising the steps of: confirming that the generated video does not contain abnormal frames, the abnormal frames being abnormal events that do not conform to normal patterns or expected behaviors.

5. The method of claim 4, wherein, confirming that the generated video does not contain abnormal frames by the following steps: extracting features of the generated video by a self-encoder, the self-encoder comprising: an encoder for extracting channel features of the generated video by convolution operation and spatial structure features of the generated video by self-attention mechanism; a decoder for reconstructing video frames according to the extracted features; clustering the extracted features by a dual-scale feature clustering module, the clustering module simultaneously compressing the channel features and the spatial structure features of the generated video to increase the boundary between normal data and abnormal data; in the task of reconstructing video frames, detecting the occurrence of the abnormal event by calculating the average frame abnormal score of the video segment; confirming that the generated video does not contain the abnormal frames by setting a threshold and comparing the reconstruction error of the video segment.

Images