A system for providing personalized responses to users based on multi-modal large models

Abstract

The application provides a system for providing personalized replies for users based on a multimodal large model, comprising: a first acquisition module configured to acquire a first instruction request of a user; a mode selection module configured to determine at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, wherein the plurality of candidate modes comprise: generating text according to text, generating pictures according to text, generating videos according to text, and generating sound according to text; a second acquisition module configured to acquire a first text request and a second instruction request of the user; a first model determination module configured to determine at least one first running model from a plurality of candidate models pre-set in the desired mode according to the second instruction request, wherein the first running model responds to the first text request; and a first output module configured to output a first personalized reply corresponding to the first text request by running the model. The application can effectively improve the user interaction experience and realize high-quality, diversified and personalized reply generation.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a system that provides personalized responses to users based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially large-scale modeling technology, generative artificial intelligence is increasingly being applied in the fields of text, image, video, and sound generation. AI generation technology based on deep learning algorithms, through the training of neural networks, can simulate complex generation processes and has achieved initial success in the market.

[0003] However, existing AI-generated image technologies still have some limitations. Currently, most technologies on the market focus on text-to-image or image style conversion, and cannot yet handle more complex generation tasks, such as text-to-video or text-to-sound generation, thus failing to fully meet user needs. Furthermore, due to insufficient model depth or inadequate algorithm optimization, existing generation technologies suffer from numerous defects in detail processing, particularly when generating facial features, hands, clothing details, and various complex objects, easily resulting in distortion, detail errors, or splicing artifacts. These issues affect the accuracy of the generated images, indicating that existing technologies urgently need further improvement and optimization. Summary of the Invention

[0004] To address at least one of the above problems, this invention provides a method and system for providing personalized responses to users based on a multimodal large model.

[0005] According to one aspect of the present invention, a method for providing personalized responses to users based on a multimodal large model is provided, comprising: obtaining a user's first instruction request; determining at least one desired mode from a plurality of candidate modes according to the user's first instruction request, the plurality of candidate modes including: generating text based on text, generating an image based on text, generating a video based on text, and generating sound based on text; obtaining the user's first text request and a second instruction request; determining at least one first running model from a plurality of candidate models pre-set in the desired mode to respond to the user's first text request according to the second instruction request; and outputting a first personalized response corresponding to the first text request through the running model.

[0006] Preferably, the method further includes: obtaining the user's second text request and third instruction request; determining at least one second running model from the plurality of candidate models to respond to the user's second text request and first personalized response based on the third instruction request; and outputting a second personalized response corresponding to the second text request and the first personalized response through the running model.

[0007] Preferably, when the desired mode is to generate video based on text, the first operating mode includes: extracting text vectors from the first text request, the text vectors including sentence vectors and word vectors, the sentence vectors representing the overall semantic information of the first text request, and the word vectors representing the semantic information of each word in the first text request; sampling random noise vectors from a normal distribution; concatenating the sentence vectors and the noise vectors to generate latent variables, the latent variables serving as input to a generator; using a first convolutional filter and a second convolutional filter in the generator to generate a temporally continuous sequence of video frames from the latent variables, wherein the sequence of video frames is consistent with the overall semantic information; optimizing local regions of the video frames using the word vectors, wherein the local details of the video frames match the semantic information of the individual words; and outputting the video generated based on the first text.

[0008] Preferably, the consistency between the video frame sequence and the overall semantic information is evaluated by the following steps: setting a first discriminator to evaluate the consistency between the video as a whole and the overall semantic information.

[0009] Preferably, the consistency between the video frame sequence and the overall semantic information is evaluated by the following steps: setting a second discriminator to analyze the video frames frame by frame, and evaluating the authenticity of each frame and the degree of matching with the semantic information of the individual words.

[0010] Preferably, the video generated from the first text is optimized through the following steps to make the generated video visually realistic and semantically consistent with the description of the first text request: Calculating the generation loss of the generator, which is based on the judgment result of the second discriminator on the authenticity of the generated video, and the generator's goal is to minimize the generation loss to generate a realistic video; calculating a global consistency loss, which ensures that the generated video is consistent with the first text request at a global level by comparing the differences between the global features of the generated video and the overall semantic information of the first text request; calculating a local consistency loss, which ensures that the generated video is consistent with the first text request in detail by comparing the local region features of the generated video with the semantic information of the individual words; and training the generator, the first discriminator, and the second discriminator by jointly calculating the generation loss, the global consistency loss, and the local consistency loss to minimize the joint loss and optimize the visual quality and semantic consistency of the generated video.

[0011] Preferably, when the desired mode is to generate a video based on text, before outputting the first personalized response corresponding to the first text request through the running model, the method further includes the step of: confirming that the generated video does not contain abnormal frames, wherein the abnormal frames are abnormal events that do not conform to the normal mode or expected behavior.

[0012] Preferably, the generated video is confirmed to contain no abnormal frames through the following steps: feature extraction is performed on the generated video using an autoencoder, the autoencoder comprising: an encoder for extracting channel features of the generated video through convolution operations and spatial structure features of the generated video through a self-attention mechanism; a decoder for reconstructing video frames based on the extracted features; clustering the extracted features using a dual-scale feature clustering module, the clustering module simultaneously compressing the channel features and spatial structure features of the generated video to increase the distinguishing boundary between normal and abnormal data; in the video frame reconstruction task, the occurrence of the abnormal event is detected by calculating the average frame anomaly score of the video segment; and by setting a threshold, the reconstruction error of the video segment is compared to confirm that the generated video does not contain the abnormal frames.

[0013] According to another aspect of the present invention, a system for providing personalized responses to users based on a multimodal large model is provided, comprising: a first acquisition module for acquiring a user's first instruction request; a mode selection module for determining at least one desired mode from a plurality of candidate modes according to the user's first instruction request, the plurality of candidate modes including: generating text from text, generating an image from text, generating a video from text, and generating sound from text; a second acquisition module for acquiring the user's first text request and a second instruction request; a first model determination module for determining at least one first running model from a plurality of candidate models pre-set in the desired mode to respond to the user's first text request according to the second instruction request; and a first output module for outputting a first personalized response corresponding to the first text request through the running model.

[0014] Preferably, it further includes: a third acquisition module, used to acquire the user's second text request and third instruction request; a second model determination module, used to determine at least one second running model from the plurality of candidate models to respond to the user's second text request and first personalized response according to the third instruction request; and a second output module, used to output a second personalized response corresponding to the second text request and the first personalized response through the running model.

[0015] Preferably, when the desired mode is to generate a video based on text, the first running model includes: a text vector extraction module, used to extract text vectors from the first text request, the text vectors including sentence vectors and word vectors, the sentence vectors representing the overall semantic information of the first text request, and the word vectors representing the semantic information of each word in the first text request; a random noise generation module, used to sample random noise vectors from a normal distribution; a latent variable generation module, used to concatenate the sentence vectors and the noise vectors to generate latent variables, the latent variables serving as input to a generator; a video generation module, the generator using a first convolutional filter and a second convolutional filter to generate a temporally continuous video frame sequence from the latent variables, the video frame sequence being consistent with the overall semantic information; a video optimization module, used to optimize local regions of the video frames using the word vectors, so that the local details of the video frames match the semantic information of the individual words; and a third output module, used to output the video generated based on the first text.

[0016] Preferably, it further includes: a first discriminator for evaluating the consistency between the overall video and the overall semantic information.

[0017] Preferably, it further includes: a second discriminator, used to analyze the video frames frame by frame, and evaluate the authenticity of each frame and the degree of matching with the semantic information of the individual word.

[0018] Preferably, the system further includes: a generator loss calculation module, used to calculate the generation loss of the generator, the generation loss of the generator being based on the judgment result of the second discriminator on the authenticity of the generated video, the generator aiming to minimize the generation loss to generate a realistic video; a global consistency loss calculation module, used to calculate the global consistency loss, the global consistency loss ensuring that the generated video is consistent with the first text request at the global level by comparing the differences between the global features of the generated video and the overall semantic information of the first text request; a local consistency loss calculation module, used to calculate the local consistency loss, the local consistency loss ensuring that the generated video matches the first text request in detail by comparing the local region features of the generated video with the semantic information of the individual words; and a joint loss optimization module, used to train the generator, the first discriminator, and the second discriminator together with the generation loss, the global consistency loss, and the local consistency loss to minimize the joint loss and optimize the visual quality and semantic consistency of the generated video.

[0019] Preferably, when the desired mode is to generate a video based on text, the system further includes: an anomaly detection module, used to confirm that the generated video does not contain any abnormal frames before outputting a first personalized response corresponding to the first text request, wherein the abnormal frames are abnormal events that do not conform to the normal mode or expected behavior.

[0020] Preferably, the anomaly detection module includes: an autoencoder module for extracting features from the generated video, the autoencoder including: an encoder for extracting channel features of the generated video through convolution operations and extracting spatial structure features of the generated video through a self-attention mechanism; a decoder for reconstructing video frames based on the extracted features; a dual-scale feature clustering module for clustering the extracted features, the clustering module simultaneously compressing the channel features and spatial structure features of the generated video to increase the distinguishing boundary between normal and abnormal data; an anomaly scoring module for detecting the occurrence of the abnormal event by calculating the average frame anomaly score of the video segment in the video frame reconstruction task; and a threshold determination module for comparing the reconstruction error of the video segment by setting a threshold to confirm that the generated video does not contain the abnormal frame.

[0021] This invention provides a method and system for providing personalized responses to users based on a multimodal large model, which has the following beneficial effects:

[0022] The flexibility and personalization of multimodal interaction: This invention supports generating text, images, videos, and audio from text, and can select the most suitable candidate mode for generation based on the user's specific needs, achieving flexible multimodal interaction. Users can not only obtain text-based answers, but also generate personalized multimedia content such as images, videos, and audio, greatly enhancing the personalization and richness of the user experience.

[0023] High-quality and consistent video generation: In the process of generating videos from text, this invention combines text vectors and random noise vectors, uses convolutional filters to generate a temporally continuous sequence of video frames, and optimizes local details using word vectors. This ensures that the generated video frames maintain global consistency while meticulously reflecting the specific semantic information in the text request. This method ensures that the video is visually more realistic and semantically highly consistent with the user's text request.

[0024] Anomaly detection ensures video reliability: This invention also introduces an anomaly detection module, which uses an autoencoder to extract features from the generated video and a dual-scale feature clustering module to compress the video's channel features and spatial structure features, further increasing the distinguishing boundary between normal and abnormal data. Before outputting personalized video responses, the average frame anomaly score of the video segments is calculated, and a threshold is set to compare reconstruction errors, ensuring that the generated video does not contain abnormal frames. This mechanism effectively prevents anomalies in the generated video, ensuring the logical rationality and visual consistency of the video content.

[0025] Optimization and judgment of generation quality: This invention uses a joint optimization mechanism of a generator, a first discriminator, and a second discriminator to evaluate and optimize the generated video at both global and local levels. By calculating generation loss, global consistency loss, and local consistency loss, it ensures that the generated video not only maintains overall consistency with the text request but also matches word semantic information in detail, thereby improving the visual quality and semantic accuracy of the generated video.

[0026] Multiple interactions improve user satisfaction: This invention supports further optimization of the first personalized response based on the user's second text request. By obtaining more user feedback information, the system can select the most suitable model for multiple interactive optimizations, thereby continuously improving the quality and accuracy of personalized responses and increasing user satisfaction.

[0027] In summary, this invention can effectively improve the user interaction experience, achieve high-quality, diversified and personalized response generation, and ensure the accuracy and consistency of video content through a sound anomaly detection and optimization mechanism, thus significantly enhancing the practicality and reliability of the system. Attached Figure Description

[0028] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0029] Figure 1 This is a schematic diagram of an apparatus according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be noted that the following embodiments do not constitute a limitation on the scope of protection claimed by this invention, but are merely illustrative examples.

[0031] One embodiment of the present invention provides a system for providing personalized responses to users based on a multimodal large model, comprising: a first acquisition module for acquiring a user's first instruction request; a mode selection module for determining at least one desired mode from a plurality of candidate modes according to the user's first instruction request, the plurality of candidate modes including: generating text from text, generating an image from text, generating a video from text, and generating sound from text; a second acquisition module for acquiring the user's first text request and a second instruction request; a first model determination module for determining at least one first running model from a plurality of pre-set candidate models in the desired mode to respond to the user's first text request according to the second instruction request; and a first output module for outputting a first personalized response corresponding to the first text request through the running model.

[0032] Multimodal large-scale models are large-scale artificial intelligence models capable of processing and understanding data from multiple modalities (such as text, images, audio, video, etc.). They can not only handle single-type data (such as processing only images or only text), but also synthesize data from multiple modalities for joint analysis and generation. The multimodal large-scale model provided by this invention can receive text input and generate corresponding videos, or generate images by combining text descriptions, or even generate more complex outputs.

[0033] For example, by inputting the command "Generate video from text" through the interface, the system selects "Generate video from text" from "Generate text / Generate image / Generate video / Generate sound" according to the user's command. In this embodiment, the system analyzes the user's request content and determines that the user wants to obtain a video. The system further obtains the user's text request, such as the specific question entered by the user: "Provide a video that tells the story of major events in modern Chinese history, with a duration of no more than 1 minute, including at least 1 war event and 1 cultural event." The user then enters a command request, requesting "stable-diffusion-3-medium," which becomes the subsequent operational model for responding to the user's text request.

[0034] The system provides users with personalized responses in the form of generated videos, allowing them to view corresponding video demonstrations. This multimodal big data system can flexibly generate personalized responses in different modalities (such as text, images, videos, or audio) according to different user needs.

[0035] According to another embodiment of the present invention, the system further includes a third acquisition module for acquiring the user's second text request and a third instruction request; a second model determination module for determining at least one second running model from the plurality of candidate models to respond to the user's second text request and the first personalized response based on the third instruction request; and a second output module for outputting a second personalized response corresponding to the second text request and the first personalized response through the running model.

[0036] For example, if a user is dissatisfied with the first personalized response or wants to continue operating based on it, the system can further fulfill the user's needs. For instance, if the user inputs a specific question: "Based on the above response, provide a video recounting major events in modern Chinese history, no longer than 2 minutes, including at least 2 war events and 2 cultural events, where at least one war event and at least one cultural event must be consistent with those in the above response," the user can then input a request for "stable-diffusion-4-medium." This becomes the operational model for subsequent responses to the user's text request and the first personalized response.

[0037] The system can further process the generated initial personalized response based on the user's specific needs, satisfying the user's customization requirements. Furthermore, the system can effectively correlate content from different modalities (such as historical event text and video content) to ensure semantic and thematic consistency in the generated content. This intermodal correlation helps improve the user experience and avoids fragmentation or incoherence between generated content.

[0038] According to another embodiment of the present invention, when the desired mode is to generate a video based on text, the first running model includes: a text vector extraction module, used to extract text vectors from the first text request, the text vectors including sentence vectors and word vectors, the sentence vectors representing the overall semantic information of the first text request, and the word vectors representing the semantic information of each word in the first text request; a random noise generation module, used to sample random noise vectors from a normal distribution; a latent variable generation module, used to concatenate the sentence vectors and the noise vectors to generate latent variables, the latent variables serving as input to a generator; a video generation module, the generator using a first convolutional filter and a second convolutional filter to generate a temporally continuous sequence of video frames from the latent variables, the video frame sequence being consistent with the overall semantic information; a video optimization module, used to optimize local regions of the video frames using the word vectors, such that the local details of the video frames match the semantic information of the individual words; and a third output module, used to output the video generated based on the first text.

[0039] In this embodiment, sampling a random noise vector from a normal distribution includes the following steps:

[0040] 1. Determine the parameters of the normal distribution: The normal distribution is defined by two main parameters: mean μ: representing the central location of the distribution, set to 0. Standard deviation σ: representing the width or dispersion of the distribution, set to 1. The random noise vector of the normal distribution is sampled from the standard normal distribution (mean 0, standard deviation 1).

[0041] 2. Define the dimension of the vector: Predefine the dimension n of the noise vector, which determines the size of the sampled random noise vector. In this embodiment, the noise vector has a dimension of 100 to provide sufficient randomness to the generator.

[0042] 3. Sampling a noise vector from a normal distribution: Using Python (using the NumPy library), call the `np.random.normal` function to generate random numbers from a normal distribution with a given mean (0) and standard deviation (1). The generated noise vector is 100-dimensional.

[0043] 4. Input the noise vector into the generator: Once the noise vector is sampled from the normal distribution, it will be passed as input to the generator. The generator uses this random noise vector and other inputs to generate output data (video frames).

[0044] 5. Updates during training: During the training of a Generative Adversarial Network (GAN), noise vector sampling is a continuously iterative process. Each time, the generator samples a new noise vector from a normal distribution, ensuring the diversity of the generated samples. The generator's weights are continuously updated through training, ultimately enabling the generation of realistic outputs.

[0045] The first convolutional filter used in this embodiment can be a 3D convolutional filter and deconvolution. The specific implementation method for generating a series of video frame sequences from latent variables using the first convolutional filter involves generative models in deep learning for video generation tasks. The key to this step is utilizing 3D convolution to process time-series information, while combining it with deconvolution (also known as transposed convolution) to generate a temporally continuous and spatially consistent video frame sequence. The following is a detailed implementation method for this process:

[0046] 1. Input Latent Variables: First, the latent variables obtained from the previous steps (a combination of text embeddings and noise vectors) are used as input to the generator network. The latent variable z is a high-dimensional random vector carrying initial information for generating the video.

[0047] 2.3D Convolutional Filters: 3D convolution can operate simultaneously in both temporal and spatial dimensions, which is crucial for generating temporally continuous video frame sequences. The convolution operation of a 3D convolution kernel not only convolves the width and height of each frame but also performs convolution operations along the temporal dimension, thereby capturing the relationship between each frame in the video and the frames before and after it.

[0048] The core operation of 3D convolution is as follows: the size of the 3D convolution kernel is (kd, kh, kw), where kd is the kernel size in the time dimension, and kh and kw are the kernel sizes in the spatial dimensions. The convolution operation generates feature maps containing temporal information, thereby generating a temporally continuous sequence of frames layer by layer.

[0049] 3. Deconvolution (Transposed Convolution) Operation: Deconvolution (i.e., transposed convolution) is used to upsample feature maps, expanding them to a larger spatial resolution, thereby generating a higher-resolution video frame sequence. In video generation, deconvolution is used to progressively increase the size of video frames until the target resolution is reached.

[0050] The core operation of deconvolution: The deconvolution layer (ConvTranspose3d) progressively upsamples low-resolution feature maps to generate larger frames. Deconvolution expands low-dimensional features into high-resolution video frames by upsampling the input.

[0051] 4. Generating Video Frame Sequences: Through several rounds of 3D convolution and deconvolution operations, the generator gradually generates a temporally continuous sequence of video frames from the latent variables. These frames not only contain spatial features (visual information of each frame) but also capture the temporal dependencies between frames through 3D convolution. The format of a video frame is (batch_size, channels, depth, height, width), where depth represents the temporal dimension of the frame, i.e., the number of frames in the video; height and width represent the resolution of each frame, and channels are typically 3, representing the RGB channels.

[0052] 5. Progressive Upsampling and Generation of High-Resolution Frames: To generate high-resolution video frame sequences, 3D convolution and deconvolution operations are typically performed layer by layer. Through layer-by-layer deconvolution operations, the feature maps are expanded from low resolution to high resolution.

[0053] 6. Output of the video frame sequence: Ultimately, the generated video frame sequence consists of multiple temporally consecutive frames, each containing visual information at a specified resolution and number of channels. The video sequence can be further output to a video file or used for other applications (analysis).

[0054] By using 3D convolutional filters to capture time-series information and combining this with deconvolution operations to progressively upsample feature maps, it is possible to generate temporally continuous video frame sequences from latent variables. 3D convolution is responsible for extracting features in both temporal and spatial dimensions, while deconvolution upsamples layer by layer to generate high-resolution video. Through multiple layers of 3D convolution and deconvolution operations, realistic and temporally continuous video sequences can be generated.

[0055] Optimizing local regions of video frames using word vectors involves mapping and matching the semantic information of words in the text description with the local regions of the video frame to ensure that the local details of the video conform to the semantics of the words. This is achieved through the following steps:

[0056] 1. Preprocessing of input text and video frames

[0057] a. Extracting word vectors: Extracting the vector representation of each word from the input text description. These word vectors can be extracted using a pre-trained language model, and each word vector represents the semantic information of the word.

[0058] b. Extracting local features from video frames: Using video frames as input, a convolutional neural network is used to extract local region features from each frame. Feature extraction can be achieved using deep convolutional networks (such as ResNet, VGG, etc.), where each local region feature represents the visual information of that region.

[0059] 2. Calculate the similarity between word vectors and local features of video frames.

[0060] a. Alignment of word vectors with local features

[0061] The extracted word vectors are aligned with local features of the video frames. For each word (such as "red", "car", etc.), its vector is matched with the features of the corresponding local region of the video frame.

[0062] b. Similarity calculation

[0063] Cosine similarity or other similarity metrics are used to calculate the similarity between word vectors and local region features to determine the degree of matching between word semantics and local visual features. Cosine similarity is used to measure the similarity between vectors.

[0064] 3. Optimize local regions of video frames based on similarity

[0065] a. Weighted fusion

[0066] Based on the calculated similarity, the features of local regions in the video frame are adjusted to better align with the semantics of the word vectors. Similarity can be used as a weighting coefficient to fuse local region features with word vectors, generating details that more closely match the text description.

[0067] Example: If a word vector represents "red", then regions that match that word vector (such as cars) will enhance visual features associated with red.

[0068] b. Generate the optimized local region

[0069] Optimized local features are used to generate specific details of video frames, ensuring that these regions are semantically consistent with the input words. For example, the generated red car is matched with the word vector for "red".

[0070] 4. Optimized video frame generation

[0071] Optimized local region features are integrated into video frames, resulting in frames that not only match the text description as a whole, but also whose local details (such as "red car" and "road") are more consistent with the semantics of the words.

[0072] 5. Video frame output

[0073] Finally, through deconvolution or other generative operations, the optimized feature maps are restored into high-resolution video frames. The optimized local regions are visually consistent with the semantics of the corresponding words, making the generated video conform to the text description in terms of local details.

[0074] According to another embodiment of the present invention, a method and system for evaluating the consistency between a video frame sequence and overall semantic information are also provided: a first discriminator is set up to evaluate the consistency between the video as a whole and the overall semantic information. A second discriminator is also set up to analyze the video frames frame by frame to evaluate the authenticity of each frame and the degree of matching with the semantic information of individual words.

[0075] The generated video based on the first text is optimized through the following steps or system to make the generated video visually realistic and semantically consistent with the description requested by the first text: The generation loss of the generator is calculated, based on the judgment result of the second discriminator on the authenticity of the generated video. The generator's goal is to minimize the generation loss to generate a realistic video; a global consistency loss is calculated, which ensures that the generated video is consistent with the first text request at the global level by comparing the differences between the global features of the generated video and the overall semantic information of the first text request; a local consistency loss is calculated, which ensures that the generated video matches the first text request in detail by comparing the local region features of the generated video with the semantic information of individual words; the generator, the first discriminator, and the second discriminator are trained by jointly calculating the generation loss, the global consistency loss, and the local consistency loss, minimizing the joint loss to optimize the visual quality and semantic consistency of the generated video. Alternatively, a generator loss calculation module can be set up to calculate the generation loss of the generator, which is based on the second discriminator's judgment of the authenticity of the generated video. The generator's goal is to minimize the generation loss to generate a realistic video. A global consistency loss calculation module is set up to calculate the global consistency loss, which ensures that the generated video is consistent with the first text request at the global level by comparing the differences between the global features of the generated video and the overall semantic information of the first text request. A local consistency loss calculation module is set up to calculate the local consistency loss, which ensures that the generated video matches the first text request in detail by comparing the local region features of the generated video with the semantic information of the individual words. A joint loss optimization module is set up to train the generator, the first discriminator, and the second discriminator together with the generation loss, the global consistency loss, and the local consistency loss to minimize the joint loss and optimize the visual quality and semantic consistency of the generated video.

[0076] In this embodiment, the discriminator's task is to distinguish between real and generated videos, and to determine whether the generated video semantically matches the text description. The discriminator's goal is to ensure that the generator cannot successfully "fool" it.

[0077] Input: The discriminator receives two types of input:

[0078] Real video: An actual sequence of video frames used as a benchmark for comparison.

[0079] Video generation: A sequence of video frames generated by the generator.

[0080] Output: The discriminator provides a probability value indicating whether the input video is real or generated. Additionally, the discriminator evaluates whether the generated video matches the given text description.

[0081] A discriminator typically consists of two sub-networks:

[0082] First discriminator: Used to evaluate the authenticity of the entire video sequence and its semantic consistency with the text description. Second discriminator: Analyzes each frame of the video frame by frame, checking its authenticity and temporal consistency between frames.

[0083] The adversarial training of the generator and discriminator is a continuous optimization process. The generator and discriminator alternately update their weights, with the aim of continuously improving performance through mutual competition.

[0084] Step 1: Generator generates video

[0085] The generator receives a text vector and a noise vector, and generates a sequence of video frames that match the input text description. The generator attempts to generate a video that looks realistic enough to "fool" the discriminator.

[0086] Step 2: The discriminator evaluates the video.

[0087] The discriminator receives real video and video generated by the generator, and evaluates them separately.

[0088] The goal of the discriminator is to identify which videos are generated and which are real. The discriminator also needs to evaluate the semantic consistency between the generated videos and the text descriptions. The discriminator outputs probability values ​​to indicate whether the input video is generated or real, and calculates a loss value based on the evaluation results.

[0089] Step 3: Calculation of the loss function

[0090] The generator and discriminator each have their own independent loss functions.

[0091] Generator Loss: The generator's goal is to minimize the probability that the discriminator classifies its generated video as "fake." The generator's loss function is calculated based on the discriminator's evaluation of the generated video. If the generator's video is identified as real by the discriminator, its loss is small; otherwise, the loss is large.

[0092] Discriminator Loss: The discriminator's loss function is calculated based on its evaluation results of real and generated videos. The discriminator needs to minimize its incorrect judgments of real videos and incorrect judgments of generated videos.

[0093] Step 4: Parameter Update

[0094] Generator Update: The generator updates its parameters based on the gradient of the loss function using the backpropagation algorithm. By optimizing its weights, the generator aims to make the generated video more realistic and match the text description, hoping to fool the discriminator.

[0095] Discriminator Update: The discriminator also updates its parameters based on the gradient of the loss function through the backpropagation algorithm. The purpose is to improve its ability to recognize generated videos and ensure that it can accurately determine the semantic consistency between the video and the text description.

[0096] Step 5: Circuit training

[0097] The generator and discriminator are trained alternately. The generator attempts to produce more realistic videos, while the discriminator continuously improves its recognition capabilities. After multiple iterations, the generator gradually produces increasingly realistic videos, and the discriminator becomes more accurate.

[0098] 4. Introduce adversarial loss and consistency loss

[0099] In addition to generative adversarial loss, this embodiment also introduces consistency loss to ensure that the generated video is not only visually realistic but also highly semantically consistent with the input text description. These losses help the generator produce video content that better meets semantic requirements.

[0100] The adversarial training between the generator and discriminator is achieved through a competitive process. The generator attempts to produce videos that are as realistic as possible, while the discriminator continuously improves its ability to distinguish between generated and real videos. The generator optimizes its output by minimizing the discriminator's loss, while the discriminator continuously improves by maximizing its ability to recognize generated videos. Ultimately, the generator can produce high-quality, realistic videos that match the text descriptions, while the discriminator can accurately distinguish between generated and real videos.

[0101] In this embodiment, the input is a sentence ∫ and a noise vector z (z∈R) sampled from a normal distribution. 100 We designed a generator network G to generate a series of frames:

[0102] {R d∫,dz}→R dc×dl×dh×dw}

[0103] Where dc, dl, dh, and dw represent the number of channels, sequence length, frame height, and frame width, respectively. To capture the spatial and temporal information of the video, 3D convolutional filters and deconvolution operations are used, while 2D convolutional filters are used to synthesize spatial information, ensuring temporal coherence between adjacent frames. First, a fully connected layer is used to learn a unified embedding m, which is obtained by concatenating the text embedding e and noise variables. This unified embedding undergoes feature transformation, as shown in the following equation:

[0104] m = W e [ze]∈R dm+dz

[0105] Among them, W e ∈R dm+dz This is the transformation matrix. Then, the generator G uses this latent variable as input to generate the corresponding video, as shown in the following equation:

[0106] Q=G(m∈R) dc×dl×dh×dw

[0107] Here, Q is the generated video, Fi is the i-th frame of the generated video, and F... i ∈R dc×dh×dw .

[0108] 3.3 Discriminator

[0109] To ensure the generated video is realistic and maintains temporal continuity between adjacent frames, we used a two-layer discriminator setup: a first discriminator and a second discriminator.

[0110] 3.3.1 First Discriminator D1(Q) ^ e)

[0111] Discriminator D1 receives two inputs: a video tensor Q^ and a text embedding e. First, it processes the video input through a 3D convolutional layer to generate a video-level tensor T. Q^ ∈R dc×dl×dh×dw This tensor represents high-level features extracted from the video, covering different spatiotemporal locations. Next, the video-level tensor is enhanced with the text embedding e, incorporating semantic information, and then processed through a dense layer with a Softmax activation function to distinguish whether the input video comes from real data (real video) or generated data (synthetic video), and whether it is semantically consistent with the given text description. This process is shown in the following equation:

[0112] D1(Q^,e)→[0,1]

[0113] Unlike traditional discriminators, the additional requirement here is to maintain the semantic relationship between the generated video and its corresponding subtitles. Therefore, a conditional discriminator is needed that evaluates not only the realism of the video but also whether it matches the given text description. The discriminator's loss function is shown below:

[0114]

[0115] 3.3.2 Second Discriminator D2(Q^,e)

[0116] To further enhance the realism of frames and their semantic consistency with the given text, while maintaining temporal coherence, we employed a second discriminator. This discriminator actually consists of two sub-networks: a frame discriminator (Dframe) and a motion discriminator (Dmotion). The frame discriminator is responsible for determining the realism of each frame in the video and evaluating whether it maintains semantic consistency with the given text description. On the other hand, the motion discriminator is responsible for checking the temporal coherence in the video, evaluating the smoothness and natural flow of motion between adjacent frames.

[0117] Frame discriminator: D frame (F i e)

[0118] First, a 2D convolutional model is used to extract the frame-level tensor T for each frame. F ∈R dc×dh×dw Then, this frame-level tensor is combined with the text embedding e and input into the frame discriminator D. frame This is used to distinguish between the authenticity of a frame and its semantic consistency with the subtitles. The loss function is shown below:

[0119]

[0120] Motion discriminator: D motion

[0121] To ensure temporal consistency between adjacent frames, the Euclidean distance of the inter-frame tensor is used to calculate the similarity between them. The size of the motion tensor is shown in the following formula:

[0122]

[0123] The temporally coherent adversarial loss function is shown below:

[0124]

[0125] 3.4 Optimization

[0126] The overall optimization of the discriminator can be achieved by minimizing the combined loss at the video and frame levels, while also considering the temporal coherence loss, as shown in the following equation:

[0127]

[0128] By minimizing the above loss, the discriminator can identify the authenticity of videos and their frames, whether the video is semantically consistent with the descriptive text, and also identify temporal variations between frames. For the generator network, the adversarial losses at the video level and frame level are as follows:

[0129]

[0130] Furthermore, the VTM loss (LVTML) is introduced to further enhance the requirement for fine-grained consistency between the video and the descriptive text. The objective function of the final generator is as follows:

[0131]

[0132] The generator's objective function enables the generator to produce realistic videos that are consistent with the descriptive text.

[0133] According to another embodiment of the present invention, when the desired mode is to generate a video based on text, the method further includes: an anomaly detection module, used to confirm that the generated video does not contain abnormal frames before outputting a first personalized response corresponding to the first text request, wherein the abnormal frames are abnormal events that do not conform to the normal mode or expected behavior.

[0134] The anomaly detection module includes: an autoencoder module for extracting features from the generated video; the autoencoder includes: an encoder for extracting channel features of the generated video through convolution operations and spatial structure features of the generated video through a self-attention mechanism; a decoder for reconstructing video frames based on the extracted features; a dual-scale feature clustering module for clustering the extracted features, wherein the clustering module simultaneously compresses the channel features and spatial structure features of the generated video to increase the distinguishing boundary between normal and abnormal data; an anomaly scoring module for detecting the occurrence of the abnormal event by calculating the average frame anomaly score of the video segment during the video frame reconstruction task; and a threshold determination module for comparing the reconstruction error of the video segment by setting a threshold to confirm that the generated video does not contain the abnormal frame.

[0135] In this embodiment, abnormal frames in the video refer to frames that do not conform to normal patterns or expected behavior. These frames may be caused by errors or abnormal behavior during the generation process. When generating personalized videos, the model generates corresponding video segments based on the input first text request. During the generation of video segments, in order to ensure that the generated video meets the user's expectations and does not contain abnormal frames, this invention adds an anomaly detection step before outputting the video.

[0136] Feature extraction of the generated video is performed using an autoencoder.

[0137] After video generation, to detect the presence of abnormal frames, the system first uses an autoencoder to extract features from the generated video. An autoencoder is an unsupervised neural network architecture that includes an encoder and a decoder, capable of learning high-level features of the input data and reconstructing the input.

[0138] Encoder: The encoder is used to extract features from video clips. It extracts channel features and spatial structure features from the video through two mechanisms:

[0139] Channel feature extraction: Features are extracted from each pixel channel of the video frame through convolutional operations (such as convolutional neural networks, CNNs). These channel features contain information such as color, texture, and edges in the video frame. Each video frame generates multiple feature channels through multiple convolutional operations, with each channel reflecting different attributes in the video.

[0140] Spatial structure feature extraction: Spatial relationships and structural features of the video are extracted using self-attention mechanisms (such as the Swin Transformer framework). Self-attention mechanisms capture local and global spatial layout information in video frames by analyzing the dependencies between different pixels. Through self-attention operations using local windows and moving windows, the model can better understand the layout and movement of objects in the video.

[0141] Reconstructing video frames using a decoder

[0142] The decoder's role is to reconstruct video frames based on the features extracted by the encoder. The decoder gradually restores the spatial and channel information of the video through deconvolution operations, mapping the features back to the original image dimensions, thereby generating reconstructed frames that approximate the input frames.

[0143] Reconstruction Process: The autoencoder's reconstruction process attempts to restore the extracted features to the original video frames. For normal video data, the encoder and decoder can extract and reconstruct features well, resulting in small reconstruction errors. However, for data containing anomalous events, the model cannot accurately reconstruct the data because it has never seen anomalous events before, leading to larger reconstruction errors.

[0144] Feature clustering is performed using a two-scale feature clustering module.

[0145] To further enhance the ability to distinguish between normal and abnormal data, the channel features and spatial structure features extracted by the autoencoder are processed through a dual-scale feature clustering module.

[0146] Channel feature clustering: Channel features represent different attributes of video frames, such as color and texture. K-Means clustering is used to compress channel features and find cluster centers that represent normal patterns. Channel feature compression removes redundant information, retaining only the most representative features, which helps to increase the difference between normal and abnormal data.

[0147] Clustering of spatial structural features: Spatial structural features describe the spatial relationships between pixels in a video frame. By clustering these spatial features, the module can further compress the spatial patterns of normal data, making the normal data more compact in the feature space. The spatial features of anomalous data often do not conform to this pattern and are difficult to cluster, thus resulting in larger errors during reconstruction.

[0148] By using dual-scale clustering, the clustering module effectively compresses the features of the video and increases the distinction boundary between normal and abnormal data, making the abnormal data more prominent in the feature space.

[0149] Calculate the average frame anomaly score in the reconstruction task.

[0150] In order to detect the occurrence of abnormal events, the system assesses the degree of abnormality by calculating the average frame anomaly score of the video segment during the task of reconstructing video frames.

[0151] Anomaly score calculation: For each reconstructed frame of a video segment, the system calculates the reconstruction error for each frame, i.e., the difference between the original frame and the reconstructed frame. In this embodiment, Euclidean distance is used to measure this difference. To improve the stability of detection, the system not only calculates the reconstruction error of a single frame, but also averages the anomaly scores of each frame in the video segment to obtain the average anomaly score for that video segment.

[0152] The significance of anomaly scoring: Normal video clips typically have low reconstruction errors, resulting in low average anomaly scores; however, video clips containing anomalous events have larger reconstruction errors, leading to higher average anomaly scores. By calculating the average frame anomaly score, the system can more accurately determine the occurrence of anomalous events.

[0153] Compare reconstruction errors by setting a threshold.

[0154] After calculating the reconstruction error or average anomaly score of the video segment, the system uses a preset threshold to determine whether there are abnormal frames.

[0155] Setting a threshold: The threshold is determined based on the reconstruction error of normal data. During the training phase, the system will statistically analyze the reconstruction error range of normal data and select a suitable error value as the threshold. Generally, the reconstruction error of normal data will not exceed this threshold.

[0156] Identifying anomalous frames: If the average frame anomaly score or reconstruction error of a video segment exceeds a threshold, the video segment is considered to contain anomalous frames; if the reconstruction error is below the threshold, the video segment is considered to have no anomalous frames. In this way, the system can effectively filter out and process anomalous frames in generated videos.

[0157] Confirm that the generated video does not contain any abnormal frames.

[0158] Finally, the system performs anomaly detection on the generated video using the steps described above to confirm whether the video segment contains any abnormal frames. If no abnormal frames are detected, the system outputs the video segment as the final result. If abnormal frames are detected, the system can either regenerate the video or correct the abnormal frames.

[0159] This method extracts and reconstructs video features using an autoencoder and combines it with a dual-scale feature clustering module to effectively detect anomalous frames in the generated video. By setting a threshold to compare reconstruction errors and calculate the average frame anomaly score, the system can accurately determine the occurrence of abnormal events and confirm the quality of the generated video. This method effectively improves the accuracy of video generation and avoids unexpected abnormal behaviors during the generation process.

[0160] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A system for providing personalized responses to users based on a multimodal large model, characterized in that, include: The first acquisition module is used to acquire the user's first instruction request; The mode selection module is used to determine at least one desired mode from multiple candidate modes according to the user's first instruction request. The multiple candidate modes include: generating text based on text, generating an image based on text, generating a video based on text, and generating sound based on text. The second acquisition module is used to acquire the user's first text request and second instruction request; The first model determination module is configured to determine at least one first running model from a plurality of candidate models pre-set in the desired pattern to respond to the user's first text request, according to the second instruction request. The first output module is used to output a first personalized response corresponding to the first text request through the running model; When the desired mode is to generate video based on text, the first running model includes: A text vector extraction module is used to extract text vectors from the first text request. The text vectors include sentence vectors and word vectors. The sentence vectors represent the overall semantic information of the first text request, and the word vectors represent the semantic information of each word in the first text request. The random noise generation module is used to sample random noise vectors from a normal distribution; A latent variable generation module is used to concatenate the sentence vector with the noise vector to generate latent variables, which serve as the input to the generator. The video generation module uses a first convolutional filter and a second convolutional filter to generate a temporally continuous sequence of video frames from the latent variables, the sequence of video frames being consistent with the overall semantic information; a first discriminator is set to evaluate the consistency between the overall video and the overall semantic information; a second discriminator is set to analyze the video frames frame by frame to evaluate the authenticity of each frame and the degree of matching with the semantic information of individual words. The video optimization module is used to optimize local regions of the video frame using the word vectors, so that the local details of the video frame match the semantic information of the single word; align the word vectors with the local features of the video frame, calculate the similarity between the word vectors and the local features of the video frame, adjust the local features of the video frame based on the similarity, and integrate the optimized local features of the video frame into the video frame. The third output module is used to output the video generated based on the first text.

2. The system according to claim 1, characterized in that, Also includes: The third acquisition module is used to acquire the user's second text request and third instruction request; The second model determination module is used to determine at least one second running model from the plurality of candidate models to respond to the user's second text request and the first personalized reply, according to the third instruction request. The second output module is used to output a second personalized response corresponding to the second text request and the first personalized response through the running model.

3. The system according to claim 2, characterized in that, Also includes: A generator loss calculation module is used to calculate the generation loss of the generator, the generation loss of the generator is based on the judgment result of the second discriminator on the authenticity of the generated video, and the goal of the generator is to minimize the generation loss to generate a realistic video; A global consistency loss calculation module is used to calculate the global consistency loss, which ensures that the generated video is consistent with the first text request at the global level by comparing the differences between the global features of the generated video and the overall semantic information of the first text request. The local consistency loss calculation module is used to calculate the local consistency loss, which ensures that the generated video matches the first text request in detail by comparing the local region features of the generated video with the semantic information of the individual words. The joint loss optimization module is used to train the generator, the first discriminator, and the second discriminator by jointly using the generation loss, the global consistency loss, and the local consistency loss, so as to minimize the joint loss and optimize the visual quality and semantic consistency of the generated video.

4. The system according to any one of claims 1 to 3, characterized in that, When the desired mode is to generate video based on text, it also includes: An anomaly detection module is used to confirm that the generated video does not contain any abnormal frames before outputting the first personalized response corresponding to the first text request. The abnormal frames are abnormal events that do not conform to the normal pattern or expected behavior.

5. The system according to claim 4, characterized in that, The anomaly detection module includes: An autoencoder module is used to extract features from the generated video, the autoencoder comprising: An encoder is used to extract channel features of the generated video through convolution operations and to extract spatial structure features of the generated video through a self-attention mechanism. Decoder, used to reconstruct video frames based on extracted features; A dual-scale feature clustering module is used to cluster the extracted features. The clustering module also compresses the channel features and spatial structure features of the generated video to increase the distinction boundary between normal and abnormal data. An anomaly scoring module is used to detect the occurrence of anomalies by calculating the average frame anomaly score of a video segment during the video frame reconstruction task. The threshold determination module is used to compare the reconstruction error of video segments by setting a threshold, and to confirm that the generated video does not contain the abnormal frames.

Images