Method for providing personalized reply for user based on multi-modal large model

Abstract

The invention provides a method for providing personalized reply for a user based on a multi-modal large model. The method comprises the following steps: acquiring a first instruction request of the user; determining at least one of a plurality of candidate modes as an expected mode according to a first instruction request of a user, wherein the plurality of candidate modes comprise: generating a text according to the text, generating a picture according to the text, generating a video according to the text, and generating a sound according to the text; obtaining a first text request and a second instruction request of the user, and determining at least one first operation model from a plurality of candidate models preset in the expected mode according to the second instruction request to respond to the first text request of the user; and outputting a first personalized reply corresponding to the first text request through the running model. According to the method, the user interaction experience can be effectively improved, and high-quality, diversified and personalized reply generation is realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for providing personalized responses to users based on a multimodal large model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially large model technology, the application of generative artificial intelligence in the fields of text, image, video, and sound generation has become increasingly widespread. AI generation technology based on deep learning algorithms can simulate complex generation processes through neural network training and has achieved initial success in the market.

[0003] However, the existing AI generation technology still has some limitations. At present, most technologies on the market are mainly based on text generation of pictures or picture style conversion, and are unable to achieve more complex generation tasks, such as text generation of videos, text generation of sounds, etc., and it is difficult to fully meet the needs of users. In addition, due to insufficient model depth or insufficient algorithm optimization, the existing generation technology has many defects in detail processing, especially when generating character faces, hands, clothing details and various complex objects, it is easy to have distortion, detail errors or splicing traces. These problems affect the accuracy of the generated images, indicating that the existing technology urgently needs further improvement and optimization. Summary of the invention

[0004] In order to solve at least one of the above problems, the present invention provides a method for providing personalized responses to users based on a multimodal large model.

[0005] According to one aspect of the present invention, there is provided a method for providing personalized replies to users based on a multimodal large model, comprising: obtaining a first instruction request from a user; determining at least one as an expected mode from a plurality of candidate modes according to the first instruction request of the user, the plurality of candidate modes comprising: generating text from text, generating pictures from text, generating videos from text, and generating sounds from text; obtaining the first text request and the second instruction request of the user, and determining at least one first operating model from a plurality of candidate models pre-set in the expected mode according to the second instruction request to respond to the first text request of the user; and outputting a first personalized reply corresponding to the first text request through the operating model.

[0006] Preferably, it also includes: obtaining the user's second text request and a third instruction request, determining at least one second operating model from the multiple candidate models to respond to the user's second text request and the first personalized reply according to the third instruction request; and outputting a second personalized reply corresponding to the second text request and the first personalized reply through the operating model.

[0007] Preferably, when the expected mode is to generate a video based on text, the first operating mode includes: extracting a text vector from the first text request, the text vector including a sentence vector and a word vector, the sentence vector representing the overall semantic information of the first text request, and the word vector being used to represent the semantic information of each word of the first text request; sampling a random noise vector from a normal distribution; concatenating the sentence vector and the noise vector to generate a latent variable, the latent variable serving as an input of a generator; using the generator to generate a time-continuous video frame sequence from the latent variable using a first convolution filter and a second convolution filter, wherein the video frame sequence is consistent with the overall semantic information; using the word vector to optimize a local area of ​​the video frame, wherein the local details of the video frame match the semantic information of the single word; and outputting a video generated based on the first text.

[0008] Preferably, the consistency between the video frame sequence and the overall semantic information is evaluated by the following steps: a first discriminator is set to evaluate the consistency between the video as a whole and the overall semantic information.

[0009] Preferably, the consistency of the video frame sequence with the overall semantic information is evaluated by the following steps: a second discriminator is set to analyze the video frames frame by frame, and evaluate the authenticity of each frame and the matching degree with the semantic information of the single word.

[0010] Preferably, the video generated according to the first text is optimized by the following steps so that the generated video is visually realistic and semantically consistent with the description of the first text request: calculating the generation loss of the generator, the generation loss of the generator is based on the determination result of the second discriminator on the authenticity of the generated video, and the goal of the generator is to minimize the generation loss to generate a realistic video; calculating the global consistency loss, the global consistency loss ensures that the generated video is consistent with the first text request at the global level by comparing the difference between the global features of the generated video and the overall semantic information of the first text request; calculating the local consistency loss, the local consistency loss ensures that the generated video is consistent with the first text request in details by comparing the local area features of the generated video with the semantic information of the single word; training the generator, the first discriminator and the second discriminator by jointly using the generation loss, the global consistency loss and the local consistency loss, minimizing the joint loss to optimize the visual quality and semantic consistency of the generated video.

[0011] Preferably, when the expected mode is to generate a video based on text, before outputting a first personalized response corresponding to the first text request through the running model, it also includes the step of: confirming that the generated video does not contain abnormal frames, and the abnormal frames are abnormal events that do not conform to the normal mode or expected behavior.

[0012] Preferably, the following steps are performed to confirm that the generated video does not contain abnormal frames: extracting features from the generated video through an autoencoder, the autoencoder comprising: an encoder for extracting channel features of the generated video through a convolution operation, and extracting spatial structural features of the generated video through a self-attention mechanism; a decoder for reconstructing video frames based on the extracted features; clustering the extracted features through a dual-scale feature clustering module, the clustering module simultaneously compressing the channel features and spatial structural features of the generated video to increase the distinguishing boundary between normal data and abnormal data; in the task of reconstructing video frames, detecting the occurrence of the abnormal event by calculating the average frame anomaly score of the video clip; and confirming that the generated video does not contain the abnormal frame by setting a threshold and comparing the reconstruction error of the video clip.

[0013] According to another aspect of the present invention, there is provided a system for providing personalized replies to users based on a multimodal large model, comprising: a first acquisition module for acquiring a first instruction request from a user; a mode selection module for determining at least one desired mode from a plurality of candidate modes according to the first instruction request from the user, the plurality of candidate modes comprising: generating text from text, generating pictures from text, generating videos from text, and generating sounds from text; a second acquisition module for acquiring the first text request and the second instruction request from the user; a first model determination module for determining at least one first operating model from a plurality of candidate models pre-set in the desired mode according to the second instruction request to respond to the first text request from the user; and a first output module for outputting a first personalized reply corresponding to the first text request through the operating model.

[0014] Preferably, it also includes: a third acquisition module, used to acquire the user's second text request and a third instruction request; a second model determination module, used to determine at least one second operating model from the multiple candidate models to respond to the user's second text request and the first personalized reply according to the third instruction request; and a second output module, used to output a second personalized reply corresponding to the second text request and the first personalized reply through the operating model.

[0015] Preferably, when the expected mode is to generate a video based on text, the first operating model includes: a text vector extraction module, used to extract a text vector from the first text request, the text vector includes a sentence vector and a word vector, the sentence vector represents the overall semantic information of the first text request, and the word vector is used to represent the semantic information of each word of the first text request; a random noise generation module, used to sample a random noise vector from a normal distribution; a latent variable generation module, used to concatenate the sentence vector and the noise vector to generate a latent variable, and the latent variable is used as an input of the generator; a video generation module, the generator uses a first convolution filter and a second convolution filter to generate a time-continuous video frame sequence from the latent variable, and the video frame sequence is consistent with the overall semantic information; a video optimization module, used to use the word vector to optimize the local area of ​​the video frame so that the local details of the video frame match the semantic information of the single word; a third output module, used to output the video generated according to the first text.

[0016] Preferably, the method further comprises: a first discriminator for evaluating the consistency between the video as a whole and the overall semantic information.

[0017] Preferably, it also includes: a second discriminator, which is used to analyze the video frames frame by frame, and evaluate the authenticity of each frame and the matching degree with the semantic information of the single word.

[0018] Preferably, it also includes: a generator loss calculation module, which is used to calculate the generation loss of the generator, and the generation loss of the generator is based on the determination result of the authenticity of the generated video by the second discriminator, and the goal of the generator is to minimize the generation loss to generate a realistic video; a global consistency loss calculation module, which is used to calculate the global consistency loss, and the global consistency loss ensures that the generated video is consistent with the first text request at the global level by comparing the difference between the global features of the generated video and the overall semantic information of the first text request; a local consistency loss calculation module, which is used to calculate the local consistency loss, and the local consistency loss ensures that the generated video is consistent with the first text request in details by comparing the local area features of the generated video with the semantic information of the single word; a joint loss optimization module, which is used to train the generator, the first discriminator and the second discriminator by combining the generation loss, the global consistency loss and the local consistency loss to minimize the joint loss and optimize the visual quality and semantic consistency of the generated video.

[0019] Preferably, when the expected mode is to generate a video based on text, it also includes: an anomaly detection module, which is used to confirm that the generated video does not contain abnormal frames before outputting a first personalized reply corresponding to the first text request. The abnormal frames are abnormal events that do not conform to the normal mode or expected behavior.

[0020] Preferably, the anomaly detection module includes: an autoencoder module for extracting features from the generated video, the autoencoder including: an encoder for extracting channel features of the generated video through a convolution operation, and extracting spatial structural features of the generated video through a self-attention mechanism; a decoder for reconstructing video frames according to the extracted features; a dual-scale feature clustering module for clustering the extracted features, the clustering module simultaneously compressing the channel features and spatial structural features of the generated video to increase the distinguishing boundary between normal data and abnormal data; an anomaly scoring module for detecting the occurrence of the abnormal event by calculating the average frame anomaly score of the video clip in the task of reconstructing the video frame; a threshold determination module for comparing the reconstruction error of the video clip by setting a threshold to confirm that the generated video does not contain the abnormal frame.

[0021] The present invention provides a method and system for providing personalized responses to users based on a multimodal large model, which has the following beneficial effects:

[0022] Flexibility and personalization of multimodal interaction: The present invention supports the generation of multiple modes such as text, pictures, videos, and sounds based on text, and can select the most suitable candidate mode for generation according to the specific needs of users, thereby realizing flexible multimodal interaction. Users can not only obtain text-based answers, but also generate personalized multimedia content such as pictures, videos, and sounds, which greatly improves the personalization and richness of the user experience.

[0023] High quality and consistency of video generation: In the process of generating videos based on text, the present invention combines text vectors and random noise vectors, uses convolution filters to generate a temporally continuous sequence of video frames, and optimizes local details through word vectors, ensuring that the generated video frames can maintain global consistency and carefully reflect the specific semantic information in the text request. This method ensures that the video is more realistic in visual effects and highly consistent with the user's text request in semantics.

[0024] Anomaly detection ensures the reliability of the video: The present invention also introduces an anomaly detection module, uses an autoencoder to extract features from the generated video, and compresses the channel features and spatial structure features of the video through a dual-scale feature clustering module to further increase the distinction between normal data and abnormal data. Before outputting a personalized video response, the average frame anomaly score of the video clip is calculated, and a threshold is set to compare the reconstruction error to ensure that the generated video does not contain abnormal frames. This mechanism effectively prevents abnormal phenomena in the generated video and ensures the logical rationality and visual consistency of the video content.

[0025] Optimization and discrimination of generation quality: The present invention evaluates and optimizes the generated video at the global and local levels through the joint optimization mechanism of the generator, the first discriminator and the second discriminator. By calculating the generation loss, the global consistency loss and the local consistency loss, it ensures that the generated video is not only consistent with the text request as a whole, but also matches the semantic information of the words in detail, thereby improving the visual quality and semantic accuracy of the generated video.

[0026] Multiple interactions improve user satisfaction: The present invention supports further optimization of the first personalized reply based on the user's second text request. By obtaining more user feedback information, the system can select the most suitable model for multiple interactive optimizations, thereby continuously improving the quality and accuracy of personalized replies and increasing user satisfaction.

[0027] In summary, the present invention can effectively improve the user interaction experience, realize high-quality, diversified and personalized response generation, and ensure the accuracy and consistency of video content through a complete anomaly detection and optimization mechanism, which significantly enhances the practicality and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 The present invention provides a flowchart according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the scope of protection claimed by the present invention and are only illustrative embodiments.

[0031] One embodiment of the present invention provides a method for providing personalized responses to users based on a multimodal large model, comprising: obtaining a first instruction request from a user; determining at least one as a desired mode from a plurality of candidate modes according to the first instruction request from the user, the plurality of candidate modes comprising: generating text from text, generating pictures from text, generating videos from text, and generating sounds from text; obtaining the first text request and the second instruction request from the user, and determining at least one first operating model from a plurality of candidate models pre-set in the desired mode according to the second instruction request to respond to the first text request of the user; and outputting a first personalized response corresponding to the first text request through the operating model.

[0032] The multimodal large model is a large-scale artificial intelligence model that can process and understand data from multiple modalities (such as text, images, audio, video, etc.). It can not only process a single type of data (such as only processing images or only processing text), but also integrate data from multiple modalities for joint analysis and generation. The multimodal large model provided by the present invention can receive text input and generate corresponding videos, or generate images or even more complex outputs by combining text descriptions.

[0033] For example, the command "Generate video based on text" is input through the interface. According to the user's command, the system selects "Generate video based on text" from "Generate text based on text / Generate picture / Generate video / Generate sound". In this embodiment, the system analyzes the content of the user's request and determines that the user wants to obtain a video. The system further obtains the user's text request, such as the specific question input by the user: "Provide a video about the main events in China's modern history, which should not exceed 1 minute and include at least 1 war event and 1 cultural event". The user then enters the command request, requiring "stable-diffusion-3-medium", which becomes the operation model for responding to the user's text request in the future.

[0034] The system provides the generated video to the user in the form of a personalized reply, and the user can see the corresponding video demonstration. This system based on a multimodal large model can flexibly generate personalized replies in different modalities (such as text, images, videos or audio) according to different user needs.

[0035] According to another embodiment of the present invention, it also includes obtaining the user's second text request and a third instruction request, determining at least one second operating model from multiple candidate models to respond to the user's second text request and the first personalized reply according to the third instruction request; and outputting a second personalized reply corresponding to the second text request and the first personalized reply through the operating model.

[0036] For example, when the user is not satisfied with the first personalized answer generated or wants to continue the operation based on the first personalized answer generated, the system can further meet the user's needs. For example, the user inputs a specific question: "Based on the results of the above answer, continue to provide a video about the main events in China's modern history, which should not exceed 2 minutes and include at least 2 war events and 2 cultural events, of which at least one war event and at least one cultural event must be consistent with the above answer", and the user then enters an instruction request, requiring "stable-diffusion-4-medium", which becomes the operating model for the subsequent response to the user's text request and the first personalized answer.

[0037] The system can further operate the first personalized response generated according to the user's specific needs to meet the user's customization requirements. In addition, the system can effectively associate content in different modes (such as historical event text and video content) to ensure that the generated content is consistent in semantics and themes. This association between modalities helps improve the user experience and avoid fragmentation or incoherence between generated content.

[0038] According to another embodiment of the present invention, when the expected mode is to generate a video based on text, the first operating mode includes: extracting a text vector from a first text request, the text vector includes a sentence vector and a word vector, the sentence vector represents the overall semantic information of the first text request, and the word vector is used to represent the semantic information of each word of the first text request; sampling a random noise vector from a normal distribution; concatenating the sentence vector and the noise vector to generate a latent variable, and the latent variable serves as an input of a generator; using the generator to generate a time-continuous video frame sequence from the latent variable using a first convolution filter and a second convolution filter, wherein the video frame sequence is consistent with the overall semantic information; using the word vector to optimize the local area of ​​the video frame, wherein the local details of the video frame match the semantic information of a single word; and outputting the video generated based on the first text.

[0039] In this embodiment, sampling a random noise vector from a normal distribution includes the following steps:

[0040] 1. Determine the parameters of the normal distribution: The normal distribution is defined by two main parameters: Mean μ: represents the center of the distribution and is set to 0. Standard deviation σ: represents the width or dispersion of the distribution and is set to 1. The random noise vector of the normal distribution is sampled from a standard normal distribution (mean 0, standard deviation 1).

[0041] 2. Define the dimension of the vector: The dimension n of the noise vector is pre-set, which determines the size of the sampled random noise vector. In this embodiment, the dimension of the noise vector is 100, which is used to provide sufficient randomness for the generator.

[0042] 3. Sampling a noise vector from a normal distribution: Using Python (using the NumPy library), call the np.random.normal function to generate random numbers from a normal distribution with a given mean (0) and standard deviation (1). The size of the generated noise vector is 100 dimensions.

[0043] 4. Feed the noise vector into the generator: Once the noise vector is sampled from the normal distribution, it is passed as input to the generator. The generator generates output data (video frames) based on this random noise vector and other inputs.

[0044] 5. Updates during training: During the training of the generative adversarial network, the sampling of noise vectors is a repetitive process. Each time the generator samples a new noise vector from a normal distribution, which ensures that the generated samples are diverse. The weights of the generator are continuously updated through training, and ultimately realistic outputs can be generated.

[0045] The first convolution filter used in this embodiment can be a 3D convolution filter and a deconvolution. The specific implementation method of generating a series of video frame sequences from latent variables through the first convolution filter involves a generative model in deep learning for video generation tasks. The key to this step is to use 3D convolution to process time series information, and combine the deconvolution operation (also called transposed convolution) to generate a temporally continuous and spatially consistent video frame sequence. The following is a specific implementation method of this process:

[0046] 1. Input latent variables: First, the latent variables (a combination of text embedding and noise vector) obtained from the previous step are used as input to the generator network. The latent variable z is a high-dimensional random vector that carries the initial information of the generated video.

[0047] 2.3D convolution filter: 3D convolution can operate simultaneously in both time and space dimensions, which is essential for generating a temporally continuous sequence of video frames. The convolution operation of the 3D convolution kernel not only convolves the width and height of each frame, but also convolves along the time dimension, thereby capturing the relationship between each frame in the video and the previous and next frames.

[0048] The core operation of 3D convolution: The size of the 3D convolution kernel is (kd, kh, kw), where kd is the kernel size of the time dimension, and kh and kw are the kernel sizes of the spatial dimension. The convolution operation generates a feature map containing temporal information, thereby generating a temporally continuous frame sequence layer by layer.

[0049] 3. Deconvolution (transposed convolution) operation: Deconvolution (i.e. transposed convolution) is used to upsample feature maps and expand them to a larger spatial resolution, thereby generating a higher resolution video frame sequence. In video generation, deconvolution operations are used to gradually expand the size of video frames until the target resolution is reached.

[0050] The core operation of deconvolution: The role of the deconvolution layer (ConvTranspose3d) is to gradually upsample the low-resolution feature map to generate a larger frame. Deconvolution expands low-dimensional features into high-resolution video frames by upsampling the input.

[0051] 4. Generate video frame sequence: After several rounds of 3D convolution and deconvolution operations, the generator gradually generates a temporally continuous sequence of video frames from the latent variables. These frames not only contain spatial features (visual information of each frame), but also capture the temporal dependencies between frames through 3D convolution. The format of the video frame is (batch_size, channels, depth, height, width), where depth represents the temporal dimension of the frame, that is, the number of frames in the video, height and width represent the resolution of each frame, and channels is usually 3, representing the RGB channels.

[0052] 5. Stepwise upsampling and generation of high-resolution frames: In order to generate high-resolution video frame sequences, 3D convolution and deconvolution operations are usually performed layer by layer. Through layer-by-layer deconvolution operations, the feature map is expanded from low resolution to high resolution.

[0053] 6. Output of video frame sequence: Finally, the generated video frame sequence includes multiple time-continuous frames, each of which contains visual information at the specified resolution and number of channels. The video sequence can be further output to a video file or used in other application scenarios (analysis).

[0054] Using 3D convolution filters to capture time series information and combining deconvolution operations to gradually upsample feature maps, it is possible to generate a time-continuous sequence of video frames from latent variables. 3D convolution is responsible for extracting features in both time and space dimensions, while deconvolution upsamples layer by layer to generate high-resolution videos. Through multiple layers of 3D convolution and deconvolution operations, realistic and time-continuous video sequences can be generated.

[0055] The core idea of ​​using word vectors to optimize local areas of video frames is to correspond and match the semantic information of words in text descriptions with local areas of video frames to ensure that the local details of the video are consistent with the semantics of the words. This can be achieved by the following steps:

[0056] 1. Preprocessing of input text and video frames

[0057] a. Extract word vectors: Extract the vector representation of each word from the input text description. These word vectors can be extracted through a pre-trained language model. Each word vector represents the semantic information of the word.

[0058] b. Extract local features of video frames: Take the video frames as input and use convolutional neural networks to extract local region features of each frame. Feature extraction can be achieved through deep convolutional networks (such as ResNet, VGG, etc.), and each local region feature represents the visual information of the region.

[0059] 2. Calculate the similarity between word vectors and local features of video frames

[0060] a. Alignment of word vectors with local features

[0061] Align the extracted word vectors with the local features of the video frame. For each word (such as "red", "car", etc.), we match its vector with the features of the corresponding local region of the video frame.

[0062] b. Similarity calculation

[0063] Use cosine similarity or other similarity measurement methods to calculate the similarity between word vectors and local region features to determine the degree of match between word semantics and local visual features. Cosine similarity is used to measure the similarity between vectors.

[0064] 3. Optimize local areas of video frames based on similarity

[0065] a. Weighted Fusion

[0066] Based on the calculated similarity, the features of the local area of ​​the video frame are adjusted to make it more consistent with the semantics of the word vector. The similarity can be used as a weighting coefficient to fuse the local area features with the word vector to generate details that are more consistent with the text description.

[0067] Example: If a word vector represents “red”, then a region matching that word vector (like car) would reinforce visual features associated with the color red.

[0068] b. Generate optimized local area

[0069] The optimized local features are used to generate specific details of the video frame, ensuring that these areas are consistent with the input word semantics. For example, the generated red car matches the word vector of "red".

[0070] 4. Optimized video frame generation

[0071] The optimized local area features are integrated into the video frames. The generated frames not only match the text description as a whole, but the local details (such as "red car", "road", etc.) are also more consistent with the word semantics.

[0072] 5. Video frame output

[0073] Finally, the optimized feature map is restored to a high-resolution video frame through deconvolution or other generative operations. The optimized local area is visually consistent with the corresponding word semantics, so that the generated video conforms to the text description in local details.

[0074] According to another embodiment of the present invention, a method for evaluating the consistency between a video frame sequence and overall semantic information is also provided: a first discriminator is set to evaluate the consistency between the video as a whole and the overall semantic information. A second discriminator is also set to analyze the video frames frame by frame to evaluate the authenticity of each frame and the matching degree with the semantic information of a single word.

[0075] The video generated according to the first text is optimized through the following steps so that the generated video is visually realistic and semantically consistent with the description of the first text request: the generation loss of the generator is calculated, the generation loss of the generator is based on the determination result of the second discriminator on the authenticity of the generated video, and the goal of the generator is to minimize the generation loss to generate a realistic video; the global consistency loss is calculated, and the global consistency loss ensures that the generated video is consistent with the first text request at the global level by comparing the difference between the global features of the generated video and the overall semantic information of the first text request; the local consistency loss is calculated, and the local consistency loss ensures that the generated video is consistent with the first text request in details by comparing the local area features of the generated video with the semantic information of a single word; the generator, the first discriminator and the second discriminator are trained by jointly generating the loss, the global consistency loss and the local consistency loss, and the joint loss is minimized to optimize the visual quality and semantic consistency of the generated video.

[0076] In this embodiment, the task of the discriminator is to distinguish between real videos and generated videos, and to determine whether the generated videos semantically match the text description. The goal of the discriminator is to ensure that the generator cannot successfully "fool" it.

[0077] Input: The discriminator receives two types of input:

[0078] Real video: a sequence of video frames that actually exist, used as a baseline for comparison.

[0079] Generated Video: A sequence of video frames produced by the generator.

[0080] Output: The discriminator gives a probability value indicating whether the input video is real or generated. In addition, the discriminator also evaluates whether the generated video matches the given text description.

[0081] The discriminator usually consists of two sub-networks:

[0082] The first discriminator is used to evaluate the authenticity of the entire video sequence and its semantic consistency with the text description. The second discriminator analyzes each frame of the video frame by frame to check its authenticity and the temporal consistency between frames.

[0083] The adversarial training between the generator and the discriminator is a continuous optimization process. The generator and the discriminator update their weights alternately in order to continuously improve performance through mutual competition.

[0084] Step 1: Generator generates video

[0085] The generator takes in a text vector and a noise vector and generates a sequence of video frames that matches the input text description. The generator tries to generate videos that look real enough to “fool” the discriminator.

[0086] Step 2: The discriminator evaluates the video

[0087] The discriminator receives real videos and videos generated by the generator and evaluates them separately.

[0088] The goal of the discriminator is to identify which videos are generated and which are real videos. The discriminator also needs to evaluate the semantic consistency between the generated video and the text description. The discriminator outputs a probability value to indicate whether the input video is generated or real, and calculates the loss value based on the evaluation result.

[0089] Step 3: Calculation of loss function

[0090] The generator and discriminator each have independent loss functions.

[0091] Generator loss: The goal of the generator is to minimize the probability that the discriminator judges its generated video as "fake". The generator's loss function is calculated by the discriminator's evaluation of the generated video. If the video generated by the generator is recognized as real by the discriminator, its loss is small, otherwise the loss is large.

[0092] Discriminator loss: The discriminator loss function is calculated based on its evaluation results of real videos and generated videos. The discriminator needs to minimize its misjudgment of real videos and its misjudgment of generated videos.

[0093] Step 4: Parameter Update

[0094] Generator update: The generator updates its parameters according to the gradient of the loss function through the back-propagation algorithm. The generator optimizes its weights to make the generated video more realistic and match the text description in order to deceive the discriminator.

[0095] Discriminator update: The discriminator also updates its parameters according to the gradient of the loss function through the back-propagation algorithm, with the aim of improving its ability to identify generated videos while ensuring that it can accurately determine the semantic consistency between the video and the text description.

[0096] Step 5: Circuit Training

[0097] The training of the generator and the discriminator is carried out alternately, the generator tries to generate more realistic videos, and the discriminator continuously improves its recognition ability. After multiple rounds of iterations, the generator gradually generates more and more realistic videos, and the discriminator becomes more accurate.

[0098] 4. Introducing adversarial loss and consistency loss

[0099] In addition to generating adversarial losses, this embodiment also introduces consistency losses to ensure that the generated video is not only visually realistic, but also highly semantically consistent with the input text description. These losses help the generator generate video content that is more semantically consistent.

[0100] The adversarial training between the generator and the discriminator is achieved in a competitive manner, where the generator tries to generate videos that are as realistic as possible, while the discriminator continuously improves its ability to identify generated videos and real videos. The generator optimizes its own output by minimizing the loss of the discriminator, while the discriminator continuously improves by maximizing its ability to identify generated videos. Ultimately, the generator is able to generate high-quality, realistic videos that are consistent with the text description, while the discriminator is able to accurately distinguish between generated videos and real videos.

[0101] In this embodiment, the input is a sentence ∫ and a noise vector z(z∈R 100 ), we design a generator network G to generate a series of frames:

[0102] {R d∫,dz}→R dc×dl×dh×dw}

[0103] Where dc, dl, dh, and dw represent the number of channels, sequence length, frame height, and frame width, respectively. In order to capture the spatial and temporal information of the video, 3D convolution filters and deconvolution operations are used, while 2D convolution filters are used to synthesize spatial information and ensure temporal coherence between adjacent frames. First, a fully connected layer is used to learn a unified embedding m, which is obtained by concatenating the text embedding e and the noise variable. The unified embedding undergoes feature transformation as shown in the following formula:

[0104] m=W e [ze]∈R dm+dz

[0105] Among them, W e ∈R dm+dz is the transformation matrix. Then, the generator G takes this latent variable as input and generates the corresponding video, as shown in the following formula:

[0106] Q=G(m)∈R dc×dl×dh×dw

[0107] Here, Q is the generated video, Fi is the i-th frame of the generated video, and F i ∈R dc×dh×dw .

[0108] 3.3 Discriminator

[0109] To ensure that the generated videos are realistic and temporally coherent between adjacent frames, we use a two-layer discriminator setting: the first discriminator and the second discriminator.

[0110] 3.3.1 The first discriminator D1 (Q ^ ,e)

[0111] The discriminator D1 receives two inputs: the video tensor Q^ and the text embedding e. First, it processes the video input through a 3D convolutional layer to generate a video-level tensor T Q^ ∈R dc×dl×dh×dw . This tensor represents high-level features extracted from the video, covering different spatiotemporal locations. Next, the video-level tensor is enhanced with the text embedding e, combined with semantic information, and processed through a dense layer with a Softmax activation function to distinguish whether the input video comes from real data (real video) or generated data (synthetic video), and whether it is semantically consistent with the given text description. The process is shown in the following formula:

[0112] D1(Q^,e)→[0,1]

[0113] Unlike traditional discriminators, the additional requirement here is to maintain the semantic relationship between the generated video and the corresponding subtitles. Therefore, a conditional discriminator is needed that not only evaluates the authenticity of the video, but also whether the video is consistent with the given text description. The loss function of the discriminator is shown as follows:

[0114]

[0115] 3.3.2 Second Discriminator D2(Q^,e)

[0116] To further improve the authenticity of the frames and their semantic consistency with the given text, and maintain temporal coherence, we use a second discriminator. This discriminator actually consists of two sub-networks: the frame discriminator (Dframe) and the motion discriminator (Dmotion). The frame discriminator is responsible for judging the authenticity of each frame of the video and evaluating whether it maintains semantic consistency with the given text description. On the other hand, the motion discriminator is responsible for checking the temporal coherence in the video and evaluating the smoothness and natural fluidity of the motion between adjacent frames.

[0117] Frame discriminator: D frame (F i ,e)

[0118] First, a 2D convolutional model is used to extract the frame-level tensor T for each frame. F ∈R dc×dh×dw This frame-level tensor is then combined with the text embedding e and fed into the frame discriminator D frame To distinguish the authenticity of the frame and the semantic consistency with the subtitles. Its loss function is as follows:

[0119]

[0120] Motion Discriminator: D motion

[0121] In order to keep adjacent frames consistent in time, the Euclidean distance of the inter-frame tensor is used to calculate the similarity between adjacent frames. The size of the motion tensor is shown in the following formula:

[0122]

[0123] The temporally coherent adversarial loss function is as follows:

[0124]

[0125] 3.4 Optimization

[0126] The overall optimization of the discriminator can be achieved by minimizing the combined loss at the video level and the frame level while considering the temporal coherence loss, as shown in the following formula:

[0127]

[0128] By minimizing the above losses, the discriminator can identify the authenticity of the video and its frames, whether the video is semantically consistent with the description text, and also identify the temporal changes between frames. For the generator network, the adversarial losses at the video level and frame level are as follows:

[0129]

[0130] In addition, VTM loss LVTML is introduced to further strengthen the requirement for fine-grained consistency between videos and description texts. The final objective function of the generator is as follows:

[0131]

[0132] This generator objective function enables the generator to generate videos that are realistic and consistent with the description text.

[0133] According to another embodiment of the present invention, when the expected mode is to generate a video based on text, before outputting a first personalized response corresponding to the first text request by running the model, the step of confirming that the generated video does not contain abnormal frames is included, and the abnormal frames are abnormal events that do not conform to the normal mode or expected behavior. Confirming that the generated video does not contain abnormal frames is performed by the following steps:

[0134] The feature extraction of the generated video is performed through an autoencoder, and the autoencoder includes: an encoder, which is used to extract the channel features of the generated video through convolution operations, and extract the spatial structure features of the generated video through a self-attention mechanism; a decoder, which is used to reconstruct the video frame according to the extracted features; the extracted features are clustered through a dual-scale feature clustering module, and the clustering module simultaneously compresses the channel features and spatial structure features of the generated video to increase the distinguishing boundary between normal data and abnormal data; in the task of reconstructing video frames, the occurrence of abnormal events is detected by calculating the average frame anomaly score of the video clip; by setting a threshold, the reconstruction error of the video clip is compared to confirm that the generated video does not contain abnormal frames.

[0135] In this embodiment, abnormal frames in the video refer to frames that do not conform to normal patterns or expected behaviors, which may be caused by errors or abnormal behaviors in the generation process. When generating personalized videos, the model generates corresponding video clips according to the input first text request. In the process of generating video clips, in order to ensure that the generated video meets the user's expectations and does not contain abnormal frames, the present invention adds an abnormality detection step before outputting the video.

[0136] Feature extraction of generated videos through autoencoders

[0137] After the video is generated, in order to detect whether there are abnormal frames, the system first uses an autoencoder to extract features from the generated video. The autoencoder is an unsupervised neural network architecture, including an encoder and a decoder, which can learn high-level features of the input data and reconstruct the input.

[0138] Encoder: The encoder part is used to extract features from video clips. The encoder extracts channel features and spatial structure features of the video through two mechanisms:

[0139] Channel feature extraction: Through convolution operations (such as convolutional neural networks, CNN), features are extracted for each pixel channel of the video frame. These channel features contain information such as color, texture, and edge in the video frame. Each frame of video is processed through multiple layers of convolution operations to generate multiple feature channels, each of which reflects different attributes in the video.

[0140] Extraction of spatial structural features: The spatial relationship and structural features of the video are extracted through the self-attention mechanism (such as the Swin Transformer framework). The self-attention mechanism captures the local and global spatial layout information in the video frame by analyzing the interdependencies between different pixels in the video frame. Through the self-attention operation of local windows and moving windows, the model can better understand the layout and movement of objects in the video.

[0141] Reconstruct video frames through the decoder

[0142] The decoder reconstructs the video frame based on the features extracted by the encoder. The decoder gradually restores the spatial and channel information of the video through deconvolution operations, maps the features back to the original image dimensions, and generates a reconstructed frame that is similar to the input frame.

[0143] Reconstruction process: The reconstruction process of the autoencoder attempts to restore the extracted features to the original video frame. For normal video data, the encoder and decoder can extract and reconstruct features well, so the reconstruction error is small; however, for data containing abnormal events, the model cannot accurately reconstruct because it has never seen abnormal data, resulting in a large reconstruction error.

[0144] Feature clustering via dual-scale feature clustering module

[0145] In order to further enhance the ability to distinguish normal and abnormal data, the channel features and spatial structure features extracted by the autoencoder are processed by a dual-scale feature clustering module.

[0146] Clustering of channel features: Channel features represent different attributes of video frames, such as color, texture, etc. Through K-Means clustering, the clustering module compresses channel features and finds the cluster center representing the normal mode. Compression of channel features can remove redundant information and retain only the most representative features, which helps to increase the difference between normal and abnormal data.

[0147] Clustering of spatial structural features: Spatial structural features describe the spatial relationship between pixels in a video frame. By clustering these spatial features, the module can further compress the spatial pattern of normal data, making the normal data more compact in the feature space. The spatial features of abnormal data often do not conform to this pattern and are difficult to cluster, resulting in large errors during reconstruction.

[0148] Through dual-scale clustering, the clustering module effectively compresses the features of the video and increases the distinction between normal data and abnormal data, making the abnormal data more obvious in the feature space.

[0149] Computing the average frame anomaly score in the reconstruction task

[0150] In order to detect the occurrence of abnormal events, the system evaluates the degree of abnormality by calculating the average frame anomaly score of the video clips in the task of reconstructing video frames.

[0151] Calculation of anomaly score: For each reconstructed frame of a video clip, the system calculates the reconstruction error of each frame, that is, the difference between the original frame and the reconstructed frame. In this embodiment, the Euclidean distance is used to measure this difference. In order to improve the stability of detection, the system not only calculates the reconstruction error of a single frame, but also averages the anomaly score of each frame in the video clip to obtain the average anomaly score of the video clip.

[0152] The significance of anomaly score: The reconstruction error of normal video clips is usually low, so the average anomaly score will also be low; while for video clips containing abnormal events, the reconstruction error is large, resulting in a higher average anomaly score. By calculating the average frame anomaly score, the system can more accurately determine the occurrence of abnormal events.

[0153] Compare the reconstruction error by setting a threshold

[0154] After calculating the reconstruction error or average anomaly score of the video clip, the system uses a preset threshold to determine whether there are abnormal frames.

[0155] Set threshold: The threshold is determined based on the reconstruction error of normal data. The system will count the reconstruction error range of normal data during the training phase and select a suitable error value as the threshold. Generally, the reconstruction error of normal data will not exceed this threshold.

[0156] Determine abnormal frames: If the average frame anomaly score or reconstruction error of a video clip exceeds the threshold, the video clip contains abnormal frames; if the reconstruction error is lower than the threshold, the video clip is considered to have no abnormal frames. In this way, the system can effectively filter out abnormal frames in the generated video and process them.

[0157] Confirm that the generated video does not contain abnormal frames

[0158] Finally, the system performs anomaly detection on the generated video through the above steps to confirm whether there are abnormal frames in the video clip. After confirming that there are no abnormal frames in the video, the system outputs the video clip as the final generated result. If an abnormal frame is detected, the system can regenerate the video or correct the abnormal frame.

[0159] This method extracts and reconstructs features from videos through an autoencoder, and combined with a dual-scale feature clustering module, can effectively detect abnormal frames in generated videos. By setting a threshold to compare reconstruction errors and calculate the average frame anomaly score, the system can accurately determine the occurrence of abnormal events and confirm the quality of the generated video. This method effectively improves the accuracy of video generation and avoids unexpected abnormal behaviors during the generation process.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for providing personalized responses to users based on a multimodal large model, characterized in that: include: Get the user's first command request; Determining at least one desired mode from a plurality of candidate modes according to the first instruction request of the user, the plurality of candidate modes comprising: generating text from text, generating pictures from text, generating videos from text, and generating sounds from text; Acquire a first text request and a second instruction request of the user, and determine at least one first operation model from a plurality of candidate models pre-set in the expected mode according to the second instruction request to respond to the first text request of the user; A first personalized response corresponding to the first text request is outputted through the running model.

2. The method according to claim 1, characterized in that Also includes: Obtaining a second text request and a third instruction request of the user, and determining at least one second operating model from the multiple candidate models to respond to the second text request and the first personalized reply of the user according to the third instruction request; A second personalized answer corresponding to the second text request and the first personalized answer is outputted through the running model.

3. The method according to claim 1, characterized in that When the desired mode is to generate a video based on text, the first operation mode includes: Extracting a text vector from the first text request, the text vector comprising a sentence vector and a word vector, the sentence vector representing overall semantic information of the first text request, and the word vector being used to represent semantic information of each word of the first text request; Sample a random noise vector from a normal distribution; Concatenate the sentence vector and the noise vector to generate a latent variable, and use the latent variable as an input of the generator; generating a temporally continuous video frame sequence from the latent variable using a first convolution filter and a second convolution filter by the generator, wherein the video frame sequence is consistent with the overall semantic information; optimizing a local region of the video frame using the word vector, wherein the local details of the video frame match the semantic information of the single word; Output a video generated according to the first text.

4. The method according to claim 3, characterized in that: The consistency of the video frame sequence with the overall semantic information is evaluated by the following steps: A first discriminator is set to evaluate the consistency of the video as a whole and the overall semantic information.

5. The method according to claim 4, characterized in that The consistency of the video frame sequence with the overall semantic information is evaluated by the following steps: A second discriminator is set to analyze the video frames frame by frame to evaluate the authenticity of each frame and the matching degree with the semantic information of the single word.

6. The method according to claim 5, characterized in that The video generated according to the first text is optimized by the following steps, so that the generated video is visually realistic and semantically consistent with the description of the first text request: Calculating a generation loss of the generator, wherein the generation loss of the generator is based on a determination result of the authenticity of the generated video by a second discriminator, and a goal of the generator is to minimize the generation loss to generate a realistic video; Calculating a global consistency loss, wherein the global consistency loss ensures that the generated video is consistent with the first text request at a global level by comparing the difference between the global features of the generated video and the overall semantic information of the first text request; Calculating a local consistency loss, wherein the local consistency loss ensures that the generated video is consistent with the first text request in details by comparing the local region features of the generated video with the semantic information of the single word; The generator, the first discriminator and the second discriminator are trained by jointly training the generation loss, the global consistency loss and the local consistency loss, and the joint loss is minimized to optimize the visual quality and semantic consistency of the generated video.

7. The method according to any one of claims 1 to 6, characterized in that: When the desired mode is to generate a video based on text, before outputting a first personalized response corresponding to the first text request through the running model, the step further includes: Confirm that the generated video does not contain abnormal frames, where the abnormal frames are abnormal events that do not conform to normal patterns or expected behaviors.

8. The method according to claim 7, characterized in that Verify that the generated video does not contain abnormal frames by following these steps: The generated video is subjected to feature extraction by an autoencoder, wherein the autoencoder comprises: An encoder, configured to extract channel features of the generated video through a convolution operation, and to extract spatial structural features of the generated video through a self-attention mechanism; A decoder, used to reconstruct video frames based on the extracted features; Clustering the extracted features through a dual-scale feature clustering module, the clustering module simultaneously compressing the channel features and spatial structure features of the generated video to increase the distinguishing boundary between normal data and abnormal data; In the video frame reconstruction task, the occurrence of the abnormal event is detected by calculating the average frame anomaly score of the video clip; By setting a threshold and comparing the reconstruction errors of the video clips, it is confirmed that the generated video does not contain the abnormal frame.

Images