Video plot question and answer method and device based on RAG

Abstract

The invention discloses a video plot question answering method and device based on RAG, and relates to the field of RAG. The method comprises the following steps: acquiring user content input by a user for video content; converting the user content into a corresponding user vector; calculating a first similarity between the user vector and the first target vector and a second similarity between the user vector and the second target vector, wherein the preset vector database comprises a plurality of target vectors; if it is determined that the first similarity is greater than the second similarity, determining first target text content corresponding to the first target vector according to a preset vector database; and inputting the first target text content and the user content into a preset RAG model, and generating reply content for the user content. By implementing the technical scheme provided by the invention, the questions of the user can be answered for the video file by vectorizing the user content and the video file.

Description

Technical Field

[0001] The present application relates to the field of RAG, and in particular to a video plot question-and-answer method and device based on RAG. Background Art

[0002] With the rapid development of Internet technology and the explosive growth of video content, users are increasingly relying on video platforms to obtain information and entertainment content. Video content not only enriches people's daily lives, but also becomes an important way to acquire knowledge. However, in the massive amount of video data, it is often difficult for users to quickly obtain the specific information they need. This demand has led to the widespread application of video plot question-and-answer systems, which help users quickly obtain relevant information and answers while watching videos.

[0003] At present, related technologies mainly rely on rule matching and simple text retrieval technology for user questions input by users. Related technologies can only process pure text content, and have limited ability to parse and understand complex multimodal data (such as videos), and cannot answer questions input by users based on video content.

[0004] Therefore, there is an urgent need for a video plot question-answering method and device based on RAG. Summary of the invention

[0005] The present application provides a RAG-based video plot question-and-answer method and device, which realizes the parsing and understanding of video files by vectorizing the user content and video files, and can answer the user's questions regarding the video files.

[0006] In a first aspect of the present application, a RAG-based video plot question-and-answer method is provided, the method comprising: obtaining user content input by a user for video content; converting the user content into a corresponding user vector; calculating a first similarity between the user vector and a first target vector and a second similarity between the user vector and a second target vector, the first target vector being any target vector in a preset vector database, the second target vector being any target vector other than the first target vector in the preset vector database, the preset vector database comprising a plurality of the target vectors, and the preset vector database being pre-constructed according to the video content; if it is determined that the first similarity is greater than the second similarity, determining a first target text content corresponding to the first target vector according to the preset vector database; inputting the first target text content and the user content into a preset RAG model to generate reply content for the user content.

[0007] By adopting the above technical solution, by converting the user input content into a user vector and calculating the similarity between the user vector and the target vector in the preset vector database, the method can achieve matching and understanding of the user input content and the video content. By comparing the similarity, the most matching target vector and the corresponding target text content are determined. By inputting the user input content and the target text content into the preset RAG model, the reply content for the user content is generated. This can enhance the ability of video processing, so that it can respond to user needs and questions more accurately and intelligently.

[0008] Optionally, before obtaining user content input by the user, the method also includes: obtaining a video file corresponding to the video content; converting the video file into an audio file using a transcoding tool; performing role separation on the audio file to obtain roles and audio content corresponding to the roles; converting the audio content into text content, and setting corresponding role tags for the text content; extracting the video file using a preset multimodal large model, extracting key frames, and converting the key frames into background descriptions and plot descriptions for the video file; integrating the text content, the role tags corresponding to the text content, the background description, and the plot description in chronological order to obtain target text content; converting the target text content into a target vector, and storing the target vector, the target text content, and the correspondence between the target vector and the target text content in a preset vector database.

[0009] By adopting the above technical solution, a transcoding tool is used to convert the video file into an audio file in preparation for subsequent audio processing and text conversion. The audio file is separated into roles, and the audio content of different roles is separated. In this way, the audio content of different roles in the video can be obtained. The separated audio content is converted into text content, and corresponding role tags are set for the text content to identify which role the text content belongs to. In this way, the text content of different roles can be distinguished in subsequent processing. The video file is processed using a preset multimodal large model to extract the key frames of the video. These key frames are then converted into background descriptions and plot descriptions for the video file. The text content, role tags, background descriptions and plot descriptions are integrated in chronological order to obtain the target text content. In this way, the various elements can be integrated together to form a complete text (target text content) that describes the video content. The target text content is converted into a target vector, and the corresponding relationship between the target vector and the target text content is stored in a preset vector database. In this way, the target text content and its corresponding vector form can be associated and stored, which is convenient for subsequent processing.

[0010] Optionally, converting the user content into a corresponding user vector specifically includes: obtaining a historical chat record corresponding to the user content; extracting the historical chat record to obtain historical key information; splicing the user content and the historical key information to obtain spliced ​​target user content; and converting the target user content into the user vector.

[0011] By adopting the above technical solution, by obtaining the user's historical chat records and extracting historical key information, the user's current input user content is spliced ​​with the historical key information, and the user's current request is combined with the past interaction context to obtain a more comprehensive and accurate semantic representation. Converting the target user content into a representation of the user vector is helpful for subsequent similarity calculation and matching. Through vectorization, user content can be converted into a computable and comparable numerical representation, thereby achieving more accurate similarity matching and reply generation.

[0012] Optionally, the historical chat records are extracted to obtain historical key information, specifically including: performing entity recognition on the historical chat records to obtain corresponding historical entities, and performing entity recognition on the user content to obtain corresponding content entities; if it is determined that the historical entity and the content entity are the same entity, then determining the historical chat records corresponding to the historical entity as the historical key information.

[0013] By adopting the above technical solution, the historical entity and the content entity are compared and matched. If it is determined that the historical entity and the content entity are the same entity, that is, they represent the same thing, then the historical chat record containing the historical entity can be determined as historical key information. Historical key information provides important contextual information related to the user content, which helps to better understand the user's current request or intention.

[0014] Optionally, a preset multimodal large model is used to extract the video file to extract key frames, specifically including: determining multiple image frames in the video file; calculating the image similarity between a first image frame and a second image frame, the first image frame and the second image frame being any two adjacent image frames among the multiple image frames; if it is determined that the image similarity is less than a preset similarity threshold, determining that the first image frame and the second image frame are key frames.

[0015] By adopting the above technical solution, the key frames in the video file can be determined by calculating the image similarity and applying the preset similarity threshold. Key frame extraction provides a basis for subsequent video content analysis tasks.

[0016] Optionally, the key frames are converted into background descriptions and plot descriptions for the video file, specifically comprising: performing feature extraction on the key frames to obtain corresponding visual feature vectors; inputting the visual feature vectors into a preset image description generation model to obtain text descriptions for the key frames; performing keyword extraction on the text descriptions to determine corresponding keywords; inputting the keywords into a preset keyword library to obtain description types corresponding to the keywords, the description types including background descriptions and plot descriptions, and the preset keyword library including correspondences between keywords and description types.

[0017] By adopting the above technical solution, by extracting features from key frames, obtaining visual feature vectors, and inputting them into a preset image description generation model, a targeted and accurate text description can be generated. The text description generated using the visual feature vector can better reflect the video content and provide more detailed and rich background and plot information. By extracting keywords and matching the preset keyword library, background descriptions and plot descriptions can be accurately distinguished. This refined description classification helps to better understand the different aspects of video content and improve the overall content comprehension ability.

[0018] Optionally, after inputting the first target text content and the user content into a preset RAG model and generating reply content for the user content, the method further includes: storing the user content and the reply content in a historical chat record.

[0019] By adopting the above technical solution, by storing user content and reply content in historical chat records, historical chat records can be referenced in subsequent conversations to provide more relevant and continuous replies.

[0020] In a second aspect of the present application, a RAG-based video plot question-and-answer device is provided, the device comprising: an acquisition module and a processing module, wherein: the acquisition module is used to acquire user content input by a user for video content; the processing module is used to convert the user content into a corresponding user vector; the processing module is also used to calculate a first similarity between the user vector and a first target vector and a second similarity between the user vector and a second target vector, the first target vector is any target vector in a preset vector database, the second target vector is any target vector in the preset vector database except the first target vector, the preset vector database includes a plurality of the target vectors, and the preset vector database is pre-constructed according to the video content; the processing module is also used to determine a first target text content corresponding to the first target vector according to the preset vector database if it is determined that the first similarity is greater than the second similarity; the processing module is also used to input the first target text content and the user content into a preset RAG model to generate reply content for the user content.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By converting the user input content into a user vector and calculating the similarity between the user vector and the target vector in the preset vector database, this method can achieve matching and understanding of the user input content and the video content. By comparing the similarity, the most matching target vector and the corresponding target text content are determined. By inputting the user input content and the target text content into the preset RAG model, the reply content for the user content is generated. This can enhance the ability of video processing, enabling it to respond to user needs and questions more accurately and intelligently.

[0024] 2. Use transcoding tools to convert video files into audio files in preparation for subsequent audio processing and text conversion. Perform role separation on audio files and separate the audio content of different roles. In this way, the audio content of different roles in the video can be obtained. Convert the separated audio content into text content, and set corresponding role tags for the text content to identify which role the text content belongs to. In this way, the text content of different roles can be distinguished in subsequent processing. Use the preset multimodal large model to process the video file and extract the key frames of the video. Then convert these key frames into background descriptions and plot descriptions for the video file. Integrate the text content, role tags, background descriptions, and plot descriptions in chronological order to obtain the target text content. In this way, the various elements can be integrated together to form a complete text (target text content) that describes the video content. Convert the target text content into a target vector, and store the corresponding relationship between the target vector and the target text content in a preset vector database. In this way, the target text content and its corresponding vector form can be associated and stored for subsequent processing.

[0025] 3. By obtaining the user's historical chat records and extracting historical key information, the user's current input user content is spliced ​​with the historical key information, and the user's current request is combined with the past interaction context to obtain a more comprehensive and accurate semantic representation. Converting the target user content into a user vector representation is helpful for subsequent similarity calculation and matching. Through vectorization, user content can be converted into a computable and comparable numerical representation, thereby achieving more accurate similarity matching and reply generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a RAG-based video plot question-answering method disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a RAG-based video plot question-and-answer device disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0027] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0028] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0029] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0031] This application provides a video plot question-answering method based on RAG, referring to Figure 1 , Figure 1 1 is a flow chart of a RAG-based video plot question-answering method provided in an embodiment of the present application. The method is applied to a server and includes steps S101 to S105, which are as follows: Step S101: obtaining user content input by a user for video content.

[0032] Before step S101, the method also includes: obtaining a video file; converting the video file into an audio file using a transcoding tool; performing role separation on the audio file to obtain roles and audio content corresponding to the roles; converting the audio content into text content, and setting corresponding role tags for the text content; extracting the video file using a preset multimodal large model, extracting key frames, and converting the key frames into background descriptions and plot descriptions for the video file; integrating the text content, the role tags corresponding to the text content, the background description, and the plot description in chronological order to obtain target text content; converting the target text content into a target vector, and storing the target vector, the target text content, and the correspondence between the target vector and the target text content in a preset vector database.

[0033] Specifically, the server obtains the video file uploaded or specified by the user through the network interface or local file system. After obtaining the video file, the server calls the pre-configured transcoding tool to process the video file. The transcoding tool is the open source FFmpeg. The server converts the video file into an audio file through the command line interface or API of the transcoding tool. The server performs role separation on the converted audio file. Role separation separates the voices of different people from the mixed audio signal. The server loads a pre-trained role separation model, such as a deep convolutional network (Deep CNN), and uses the role separation model to infer the audio file. The role separation model automatically identifies the voiceprints of different speakers in the audio by analyzing the audio's spectrum, fundamental frequency, resonance peak and other features, and separates them into different audio tracks. The separated audio tracks correspond to different roles, each role corresponds to its own audio content, and the audio content of each role is stored separately.

[0034] After the character audio content is separated, the server uses speech recognition technology to convert the audio content of each character into text content. The server calls pre-trained speech recognition models, such as DeepSpeech and Wav2Letter, to recognize the audio content of each character. The speech recognition model converts the speech signal into a corresponding text sequence by analyzing the acoustic and linguistic features of the audio content. The recognized text content corresponds one-to-one with the corresponding character audio to form a character-text content pair. The server stores the text content and sets a corresponding character label for each text content segment for subsequent processing.

[0035] While obtaining the character text content, the server also needs to extract key information from the video file, including background description and plot description. This step is achieved by calling the preset multimodal large model. The preset multimodal large model is an AI model that processes multiple modal information such as vision, speech, and text at the same time, such as ViLBERT. The server first uses video analysis tools (such as OpenCV) to decompose the video file into a series of image frames, and then selects key frames that can represent the main content of the video through a key frame extraction algorithm.

[0036] In a possible implementation, a preset multimodal large model is used to extract key frames from a video file, specifically including: determining multiple image frames in the video file; calculating image similarity between a first image frame and a second image frame, where the first image frame and the second image frame are any two adjacent image frames in the multiple image frames; if it is determined that the image similarity is less than a preset similarity threshold, determining that the first image frame and the second image frame are key frames.

[0037] Specifically, the server needs to decode the video file and obtain all the image frames therein. After obtaining all the image frames, the server needs to calculate the similarity between any two adjacent image frames. Similarity is used to measure the similarity between two images in visual content. The image similarity calculation method includes histogram comparison, feature point matching, perceptual hashing, etc. In this solution, the server uses a preset multimodal large model to calculate image similarity. The preset multimodal large model is a model based on deep learning, which can simultaneously process data of multiple modalities such as text, image, audio, etc., and learn the association and mapping relationship between them. The server inputs two adjacent image frames (the first image frame and the second image frame) into the preset multimodal large model. The preset multimodal large model extracts the high-level semantic features of the first image frame and the second image frame through the convolutional neural network (CNN) technology, and then obtains the similarity between the two image frames through feature comparison and similarity calculation. Similarity is usually a real number between 0 and 1, indicating the similarity between two images. The greater the similarity, the closer the content of the two images; the smaller the similarity, the greater the difference in the content of the two images. After calculating the similarity of all adjacent image frames, the server needs to determine which image frames are key frames based on the size of the similarity. The selection criteria for key frames are: the content difference between the previous and next image frames is large, that is, the image frames with lower similarity. A preset similarity threshold is pre-set inside the server to determine whether two image frames are different enough. If the similarity between the first image frame and the second image frame is less than the preset similarity threshold, it is considered that the two image frames have a large change in visual content and can be used as key frames. For example, assuming that the preset similarity threshold is 0.8, if the similarity between the first image frame and the second image frame is 0.7, which is less than the preset similarity threshold, both image frames are marked as key frames. The server traverses all adjacent image frame pairs, performs threshold comparison on their similarities, and finally obtains a set of key frames.

[0038] After selecting the keyframes, the server inputs the keyframes into the preset multimodal large model for analysis. The preset multimodal large model extracts features from the keyframes and obtains the visual features of the image through a convolutional neural network (CNN). The preset multimodal large model visual features are aligned and fused with the text features to form a multimodal semantic representation. Based on the semantic representation, the preset multimodal large model can generate a natural language description of the image content of the keyframes, that is, convert the keyframes into background descriptions and plot descriptions. Background descriptions generally involve static information such as scenes, environments, and objects in the image, while plot descriptions involve dynamic information such as actions, events, and interactions of characters in the image.

[0039] In a possible implementation, the key frames are converted into background descriptions and plot descriptions for the video file, specifically including: performing feature extraction on the key frames to obtain corresponding visual feature vectors; inputting the visual feature vectors into a preset image description generation model to obtain text descriptions for the key frames; performing keyword extraction on the text descriptions to determine corresponding keywords; inputting the keywords into a preset keyword library to obtain description types corresponding to the keywords, the description types including background descriptions and plot descriptions, and the preset keyword library includes correspondences between keywords and description types.

[0040] Specifically, the server inputs the image data of each key frame into the preset multimodal large model, and the preset multimodal large model converts the input image into a visual feature vector of a fixed length (such as 4096 dimensions). The visual feature vector highly condenses the semantic content information of the image. The extracted visual feature vectors of all key frames are temporarily stored in the memory. At the same time, the server batch-inputs the visual feature vectors of the key frames into the preset image description generation model. The preset image description generation model adopts an encoder-decoder structure, takes the visual feature vector as input, and generates the corresponding natural language description text (text description). At the same time, the server establishes a mapping relationship between the generated text description and the corresponding key frame. For the text description generated by each key frame, the server calls the preset keyword extraction module for processing. The module uses algorithms based on TF-IDF, TextRank, etc. to extract several keywords from the text description, and these keywords can usually highly summarize the core content of the description. The server establishes a mapping relationship between the extracted keywords and the corresponding key frames. The server queries the keywords extracted from each key frame in the preset keyword library. The preset keyword library pre-manually defines the description types (background description or plot description) corresponding to various common keywords. For example, "trees" and "buildings" usually correspond to background descriptions, while "running" and "talking" usually correspond to plot descriptions. By matching keywords, it is possible to determine whether a key frame is more likely to be a background description or a plot description. After the above processing steps, each key frame in the video has obtained two pieces of information: background description (keywords) and plot description (keywords).

[0041] For example, suppose a key frame is a picture of two people chatting in a park. The background description generated by the multimodal model may be "This is a sunny park with green trees, benches and fountains", while the plot description may be "Two young people are sitting on a bench and chatting. They look relaxed and happy, and seem to be discussing some interesting topics."

[0042] Finally, the server integrates all the information extracted from the video, including text content, character labels, background descriptions, plot descriptions, etc., in chronological order to form a complete target text content. Since the previous character separation and key frame extraction retain the time information, the server can align them according to the timestamp to ensure that the integrated text content is synchronized with the video content on the timeline. The integrated target text content represents the complete semantic information of the video, but in order to facilitate subsequent retrieval and generation, the server also needs to convert it into a semantic vector representation, namely the target vector. The server uses natural language models such as BERT and GPT to encode the target text content and map it to a high-dimensional semantic space. The encoding process makes full use of the self-attention mechanism in the transformer structure to capture long-distance dependencies and contextual information in the text content. The target vector obtained after encoding is a real-valued vector of fixed length, which contains the semantic features of the target text content.

[0043] To facilitate subsequent retrieval and matching, the server stores the target vector, target text content, and the corresponding relationship between the two in a preset vector database. The preset vector database is a database specifically used to store and retrieve high-dimensional vectors, such as Faiss, Annoy, etc. The server uses the target vector as a key and the target text content as a value to store them in the preset vector database in the form of a key-value pair. During retrieval, the preset vector database can quickly calculate the similarity between the query vector entered by the user and all target vectors, and return the target text content that is most relevant to the query.

[0044] In step S101, in the video plot question and answer method, obtaining user content input by the user is the starting point and basis of the entire question and answer process. User content refers to questions or queries raised by users about the video plot, and is the basis for the server to perform subsequent processing and generate answers. The server needs to provide users with an interactive interface for inputting questions or queries. This interface can be in various forms such as a web page, a mobile application, a chat window, etc. The main purpose is to allow users to ask their own questions conveniently and intuitively. The interactive interface usually includes an input box for users to enter questions in text form. The input box has an appropriate size and style to ensure that users can clearly see what they enter. For example, the server can provide a search box on the web page and display a prompt text "Please enter your question about the video plot" above the search box. Users can enter their own questions in the search box, such as "Who is the protagonist of this movie?" Step S102: converting user content into corresponding user vectors.

[0045] In step S102, the user content is converted into a corresponding user vector, specifically including: obtaining the historical chat records corresponding to the user content; extracting the historical chat records to obtain historical key information; splicing the user content and the historical key information to obtain the spliced ​​target user content; and converting the target user content into a user vector.

[0046] In a possible implementation, historical chat records are extracted to obtain historical key information, specifically including: performing entity recognition on historical chat records to obtain corresponding historical entities, and performing entity recognition on user content to obtain corresponding content entities; if it is determined that the historical entity and the content entity are the same entity, then the historical chat record corresponding to the historical entity is determined to be the historical key information.

[0047] Specifically, when a user enters a question or query, the server processes it as user content. In order to better understand the context and background of the user content, the server needs to obtain the historical chat records corresponding to the user content. The server identifies the current user through the user authentication mechanism and retrieves the user's historical chat records from the chat record database. The chat record database stores the complete history of each user interaction, including the user's question, the reply to the question, and metadata such as timestamps.

[0048] After obtaining the historical chat records, the server needs to extract the historical key information that is most relevant to the current user content. This step is achieved through entity recognition technology. Entity recognition can identify entities (such as names of people, places, and organization names) from text and determine their types. The server performs entity recognition on the historical chat records and the current user content respectively to obtain historical entities and content entities.

[0049] The server performs the same processing and entity recognition steps on the user content currently input by the user, obtains the content entities mentioned in the user's current input, and forms a content entity list. The server matches and compares the historical entity list with the content entity list. The matching method is based on string equality, and the server checks whether there are identical entities in the two lists.

[0050] For example, suppose the historical entity list obtained by the server is ["Zhang San", "Beijing", "Microsoft", "2022"], and the content entity list is ["Zhang San", "Apple", "San Francisco"]. Through matching, the server finds that the entity "Zhang San" appears in both lists, which means that the "Zhang San" currently being discussed by the user and the "Zhang San" mentioned in the previous chat are likely to be the same person. This indirectly reflects that the user's current input is semantically related to some previous chat content.

[0051] For entities that have been matched successfully, the server further locates their original positions in the historical chat records. Taking the above "Zhang San" as an example, the server will extract all historical chat segments containing the entity "Zhang San", which are likely to contain background information related to the current discussion. According to the configured strategy (such as time distance, keyword relevance, etc.), the server selects one or more most relevant segments from all relevant segments as the historical key information currently input. The server splices the extracted historical key information with the user content input by the current user to form a complete user input (target user content) containing contextual information. This spliced ​​target user content can replace the original user input for subsequent semantic understanding, dialogue generation and other tasks.

[0052] Finally, the server converts the target user content into a semantic vector representation, namely the user vector. The server encodes the target user content through the BERT language model. The BERT language model uses the Transformer architecture to capture contextual information and long-distance dependencies in the text through the self-attention mechanism. After the target user content is input into the BERT language model, the BERT language model generates a fixed-length vector representation, namely the user vector, which contains the semantic features of the target user content. The user vector encodes the semantic information of the user content in a compact and continuous form, which is convenient for subsequent retrieval and matching. The server can use the user vector to calculate the similarity with the target vector in the preset vector database to find the background description and plot description most relevant to the user content, thereby generating accurate and coherent question and answer results.

[0053] Step S103: Calculate a first similarity between the user vector and a first target vector and a second similarity between the user vector and a second target vector, where the first target vector is any target vector in a preset vector database, and the second target vector is any target vector in the preset vector database except the first target vector, and the preset vector database includes multiple target vectors.

[0054] In step S103, the server first obtains any two target vectors from the preset vector database, which are called the first target vector and the second target vector. The preset vector database is previously constructed, which stores a large number of target vectors. Each target vector corresponds to the target text content of a video plot, including key information such as background description and plot description. After the server selects the first target vector and the second target vector, the server can calculate their similarity with the user vector. This application uses cosine similarity to calculate similarity. Let the user vector be u, the first target vector be v1, and the second target vector be v2. The calculation formula of cosine similarity is: cos(u, v)=(u·v) / (||u||*||v||) Among them, u·v represents the dot product of two vectors, ||u|| and ||v|| represent the L2 norm of the two vectors respectively. The value range of cosine similarity is [-1, 1]. The larger the value, the closer the directions of the two vectors are, that is, the higher the similarity.

[0055] The server calculates the cosine similarity cos(u, v1) between the user vector u and the first target vector v1, and the cosine similarity cos(u, v2) between the user vector u and the second target vector v2. These two similarity values ​​reflect the semantic relevance between the user content and the video plot descriptions corresponding to the two target vectors. Through similarity calculation, the server can quickly find the video plot clips most relevant to the user content, providing important reference information for subsequent question and answer generation.

[0056] Step S104: If it is determined that the first similarity is greater than the second similarity, determining the first target text content corresponding to the first target vector according to the preset vector database.

[0057] In step S104, the server first needs to compare the similarity between the user vector and the first target vector (first similarity) and the similarity between the user vector and the second target vector (second similarity). The server implements the comparison of the similarity through a conditional judgment statement. If the first similarity is greater than the second similarity, it means that the user vector is more matched with the first target vector, that is, the user content is more related to the first target text content corresponding to the first target vector. At this time, the server will perform subsequent operations to determine the first target text content corresponding to the first target vector according to the preset vector database. The first target text content includes the background description, plot description, etc. of the clip. When it is determined that the first similarity is greater than the second similarity, the server needs to search the text content corresponding to the first target vector in the preset vector database. First, the server uses the first target vector as a query condition and searches in the index of the preset vector database. After obtaining the first target text content corresponding to the first target vector, the server uses it as the output result of step S104 for the subsequent question and answer generation task. The first target text content usually contains rich video plot information, such as text content, corresponding character labels, background description, and plot description, which can provide important context clues and knowledge support for question and answer.

[0058] Step S105: input the first target text content and the user content into a preset RAG model to generate reply content for the user content.

[0059] In step S105, the server inputs the first target text content and the user content into the preset RAG model to generate reply content for the user content. The preset RAG model is a method based on retrieval-enhanced generation, which combines the two tasks of information retrieval and text generation, and can generate answers that are relevant to the user's question and in line with the context based on the retrieved relevant text information. The core components of the preset RAG model include a retriever and a generator. The retriever is responsible for retrieving several documents that are most relevant to the user's question from a large-scale text library, and the generator generates the final answer text based on these documents and the user's question.

[0060] In this example, the server has found the first target text content that is most relevant to the user content through similarity calculation. Therefore, the retriever's task has been completed, and the server directly inputs the first target text content and the user content into the generator of the RAG model.

[0061] The generator of the preset RAG model adopts a Transformer-based Sequence-to-Sequence (Seq2Seq) model, such as BART, T5, etc. The generator receives two inputs: one is the first target text content, which serves as the context information for answer generation; the other is the user content, which serves as the question to be answered. The server needs to convert the first target text content and the user content into an input format acceptable to the preset RAG model. The preset RAG model adopts a sequence format similar to "question: ...\ncontext: ...\nanswer:", where the question part is the user content, the context part is the first target text content, and the answer part is left blank and generated by the preset RAG model.

[0062] For example, if the user content is "Do the hero and the heroine end up together?", and the first target text content is "At the end of the movie, the hero and the heroine finally resolve their misunderstanding, reconcile, and live happily together.", then the input sequence can be constructed as: "Question: Did the hero and heroine end up together?\nContext: At the end of the movie, the hero and heroine finally resolved their misunderstandings, reconciled, and lived happily together.\nAnswer:" The generator part of the preset RAG model is usually based on the Transformer architecture, such as BART. The preset RAG model uses an encoder based on the self-attention mechanism to encode the input sequence and convert it into a set of vector representations. The server passes the constructed input sequence to the encoder of the preset RAG model. The encoder first tokenizes the sequence, decomposes it into individual words or subwords, and maps them to corresponding embedding vectors. Then, the encoder transforms and combines the embedding vectors through multiple layers of self-attention mechanisms and feedforward neural networks, extracts the semantic information therein, and generates a set of context vectors. The context vector not only captures the semantics of the user's question, but also incorporates the background information provided by the first target text content. The context vector generated by the encoder is passed to the decoder of the preset RAG model. The decoder is also based on the Transformer architecture. Through the self-attention mechanism and the cross-attention mechanism, it gradually generates a complete answer sequence based on the context vector and the generated answer fragments. The decoding process usually adopts an autoregressive approach, that is, when generating each word, the previously generated word is used as input to predict the next most likely word. This cyclic prediction process continues until a special end tag (such as "[EOS]") is generated. After post-processing, the server obtains a format-standard, easy-to-read reply text for the user's content. Finally, the server returns the answer text to the user, completing the task of step S105.

[0063] For example, if the answer generated by the preset RAG model for the above input sequence is: "Yes", the server will post-process it into: "Yes, [EOS]", and return it to the user. The final answer seen by the user is: "Yes. The hero and the heroine finally resolved the misunderstanding and lived happily together." In a possible implementation, after inputting the first target text content and the user content into a preset RAG model and generating reply content for the user content, the method further includes: storing the user content and the reply content in a historical chat record.

[0064] Specifically, the server stores the user content and the generated reply content in a historical chat record database for subsequent query and context processing.

[0065] Reference Figure 2The present application also provides a RAG-based video plot question-and-answer device, which is a server. The server includes an acquisition module 201 and a processing module 202. The acquisition module 201 is used to acquire user content input by the user for the video content; the processing module 202 is used to convert the user content into a corresponding user vector; the processing module 202 is also used to calculate a first similarity between the user vector and the first target vector and a second similarity between the user vector and the second target vector, the first target vector is any target vector in a preset vector database, and the second target vector is any target vector in the preset vector database except the first target vector, and the preset vector database is pre-constructed according to the video content; the processing module 202 is also used to determine the first target text content corresponding to the first target vector according to the preset vector database if it is determined that the first similarity is greater than the second similarity; the processing module 202 is also used to input the first target text content and the user content into a preset RAG model to generate reply content for the user content.

[0066] In a possible implementation, before the acquisition module 201 acquires the user content input by the user, the method also includes: the acquisition module 201 acquires the video file; the processing module 202 uses a transcoding tool to convert the video file into an audio file; the processing module 202 separates the roles of the audio file to obtain the roles and the audio content corresponding to the roles; the processing module 202 converts the audio content into text content, and sets corresponding role tags for the text content; the processing module 202 uses a preset multimodal large model to extract the video file, extracts the key frames, and converts the key frames into background descriptions and plot descriptions for the video file; the processing module 202 integrates the text content, the role tags corresponding to the text content, the background description, and the plot description in chronological order to obtain the target text content; the processing module 202 converts the target text content into a target vector, and stores the target vector, the target text content, and the correspondence between the target vector and the target text content in a preset vector database.

[0067] In a possible implementation, the processing module 202 converts the user content into a corresponding user vector, specifically including: the acquisition module 201 obtains the historical chat records corresponding to the user content; the processing module 202 extracts the historical chat records to obtain historical key information; the processing module 202 splices the user content and the historical key information to obtain the spliced ​​target user content; the processing module 202 converts the target user content into a user vector.

[0068] In a possible implementation, the processing module 202 extracts the historical chat records to obtain historical key information, which specifically includes: the processing module 202 performs entity recognition on the historical chat records to obtain the corresponding historical entity, and performs entity recognition on the user content to obtain the corresponding content entity; if the processing module 202 determines that the historical entity and the content entity are the same entity, then the historical chat record corresponding to the historical entity is determined as the historical key information.

[0069] In a possible implementation, the processing module 202 uses a preset multimodal large model to extract key frames from the video file, specifically including: the processing module 202 determines multiple image frames in the video file; the processing module 202 calculates the image similarity between the first image frame and the second image frame, and the first image frame and the second image frame are any two adjacent image frames in the multiple image frames; if the processing module 202 determines that the image similarity is less than a preset similarity threshold, then the first image frame and the second image frame are determined to be key frames.

[0070] In one possible implementation, the processing module 202 converts the key frames into background descriptions and plot descriptions for the video file, specifically including: the processing module 202 performs feature extraction on the key frames to obtain corresponding visual feature vectors; the processing module 202 inputs the visual feature vectors into a preset image description generation model to obtain text descriptions for the key frames; the processing module 202 performs keyword extraction on the text description to determine corresponding keywords; the processing module 202 inputs the keywords into a preset keyword library to obtain description types corresponding to the keywords, the description types include background descriptions and plot descriptions, and the preset keyword library includes correspondences between keywords and description types.

[0071] In a possible implementation, after the processing module 202 inputs the first target text content and the user content into a preset RAG model and generates reply content for the user content, the method further includes: the processing module 202 stores the user content and the reply content in the historical chat record.

[0072] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0073] The present application also provides an electronic device. Figure 3 , Figure 3The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0074] The communication bus 302 is used to realize the connection and communication between these components.

[0075] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0076] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0077] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0078] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a video processing method based on RAG.

[0079] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call an application program storing a RAG-based video processing method in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0080] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0083] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0086] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0087] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A video plot question-answering method based on RAG, characterized in that: The method comprises: Obtain user content input by the user for the video content; Converting the user content into a corresponding user vector; Calculating a first similarity between the user vector and a first target vector and a second similarity between the user vector and a second target vector, wherein the first target vector is any target vector in a preset vector database, and the second target vector is any target vector in the preset vector database except the first target vector, wherein the preset vector database includes a plurality of the target vectors, and the preset vector database is pre-constructed according to the video content; If it is determined that the first similarity is greater than the second similarity, determining the first target text content corresponding to the first target vector according to the preset vector database; The first target text content and the user content are input into a preset RAG model to generate reply content for the user content.

2. The method according to claim 1, characterized in that: Before obtaining the user content input by the user for the video content, the method further includes: Obtaining a video file corresponding to the video content; Using a transcoding tool to convert the video file into an audio file; Separating the audio file into roles to obtain roles and audio content corresponding to the roles; Convert the audio content into text content and set corresponding role tags for the text content; The video file is extracted using a preset multimodal macro model to extract key frames, and the key frames are converted into background descriptions and plot descriptions for the video file; Integrate the text content, the character label corresponding to the text content, the background description, and the plot description in chronological order to obtain target text content; The target text content is converted into a target vector, and the target vector, the target text content, and the corresponding relationship between the target vector and the target text content are stored in a preset vector database.

3. The method according to claim 1, characterized in that The converting the user content into a corresponding user vector specifically includes: Obtaining historical chat records corresponding to the user content; Extracting the historical chat records to obtain historical key information; splicing the user content and the historical key information to obtain spliced ​​target user content; The target user content is converted into the user vector.

4. The method according to claim 3, characterized in that The extracting of the historical chat records to obtain historical key information specifically includes: Performing entity recognition on the historical chat records to obtain corresponding historical entities, and performing entity recognition on the user content to obtain corresponding content entities; If it is determined that the historical entity and the content entity are the same entity, the historical chat record corresponding to the historical entity is determined as the historical key information.

5. The method according to claim 2, characterized in that: The method of extracting the video file by using a preset multimodal large model to extract key frames specifically includes: Determining a plurality of image frames in the video file; Calculating an image similarity between a first image frame and a second image frame, where the first image frame and the second image frame are any two adjacent image frames among the plurality of image frames; If it is determined that the image similarity is less than a preset similarity threshold, the first image frame and the second image frame are determined to be key frames.

6. The method according to claim 2, characterized in that The converting the key frame into a background description and a plot description for the video file specifically includes: Extracting features from the key frame to obtain a corresponding visual feature vector; Inputting the visual feature vector into a preset image description generation model to obtain a text description for the key frame; Extract keywords from the text description to determine corresponding keywords; The keyword is input into a preset keyword library to obtain a description type corresponding to the keyword, wherein the description type includes a background description and a plot description, and the preset keyword library includes a correspondence between the keyword and the description type.

7. The method according to claim 1, characterized in that After inputting the first target text content and the user content into a preset RAG model to generate reply content for the user content, the method further includes: The user content and the reply content are stored in the historical chat record.

8. A video processing device based on RAG, characterized in that: The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to acquire user content input by the user for the video content; The processing module (202) is used to convert the user content into a corresponding user vector; The processing module (202) is further used to calculate a first similarity between the user vector and a first target vector and a second similarity between the user vector and a second target vector, wherein the first target vector is any target vector in a preset vector database, and the second target vector is any target vector in the preset vector database other than the first target vector, wherein the preset vector database includes a plurality of the target vectors, and the preset vector database is pre-constructed according to the video content; The processing module (202) is further configured to determine, if it is determined that the first similarity is greater than the second similarity, a first target text content corresponding to the first target vector according to the preset vector database; The processing module (202) is further used to input the first target text content and the user content into a preset RAG model to generate reply content for the user content.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images