Video plot question-answering system based on RAG

Abstract

The invention relates to the technical field of question answering systems, in particular to a video plot question answering system based on RAG, comprising a receiving module, an image construction module electrically connected to the side of the receiving module, and a preprocessing module electrically connected to the side of the image construction module. The side face of the preprocessing module is electrically connected with a deep analysis module, an answering module and a model fusion module, the question and answer system based on big data model and knowledge graph fusion comprises the preprocessing module, the preprocessing module can preprocess query statements of a user, and therefore the generalization ability of the model is enhanced; the mixed model fuses the word alignment model and the sentence vector model, the two models can be respectively utilized to predict the query statement of the user, and the prediction results are fused to improve the prediction accuracy. The preprocessing module is responsible for conducting preprocessing operation such as semantic analysis, word segmentation and stop word removal on a query statement input by a user so that a subsequent model can conduct processing more effectively.

Description

Technical Field

[0001] The present invention relates to the technical field of question-answering systems, and in particular to a RAG-based video plot question-answering system. Background Art

[0002] Video-based question-and-answer systems utilize advanced artificial intelligence technologies, particularly information retrieval and text generation, to understand and answer questions related to the video content. With the widespread adoption of internet technology, users are increasingly turning to service providers' platforms to consult customer service personnel regarding issues like service quality and maintenance. To improve customer service efficiency and reduce labor costs, service providers often employ question-and-answer systems, such as customer service bots, to interact with users and provide relevant responses. For example, a customer service bot first trains a question-and-answer model using a knowledge database containing standard question / answer pairs and historical question / answer examples. The trained question-and-answer model then generates answers to user-entered questions.

[0003] Existing question-answering systems still have certain shortcomings during use: Although current knowledge base searches primarily rely on static data, the real world is dynamic, involving complex information such as emotions and preferences. Therefore, a knowledge base that integrates both static and dynamic multi-dimensional information is crucial for question-answering systems. Based on rich and comprehensive data analysis, the system can seamlessly switch between various types of conversations, more closely resembling the way humans communicate. Fusion graph-based conversation systems are designed to meet these needs. During conversations, people incorporate a variety of factors, including their knowledge, habits, hobbies, and emotions. Existing conversation systems struggle to fully account for these factors. Summary of the Invention

[0004] The purpose of the present invention is to provide a video plot question-answering system based on RAG, so as to solve the problem raised in the above background technology that although the current knowledge base search mainly relies on static data, the development of things in the real world is dynamic, involving complex information such as emotions and preferences. Therefore, a knowledge base that integrates static and dynamic multi-dimensional information is crucial for the question-answering system. Based on rich and comprehensive data analysis, the system can achieve seamless switching of various types of dialogues, which is closer to the way humans communicate. The dialogue system based on fusion graphs is designed to meet these needs. During the dialogue process, people will incorporate a variety of factors, including their knowledge, habits, hobbies, emotions, etc. It is difficult for existing dialogue systems to fully handle these factors.

[0005] To achieve the above object, the present invention provides the following technical solutions: A video plot question-answering system based on RAG includes a receiving module, the side of the receiving module is electrically connected to an image construction module, the side of the image construction module is electrically connected to a preprocessing module, the sides of the preprocessing module are electrically connected to a deep analysis module, an answer module and a model fusion module, the receiving module is used to obtain user questions, and convert the data into structured data and clean invalid data through manually formulated rules and machine learning algorithms, the graph construction is used to obtain data, and construct a knowledge graph, a matter graph and a user portrait according to the data, and fuse them, the preprocessing module is used to preprocess the user questions, and is used to divide long question sentences into short sentences, and then perform semantic analysis on each short sentence respectively, the long sentences are parallel sentences containing commas and semicolons or containing and, the model fusion module uses the constructed knowledge graph and the collected data to train a big data model through machine learning and deep learning methods, and is used to fuse the word alignment model and the sentence vector model to obtain the probability that the user question belongs to each category.

[0006] As a preferred solution of the present invention, the answer module is used to obtain the intention of the question through question semantic analysis, find the corresponding data information from the fusion graph according to the intention of the question, and compose natural language as the answer. The graph construction module includes a knowledge graph construction module, which is used to use the static data in the cleaned structured data as knowledge data, and directly extract entities from the regularized data and proper nouns in the knowledge data; for discrete data in the knowledge data, statistical machine learning methods are used to complete the classification and clustering of entities, and pattern matching methods are used to obtain the relationship between entities and complete the extraction of entity and relationship attributes.

[0007] As a preferred solution of the present invention, the image construction module is used to use the dynamic data in the cleaned structured data as event data, including events and logical relationships, formulate relationship connection words according to the logical relationships, generate rule templates based on the connection words, first extract sentences with specified logical relationships according to the templates, and then extract events in the sentences according to the subject-object relationship as nodes in the event graph.

[0008] As a preferred solution of the present invention, the user portrait construction module is used to construct a user portrait based on the user's personal information data including habits, attributes, interests and hobbies; the graph fusion module is used to associate the knowledge graph with the user portrait through the same entity; the event graph associates the entities contained in the event and the entities related to the event with the entities in the knowledge graph, and uses entity alignment or ontology alignment to match the same entity from multiple knowledge graphs.

[0009] As a preferred solution of the present invention, the preprocessing module includes a rule template layer, which is used to convert the question statement into a logical expression form and match it with the defined template. If the match is successful, the parsing is successful, otherwise it enters the keyword template layer.

[0010] As a preferred solution of the present invention, the keyword template layer is used to perform semantic intent matching based on the content word types extracted from the question sentence and pre-defined keywords. If the match is successful, the parsing is successful, otherwise it enters the classification model layer.

[0011] As a preferred solution of the present invention, the classification model layer is used to adopt a neural network deep learning method to train pre-annotated corpus text data to generate a semantic classification model, match the question sentence with the semantic classification model, obtain the content words in the sentence, and complete semantic analysis.

[0012] As a preferred solution of the present invention, the probability distribution function of the model fusion module is: p(i|q, X) = α*F1(q, X)+β*F2(q, X) where α + β = 1, F1 is the word alignment model, and F2 is the sentence vector model.

[0013] As a preferred solution of the present invention, the answer module includes the following steps: after extracting the content words in the question sentence, the question sentence is converted into a query sentence for graph query through word slot processing, and the corresponding answer is preset in the knowledge base through the query sentence.

[0014] As a preferred solution of the present invention, for situations where the problem cannot be converted into a query statement, a vector space modeling method is adopted, in which the vector space is used to describe the question statement and the entities and relationships of the graph, and the vector space is trained. Knowledge is searched by comparing the distance between the question statement and the alternative answers in the vector space, and an answer statement is constructed based on the knowledge found from the fused graph.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. In the present invention, the question-answering system based on the fusion of big data model and knowledge graph includes a preprocessing module, which can preprocess the user's query statement, thereby enhancing the generalization ability of the model; the hybrid model integrates the word alignment model and the sentence vector model, and can use these two models to predict the user's query statement respectively, and fuse the prediction results to improve the accuracy of the prediction. The preprocessing module is responsible for performing semantic analysis, word segmentation, stop word removal and other preprocessing operations on the query statement input by the user so that the subsequent model can process it more effectively. The word alignment model can match the keywords in the user's query statement with the entities in the knowledge graph, and locate the most similar entities and their related information. The sentence vector model converts the user's query statement into a vector form, and then calculates the similarity with the sentence vector in the knowledge graph to find the plot content that best matches the user's query statement. By fusing the prediction results of these two models.

[0016] 2. In this invention, by fusing graphs, information barriers between different graphs are eliminated, connections are established between them, and knowledge and form complementarity are achieved. This not only broadens the scope of Q&A coverage, but also expands this scope not only in practical application areas but also in application forms. It can satisfy queries about entities as well as questions and answers about events. The answer generation process fully considers the user's emotions, so the generated sentences are also rich in emotion. The fused graphs achieve interconnectivity, thereby greatly improving the accuracy of answers to questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall structure of the present invention; Figure 2 Schematic diagram of the question-answering step structure of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] For examples, see Figure 1-Figure 2 , the present invention provides a technical solution: A video plot question-answering system based on RAG includes a receiving module, the side of the receiving module is electrically connected to an image construction module, the side of the image construction module is electrically connected to a preprocessing module, the sides of the preprocessing module are electrically connected to a deep analysis module, an answering module and a model fusion module, the receiving module is used to obtain user questions, and convert the data into structured data and clean invalid data through manually formulated rules and machine learning algorithms, the graph construction is used to obtain data, and construct a knowledge graph, a fact graph and a user portrait according to the data, and fuse them, the preprocessing module is used to preprocess user questions, and is used to divide long sentences into short sentences for question sentences, and then perform semantic analysis on each short sentence respectively, the long sentence is a parallel sentence containing commas and semicolons or containing and, the model fusion module uses the structure The constructed knowledge graph and the collected data are used to train a big data model through machine learning and deep learning methods to fuse the word alignment model and the sentence vector model to obtain the probability that the user's question belongs to each category. The answer module is used to obtain the intention of the question through question semantic analysis, and find the corresponding data information from the fused graph according to the intention of the question to compose natural language as the answer. The graph construction module, including the knowledge graph construction module, is used to use the static data in the cleaned structured data as knowledge data, and directly extract entities from the regularized data and proper nouns in the knowledge data; for discrete data in the knowledge data, statistical machine learning methods are used to complete the classification and clustering of entities, and pattern matching methods are used to obtain the relationship between entities and complete the extraction of entity and relationship attributes.

[0020] The two models can be used to predict user queries and the prediction results can be integrated to improve the accuracy of the prediction. The preprocessing module is responsible for performing semantic analysis, word segmentation, and stop word removal on the query input by the user, so that the subsequent model can process it more effectively.

[0021] In this embodiment, Figure 1 and Figure 2As shown, the image construction module is used to take the dynamic data in the cleaned structured data as the event data, including events and logical relations, formulate relational conjunctions according to the logical relations, generate rule templates based on the conjunctions, first extract sentences with specified logical relations according to the templates, and then extract events as nodes in the event graph according to the subject-object relations in the sentences. The user portrait construction module is used to construct user portraits based on the user's personal information data including habits, attributes, interests and hobbies. The graph fusion module is used to associate the knowledge graph with the user portrait through the same entities. The event graph associates the entities contained in the event and the entities related to the event with the entities in the knowledge graph respectively, and uses entity alignment or ontology alignment to obtain the information from multiple knowledge graphs. The same entity is matched in the recognition graph. The preprocessing module includes a rule template layer, which is used to convert the question statement into the form of a logical expression and match it with the defined template. If the match is successful, the parsing is successful, otherwise it enters the keyword template layer. The keyword template layer is used to match the semantic intention according to the type of content words extracted from the question statement and the pre-defined keywords. If the match is successful, the parsing is successful, otherwise it enters the classification model layer. The classification model layer is used to use the neural network deep learning method to train the pre-annotated corpus text data to generate a semantic classification model, match the question statement with the semantic classification model, obtain the content words in the sentence, and complete the semantic parsing. The probability distribution function of the model fusion module is: p(i|q, X) = α*F1(q, X)+β*F2(q, X) where α +β = 1, F1 is the word alignment model, and F2 is the sentence vector model.

[0022] This not only broadens the scope of question-and-answering, but also extends beyond practical application areas to encompass all forms of application. It can handle queries about entities as well as questions about events. The answer generation process fully considers user emotions, resulting in emotionally rich sentences.

[0023] The workflow of the present invention is as follows: When the RAG-based video plot question-answering system designed by the present invention is working, after extracting the content words in the question sentence, the question sentence is converted into a query sentence for graph query through word slot processing. The corresponding answer is preset in the knowledge base through the query sentence. For the situation that cannot be converted into a query sentence, the vector space modeling method is adopted, and the vector space is used to describe the question sentence and the entities and relationships of the graph, and the vector space is trained. The knowledge search is realized by comparing the distance between the question sentence and the alternative answers in the vector space, and the answer sentence is constructed according to the knowledge found from the fused graph.

[0024] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A RAG-based video plot question-answering system, comprising a receiving module, characterized in that: The side of the receiving module is electrically connected to an image construction module, and the side of the image construction module is electrically connected to a preprocessing module. The side of the preprocessing module is electrically connected to a deep analysis module, an answer module and a model fusion module respectively. The receiving module is used to obtain user questions, and convert the data into structured data and clean invalid data through manually formulated rules and machine learning algorithms. The graph construction is used to obtain data, and construct a knowledge graph, a matter graph and a user portrait according to the data, and fuse them. The preprocessing module is used to preprocess the user questions, and is used to divide long sentences into short sentences for question statements, and then perform semantic analysis on each short sentence respectively. The long sentences are parallel sentences containing commas, semicolons or and. The model fusion module uses the constructed knowledge graph and the collected data to train a big data model through machine learning and deep learning methods, which is used to fuse the word alignment model and the sentence vector model to obtain the probability that the user question belongs to each category.

2. The RAG-based video plot question-answering system according to claim 1, characterized in that: The answer module is used to obtain the intention of the question through question semantic analysis, find the corresponding data information from the fusion graph according to the intention of the question, and compose natural language as the answer. The graph construction module includes a knowledge graph construction module, which is used to use the static data in the cleaned structured data as knowledge data, and directly extract entities from the regularized data and proper nouns in the knowledge data; for discrete data in the knowledge data, statistical machine learning methods are used to complete the classification and clustering of entities, and pattern matching methods are used to obtain the relationship between entities and complete the extraction of entity and relationship attributes.

3. The RAG-based video plot question-answering system according to claim 1, characterized in that: The image construction module is used to use the dynamic data in the cleaned structured data as event data, including events and logical relationships, formulate relationship connection words according to the logical relationships, generate rule templates based on the connection words, first extract sentences with specified logical relationships according to the templates, and then extract events in the sentences according to the subject-object relationship as nodes in the event graph.

4. The RAG-based video plot question-answering system according to claim 1, characterized in that: The user portrait construction module is used to construct a user portrait based on the user's personal information data including habits, attributes, interests and hobbies. The graph fusion module is used to associate the knowledge graph with the user portrait through the same entity. The event graph associates the entities contained in the event and the entities related to the event with the entities in the knowledge graph respectively, and uses entity alignment or ontology alignment to match the same entity from multiple knowledge graphs.

5. The RAG-based video plot question-answering system according to claim 1, characterized in that: The pre-processing module includes a rule template layer, which is used to convert the question statement into a logical expression form and match it with the defined template. If the match is successful, the parsing is successful, otherwise it enters the keyword template layer.

6. The RAG-based video plot question-answering system according to claim 5, characterized in that: The keyword template layer is used to perform semantic intent matching based on the content word types extracted from the question sentence and pre-defined keywords. If the match is successful, the parsing is successful, otherwise it enters the classification model layer.

7. The RAG-based video plot question-answering system according to claim 6, characterized in that: The classification model layer is used to adopt a neural network deep learning method to train pre-labeled corpus text data to generate a semantic classification model, match the question sentence with the semantic classification model, obtain the content words in the sentence, and complete semantic analysis.

8. The RAG-based video plot question-answering system according to claim 1, characterized in that: The probability distribution function of the model fusion module is: p(i|q, X) = α*F1(q, X) + β*F2(q, X) where α + β = 1, F1 is the word alignment model, and F2 is the sentence vector model.

9. The RAG-based video plot question-answering system according to claim 1, characterized in that: The answer module includes the following steps: after extracting the content words in the question sentence, the question sentence is converted into a query sentence for graph query through word slot processing, and the corresponding answer is preset in the knowledge base through the query sentence.

10. The RAG-based video plot question-answering system according to claim 9, characterized in that: For situations where the answer cannot be converted into a query statement, a vector space modeling method is adopted. The vector space is used to describe the question statement and the entities and relationships of the graph, and the vector space is trained. Knowledge is searched by comparing the distance between the question statement and the alternative answers in the vector space, and an answer statement is constructed based on the knowledge found from the fused graph.

Images