RAG data acquisition and question and answer generation method and system based on encyclopedia website

Abstract

The invention discloses a RAG data acquisition and question and answer generation method and system based on an encyclopedia website, and relates to the technical field of data processing.The method comprises the steps that entry content in the encyclopedia website is captured, encrypted and stored in a database, and an index is established; extracting semantics of the vocabulary entry content to generate a knowledge graph; obtaining interaction data of the user, and carrying out interaction willingness analysis to identify interaction willingness data of the user for each entry; dynamically updating the database; and generating content reply according to the knowledge graph. The method comprises the following steps: capturing entry contents of an encyclopedia website in real time, combining data cleaning, encryption and index construction technologies to ensure the security and efficient and timely retrieval of data, constructing a knowledge graph based on semantic information of entries, realizing information structuring and semantic association, and generating personalized reply of a user based on a knowledge graph retrieval and generation technology. According to the overall scheme, the data real-time performance, the content generation accuracy and the user experience are remarkably improved, and meanwhile data privacy and system safety are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data crawling and querying, and in particular to a method and system for acquiring RAG data and generating questions and answers based on an encyclopedia website. Background Art

[0002] At present, in the vast field of natural language processing (NLP), retrieval-augmented generation (RAG) technology is gradually becoming an important tool for handling complex question-answering, text summarization and content creation tasks with its unique retrieval and generation mechanism. RAG technology generates text that meets user needs by retrieving information from relevant knowledge bases and combining the ability of generative models. However, traditional RAG technology still has limitations in the construction and updating of knowledge bases. It mainly relies on static or field-specific data sets, and it is difficult to fully cover and update a wide range of knowledge content in real time. In the process of knowledge base construction, the content of entries on encyclopedia websites is constantly updated, and the RAG system requires a stable data source to ensure the quality of generated text. Therefore, how to achieve real-time synchronization with the content of encyclopedia websites while maintaining data stability is an urgent problem to be solved. At the same time, the content of entries on encyclopedia websites contains rich semantic information, but the RAG system needs to be able to accurately understand and match this semantic information in order to generate text that meets user needs. In addition, user interactive content on encyclopedia websites, such as comments and editing history, is also a valuable knowledge resource. However, how to effectively use these interactive contents to further enrich the knowledge base of the RAG system and improve its generation capabilities is a direction worth exploring. Furthermore, when integrating encyclopedia websites with RAG technology, data privacy and security issues must be considered. How to ensure that the knowledge content contributed by users is properly protected during transmission and use is an important part of technical implementation that cannot be ignored. Summary of the invention

[0003] In view of this, the present invention proposes a method and system for obtaining RAG data and generating questions and answers based on encyclopedia websites, which can realize real-time content capture, user interaction data analysis and knowledge graph generation technology of encyclopedia websites, accurately analyze user query semantics, dynamically generate personalized and diversified high-relevance content replies, and greatly improve the real-time performance and accuracy of the question and answer system. The present invention provides the following technical solutions: A method for acquiring RAG data and generating questions and answers based on an encyclopedia website, the method comprising: crawling the entry content in the encyclopedia website in real time, pre-processing the entry content, encrypting and storing it in a database, and establishing an index; extracting semantic information in the entry content, and constructing a semantic network to generate a knowledge graph; acquiring user interaction data on the encyclopedia website for the entry content, and performing interaction willingness analysis on the interaction data to identify the user's interaction willingness data for each entry; matching the user data and its interaction willingness data with the entry content, and dynamically updating the database; receiving user query requests, and generating content replies based on the knowledge graph.

[0004] Optionally, the method of real-time crawling of entry contents in an encyclopedia website, pre-processing the entry contents and encrypting and storing them in a database, and establishing an index includes: applying web crawler technology to crawl entry contents from an encyclopedia website in real time; performing data cleaning, deduplication and version control pre-processing on the entry contents to ensure the accuracy and consistency of the entry content data; calling an encryption algorithm to encrypt the pre-processed entry contents and storing them in a database; extracting keywords from the entry contents, and generating a mapping relationship between the keywords and the entry contents, and storing the mapping relationship in an index structure of the database to establish an index.

[0005] Optionally, the method of extracting semantic information from entry content and constructing a semantic network to generate a knowledge graph includes: calling a deep learning model to encode the entry content and extract a sentence vector representation; extracting keywords, entities and relationships in the sentence vector, and marking named entities through text classification and entity recognition; calling syntactic analysis and relationship extraction algorithms to identify semantic relationships between entities to construct a semantic network; using the marked entities as nodes, the semantic relationships between entities as edges, and associating edges with nodes according to semantic relationships to generate a knowledge graph.

[0006] Optionally, the method of obtaining the user's interaction data on the content of an entry on the encyclopedia website and performing interaction willingness analysis on the interaction data to identify the user's interaction willingness data for each entry includes: identifying the user's interaction data on different entries on the encyclopedia website, including likes, comments, favorites and edits; obtaining the user's access data through an event listener, including the entry click history and the length of time the entry stays; calling the K-means algorithm to perform cluster analysis on the user's interaction data and access data to distinguish the user's interest scores for different entries; identifying the user's behavior data in the interaction data and access data to determine the user's sentiment classification for the current entry, the sentiment classification including positive, negative and neutral; and calculating the user's interaction willingness scores for different entries based on the user's interest scores and sentiment classification for different entries.

[0007] Optionally, the method for calculating the user's interaction willingness scores for different terms based on the user's interest scores and sentiment classifications for different terms includes: Defining Users For entry The interest score is ; Defining Users For entry The sentiment classification score is , when the sentiment is classified as positive, The value of is +1, when the sentiment is classified as negative, The value of is -1, when the sentiment is classified as neutral, The value of is 0; Defining Users For entry The access frequency is , indicating the number of clicks on the entry; Defining Users In the entry Length of stay on the website; Defining time decay weights , ,in, is the time decay factor, is the difference in days between the current time and the time when the interaction occurred. The value of is [0,1]; Defining Emotion Enhancers , the calculation formula is: ,in, is the emotional influencing factor; Constructing an interaction willingness score , the calculation formula is: ,in, and are the weights of interest scoring and emotion enhancement respectively, and , For users In the entry The regularization term for the residence time on .

[0008] Optionally, the method of matching user data and its interaction willingness data with entry content to dynamically update the database includes: obtaining log data from the encyclopedia website, identifying user identification data and entry content that interacts with the identification data based on the log data; matching the user identification data and the corresponding interaction willingness data for different entry contents with the entry content, and dynamically updating the entry content in the database using a version control mechanism.

[0009] Optionally, the method for receiving a user query request and generating a content response based on the knowledge graph includes: identifying user identification data, calling a deep learning model to segment the user query request, and identifying keywords and intents; converting keywords into semantic vectors, and using a similarity calculation method to query the nodes in the knowledge graph that match them; based on the edge extension relationship of the knowledge graph, obtaining the adjacent nodes that are most relevant to the query semantic vector; screening the content that matches the user identification data among the matched nodes, edges, and adjacent nodes, and indexing the corresponding entry content, sorting the entry content from high to low according to the interaction willingness score of the entry content, and outputting the structured information of the entry content with the highest ranking. If there is no content that matches the user identification data, obtaining the structured information integrated from the matched nodes, edges, and adjacent nodes; calling the generation model to generate a content response based on the structured information.

[0010] The present invention further discloses a RAG data acquisition and question-answer generation system based on an encyclopedia website, comprising: The crawling module is used to crawl the entry content in the encyclopedia website in real time, encrypt and store the entry content in the database after pre-processing, and create an index; Semantic recognition module, used to extract semantic information from entry content and build a semantic network to generate a knowledge graph; The interaction analysis module is used to obtain the user's interaction data on the encyclopedia website for the content of the entries, and to perform interaction intention analysis on the interaction data to identify the user's interaction intention data for each entry; Dynamic update module, used to match user data and their interaction intention data with the entry content and dynamically update the database; A generation module is used to receive user query requests and generate content responses based on the knowledge graph.

[0011] The present invention further discloses a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned RAG data acquisition and question and answer generation method based on the encyclopedia website is implemented.

[0012] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned RAG data acquisition and question and answer generation method based on the encyclopedia website is implemented.

[0013] According to the technical solution of the present invention, by real-time crawling of encyclopedia website entry contents, combined with data cleaning, encryption and index construction technologies, the security of data, timely update and expansion, and efficient retrieval are ensured, deep learning models are used to parse entry semantic information, and semantic networks are constructed to generate knowledge graphs to achieve information structuring and semantic association. Through clustering analysis of user interaction data and access behavior, sentiment classification and other technologies, users' willingness to interact with entries is dynamically evaluated, and user demand perception is enhanced. Based on knowledge graph retrieval and generation technology, combined with semantic analysis, templating and dynamic generation logic, personalized responses that meet user queries are generated. The overall solution significantly improves data real-time performance, content generation accuracy and user experience, while ensuring data privacy and system security. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which: Figure 1 It is a flowchart of a method for obtaining RAG data and generating questions and answers based on an encyclopedia website in an embodiment of the present invention; Figure 2 It is a schematic diagram of the composition of a system for acquiring RAG data and generating questions and answers based on an encyclopedia website in an embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the implementation mode of the present application. Obviously, the described implementation mode is only a part of the implementation mode of the present application, not all the implementation modes. Based on the implementation mode in the present application, all other implementation modes obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0016] It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0017] refer to Figure 1 This embodiment discloses a method for obtaining RAG data and generating questions and answers based on an encyclopedia website, the method comprising: S100: Crawl the entry content on the encyclopedia website in real time. After preprocessing the entry content, encrypt it and store it in the database, and establish an index. For crawling the entry content, use a web crawler based on the HTTP protocol such as Scrapy to crawl the entry content on the encyclopedia website. The entry content includes the entry title, text, and edit history, and an identity tag is defined for each entry content.

[0018] Perform data cleaning on the crawled entry content to remove HTML tags and irrelevant characters, deduplicate duplicate entries, and at the same time perform version control on the crawled entry content. Compare the new and old data versions to identify the changed content of the entry to ensure the accuracy and consistency of the data.

[0019] After the preprocessing of the entry content is completed, call an encryption algorithm to encrypt the entry content to ensure data security. In this implementation, the AES-256 symmetric encryption algorithm is used, and at the same time, a key management system is configured to dynamically generate a unique key for each encryption and securely store the key in the key management module. Specifically, obtain the preprocessed entry content, convert the entry content into a byte stream, and call the AES encryption function to perform encryption processing on it. The encryption mode uses GCM (Galois / Counter Mode), which supports data integrity verification, and outputs the encrypted data stream and the verification value. Use a distributed database to store the encrypted data. The data table of the encrypted content includes the entry content ID, the encrypted entry content, and the encryption key ID.

[0020] For the entry content, it is also necessary to establish a data index structure for it. Specifically, use natural language processing tools (such as jieba or spaCy) to extract keywords from the entry content: perform word segmentation on the entry content; based on the TF-IDF algorithm, calculate the weight of each word, and select the words with higher weights as keywords; filter out stop words (such as "is", "of", etc.) to ensure the effectiveness of the keywords. Handle the problem of long and short sentences: for long texts, perform sentence segmentation and independently extract keywords, and then perform global summarization to construct a keyword set corresponding to multiple entries, and generate a keyword list and its corresponding entry content ID for each entry. Further, store the mapping relationship between the keywords and the corresponding entry content ID as an inverted index structure, and call a full-text retrieval tool to store the index structure in the database. Since the entry content is collected in real time, when the entry content changes or new entries are added, re-extract keywords in real time and update the index database. And regularly check for invalid keywords and remove them. By encrypting and storing the preprocessed entry content as above, the security of the data during transmission and storage is guaranteed. The combination of keyword extraction and inverted index enables the system to efficiently retrieve relevant entries, providing accurate data support for subsequent semantic network construction and content generation.

[0021] S200: Extract semantic information from the content of the entry and build a semantic network to generate a knowledge graph. By identifying the semantic information of the content of the entry, the corresponding retrieval requirements can be supported. Specifically, the pre-trained language model (such as BERT, RoBERTa, GPT) is used to semantically encode the content of the entry. If the content of the entry in a specific field needs to be processed, the pre-trained language model fine-tuned for the field can be selected. Input the segmented and cleaned entry content into the pre-trained language model, divide the text into clauses of fixed length to adapt to the model input restrictions, pass the clauses into the model, generate a vector representation of each sentence, and output the sentence vector of each clause, which is represented as a high-dimensional feature vector of fixed dimension. Use TF-IDF and TextRank algorithms to extract keywords from the sentence vector, call the NER module of the deep learning model, identify the category of the named entity, use the classification model to classify the sentence vector, and add semantic category labels to the sentence, and finally output the extracted keywords, entities and their categories to form a structured data format.

[0022] Use dependency parsing tools to analyze the sentence dependencies. Output the syntactic tree structure, annotate the subject, predicate, object, and modifiers. Use rule templates to identify common relationship patterns and call deep learning relationship extraction models to process complex relationships to extract semantic relationships between entities. Exemplary relationship types include inclusion, composition, and dominance.

[0023] After obtaining the entities and entity relationships after semantic analysis, the entity is defined as a node with the attributes of its corresponding entry content. The semantic relationship between entities is used as the edge of the node, connecting at least two nodes. Similarly, the edge also has the attributes of the corresponding entry content. A knowledge graph is constructed based on nodes and edges, and graph query rules are defined. Based on a database that stores entry content that is updated in real time, the knowledge graph can also be dynamically updated. In this way, deep semantic analysis of encyclopedia entry content is achieved, keywords, entities, and relationships are automatically extracted, and the semantic associations between entities are accurately identified through dependency syntax parsing and relationship extraction. A dynamically updated knowledge graph is constructed to provide efficient support for subsequent semantic retrieval and question-answer generation.

[0024] S300: Obtaining the user's interaction data on the encyclopedia website for the entry content, and performing interaction intention analysis on the interaction data to identify the user's interaction intention data for each entry. The interaction data is used to identify the user's attitude towards different entry contents, such as attention, lack of interest, etc. By identifying the user's attitude towards different entry contents, targeted answer content can be output when responding to user query requests.

[0025] Specifically, define the user interaction data that can be collected, and collect the interaction data through the API interface or web crawler tool provided by the encyclopedia website. You can also parse the log data to extract the user ID, entry ID, interaction type and timestamp to identify the user interaction data for different entries in the encyclopedia website, including likes, comments, favorites and edits. Furthermore, deploy event listeners on the front end or server of the encyclopedia website. The listening event types include: page loading events (recording entry click history) and page closing events (calculating the length of stay).

[0026] The user's interaction data and access data can be used to evaluate the user's interest in different entries. Specifically, the features of the user's different entries are collected, including the counts of likes, comments, and collections, the length of stay, and the number of entry clicks. The above feature data is normalized, and the K-means algorithm is called to perform cluster analysis on the user features. The input is the feature matrix after data normalization, and the output is the interest category of each user in the entry, which specifically includes high, medium, and low. The interest category is mapped to an interest score. For example, high = 1.0, medium = 0.5, and low = 0.1. For the interest score, it can be freely set according to specific needs. Further, a pre-trained sentiment analysis model (such as BERT, ALBERT) is used to perform sentiment classification on the comment content. Classification output: positive (+1), neutral (0), negative (-1). For example, based on the distribution of the number of likes and dislikes, the sentiment classification results are adjusted: when the like ratio is high, the positive tendency is enhanced, and when the dislike ratio is high, the negative tendency is enhanced; introduce visit behavior (such as stay time) as an auxiliary factor for sentiment classification: shorter stay time: tends to be neutral or negative, longer stay time: tends to be positive.

[0027] After obtaining the user's interest scores and sentiment classifications for different terms, calculate the user's interaction willingness scores for different terms. Specifically: Defining Users For entry The interest score is ; Defining Users For entry The sentiment classification score is , when the sentiment is classified as positive, The value of is +1, when the sentiment is classified as negative, The value of is -1, when the sentiment is classified as neutral, The value of is 0; Defining Users For entry The access frequency is , indicating the number of clicks on the entry; Defining Users In the entry Length of stay on the website; Defining time decay weights , ,in, is the time decay factor, is the difference in days between the current time and the time when the interaction occurred. The value of is [0,1]; Defining Emotion Enhancers , the calculation formula is: ,in, is the emotional influencing factor; Constructing an interaction willingness score , the calculation formula is: ,in, and are the weights of interest scoring and emotion enhancement respectively, and , For users In the entry The regularization term for the residence time on .

[0028] Exemplary: , , stay time 120 seconds, visit frequency 3 times, the final interaction willingness score is: .

[0029] S400: Matching user data and its interaction willingness data with the entry content, and dynamically updating the database. By obtaining the log data of the encyclopedia website, the user's identification information, such as user ID, IP address, etc., is identified to match the entry content in which the user participates in the interaction, and the interaction willingness data in step S300 is merged and recorded in the database. The snapshot information of the current version is saved each time the database is updated to record the update time and the operation source to avoid data conflicts. For the dynamic update of the database, it is realized by identifying whether the user's interaction willingness score between different versions of the entry content has changed, and a trigger threshold is introduced. For example, if the difference in the interaction willingness score between the previous and next two versions exceeds the trigger threshold, the entry content stored in the current database needs to be updated, and the new version of the entry content replaces the old version, and its interaction willingness score is updated at the same time. In this way, by extracting user identification and interaction behavior, the user and the entry content are accurately associated, providing a reliable data source for subsequent interaction willingness analysis, and combining interaction willingness score and version control to ensure that the entry content in the database is updated in real time, maintaining data consistency and dynamic reflection of user behavior.

[0030] S500: Receive a user query request and generate a content response based on the knowledge graph.

[0031] When receiving a user's query request, identify the user ID of the current user, call the deep learning model to segment the user's query request, extract keywords and identify intent. Call the embedding model to convert keywords into fixed-dimensional semantic vectors, and calculate the similarity based on the semantic vector and the knowledge graph stored in the database, and select the nodes corresponding to the similarity that meets the preset threshold as matching nodes. Furthermore, for the matched nodes, obtain the direct adjacent nodes through the edge weights, and at the same time, calculate the relevance score of the adjacent nodes in combination with the edge and node similarity, and select the most relevant adjacent nodes according to the relevance score.

[0032] Based on the identified user ID, the matching nodes and edges are filtered through the index structure of the database to summarize the entry contents with which the user has had interaction records, and the user's interaction willingness score for each entry content is extracted. The entry contents are sorted from high to low according to the interaction willingness score, and the nodes and edges of the top-ranked entry contents are output, and the structured information corresponding to the current entry content is constructed based on the nodes and edges.

[0033] However, if the user's query request is not the content in his historical interaction data, the interaction intention score is ignored, and only the nodes, edges and adjacent nodes with the highest semantic similarity are output, and the structured information of the terms with the highest semantic similarity is generated.

[0034] After obtaining the structured information, the generation template is called to assist the generation model in generating natural language responses.

[0035] In summary, the technical solution of the present invention provides a RAG data acquisition and question-answer generation method based on an encyclopedia website, and realizes efficient dynamic content generation through the following technical solutions: (1) Real-time capture of the content of the encyclopedia website entries, cleaning, encryption and indexing of the data to ensure data security, consistency and retrieval efficiency; (2) Extracting the semantic information of the entries using a deep learning model, constructing a semantic network to generate a knowledge graph, and forming a structured highly correlated knowledge system; (3) Obtaining the user's interaction data on the encyclopedia website, combining the access behavior, calculating the interaction willingness score through K-means clustering and sentiment classification, and quantifying the user's interest and preference for the entry; (4) Matching the user's interaction willingness data with the entry content, dynamically updating the database through a version control mechanism, and ensuring real-time synchronization of data and user behavior; (5) Receive user query requests, combine the knowledge graph and the interaction willingness score, filter and sort the most relevant content, and call the generation model to output personalized question-answer content. The above scheme effectively improves the system's performance in data real-time, generated content accuracy and user experience, not only meets the personalized needs of users, but also significantly improves the efficiency of complex question-answer generation and information retrieval.

[0036] refer to Figure 2 This embodiment further discloses a RAG data acquisition and question-answer generation system based on an encyclopedia website, including: The crawling module 21 is used to crawl the entry content in the encyclopedia website in real time, encrypt and store the entry content in the database after pre-processing, and establish an index; it is also used to apply web crawler technology to crawl the entry content from the encyclopedia website in real time; perform data cleaning, deduplication and version control pre-processing on the entry content to ensure the accuracy and consistency of the entry content data; call the encryption algorithm to encrypt the pre-processed entry content and store it in the database; extract the keywords of the entry content, and generate a mapping relationship between the keywords and the entry content, and store the mapping relationship in the index structure of the database to establish an index.

[0037] The semantic recognition module 22 is used to extract semantic information from the entry content and build a semantic network to generate a knowledge graph; it is also used to call a deep learning model to encode the entry content and extract a sentence vector representation; extract keywords, entities and relationships in the sentence vector, and mark the named entities through text classification and entity recognition; call syntactic analysis and relationship extraction algorithms to identify the semantic relationships between entities to build a semantic network; use the marked entities as nodes, the semantic relationships between entities as edges, and associate the edges with nodes according to the semantic relationships to generate a knowledge graph.

[0038] The interaction analysis module 23 is used to obtain the user's interaction data on the encyclopedia website for the content of the entry, and to perform interaction willingness analysis on the interaction data to identify the user's interaction willingness data for each entry; it is also used to identify the user's interaction data on different entries in the encyclopedia website, including likes, comments, favorites and edits; obtain the user's access data through the event listener, including the entry click history and the entry stay time; call the K-means algorithm to perform cluster analysis on the user's interaction data and access data to distinguish the user's interest scores for different entries; identify the user's behavior data in the interaction data and access data to determine the user's sentiment classification for the current entry, and the sentiment classification includes positive, negative and neutral; calculate the user's interaction willingness score for different entries based on the user's interest score and sentiment classification for different entries, including: defining the user For entry The interest score is ; Defining Users For entry The sentiment classification score is , when the sentiment is classified as positive, The value of is +1, when the sentiment is classified as negative, The value of is -1, when the sentiment is classified as neutral, The value of is 0; Defining Users For entry The access frequency is , indicating the number of clicks on the entry; Defining Users In the entry Length of stay on the website; Defining time decay weights , ,in, is the time decay factor, is the difference in days between the current time and the time when the interaction occurred. The value of is [0,1]; Defining Emotion Enhancers , the calculation formula is: ,in, is the emotional influencing factor; Constructing an interaction willingness score , the calculation formula is: ,in, and are the weights of interest scoring and emotion enhancement respectively, and , For users In the entry The regularization term for the residence time on .

[0039] The dynamic update module 24 is used to match the user data and its interaction willingness data with the entry content and dynamically update the database; it is also used to obtain the log data of the encyclopedia website, identify the user identification data and the entry content that interacts with the identification data based on the log data; match the user identification data and the corresponding interaction willingness data for different entry contents with the entry content, and dynamically update the entry content in the database using the version control mechanism.

[0040] The generation module 25 is used to receive user query requests and generate content replies based on the knowledge graph; it is also used to identify user identification data, call a deep learning model to segment user query requests, and identify keywords and intentions; convert keywords into semantic vectors, and use a similarity calculation method to query the nodes matching them in the knowledge graph; based on the edge extension relationship of the knowledge graph, obtain the adjacent nodes most relevant to the query semantic vector; filter the content that matches the user identification data in the matched nodes, edges and adjacent nodes, and index the corresponding entry content, sort the entry content from high to low according to the interaction willingness score of the entry content, and output the structured information of the entry content with the highest ranking; if there is no content that matches the user identification data, output the structured information integrated from the matched nodes, edges and adjacent nodes; call the generation model to generate content replies based on the structured information.

[0041] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503 ; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above-mentioned method implementation methods.

[0042] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments.

[0043] A person skilled in the art can understand that all or part of the steps for implementing the above-mentioned method implementation method can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method implementation method; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0044] The device implementation described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme. Those of ordinary skill in the art may understand and implement it without creative effort.

[0045] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for obtaining RAG data and generating questions and answers based on a generative enhanced retrieval of an encyclopedia website, characterized in that: The method comprises: Capture the content of entries in the encyclopedia website in real time, encrypt and store the entry content in the database after preprocessing, and create an index; Extract semantic information from the entry content and construct a semantic network to generate a knowledge graph; Obtaining user interaction data on the encyclopedia website for the content of entries, and performing interaction intention analysis on the interaction data to identify the user's interaction intention data for each entry; Match user data and their interaction intention data with the entry content and dynamically update the database; Receive user query requests and generate content responses based on the knowledge graph.

2. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 1, characterized in that: The method of real-time crawling of entry contents in an encyclopedia website, pre-processing the entry contents, encrypting and storing them in a database, and establishing an index comprises: Apply web crawler technology to grab entry content from encyclopedia websites in real time; Perform data cleaning, deduplication and version control preprocessing on the entry content to ensure the accuracy and consistency of the entry content data; Call the encryption algorithm to encrypt the preprocessed entry content and store it in the database; Extract keywords from the entry content, generate a mapping relationship between the keywords and the entry content, and store the mapping relationship in the index structure of the database to create an index.

3. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 1, characterized in that: The method of extracting semantic information from the entry content and constructing a semantic network to generate a knowledge graph includes: Calling a deep learning model to encode the content of the entry and extract a sentence vector representation; Extract keywords, entities, and relations from sentence vectors, and annotate named entities through text classification and entity recognition; Calling syntactic analysis and relation extraction algorithms to identify semantic relations between entities to construct a semantic network; The labeled entities are taken as nodes, the semantic relationships between entities are taken as edges, and the edges and nodes are associated according to the semantic relationships to generate a knowledge graph.

4. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 1, characterized in that: The method of obtaining the user's interaction data on the encyclopedia website for the content of the entries and performing interaction intention analysis on the interaction data to identify the user's interaction intention data for each entry includes: Identify user interaction data for different entries on encyclopedia websites, including likes, comments, favorites, and edits; Obtain user access data through event listeners, including entry click history and entry dwell time; Call the K-means algorithm to perform cluster analysis on user interaction data and access data to distinguish users' interest scores for different terms; Identify the user's behavior data in the interaction data and the access data to determine the user's sentiment classification for the current term, wherein the sentiment classification includes positive, negative, and neutral; The user's interaction willingness score for different terms is calculated based on the user's interest score and sentiment classification for different terms.

5. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 4, characterized in that: The method for calculating the user's interaction willingness scores for different terms based on the user's interest scores and sentiment classification for different terms includes: Defining Users For entry The interest score is ; Defining Users For entry The sentiment classification score is , when the sentiment is classified as positive, The value of is +1, when the sentiment is classified as negative, The value of is -1, when the sentiment is classified as neutral, The value of is 0; Defining Users For entry The access frequency is , indicating the number of clicks on the entry; Defining Users In the entry Length of stay on the website; Defining time decay weights , ,in, is the time decay factor, is the difference in days between the current time and the time when the interaction occurred. The value of is [0,1]; Defining Emotion Enhancers , the calculation formula is: ,in, is the emotional influencing factor; Constructing an interaction willingness score , the calculation formula is: ,in, and are the weights of interest scoring and emotion enhancement respectively, and , For users In the entry The regularization term for the residence time on .

6. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 1, characterized in that: The method of matching the user data and the interaction willingness data with the entry content to dynamically update the database includes: Obtain log data from the encyclopedia website, and identify user identification data and the content of entries that interact with the identification data based on the log data; The user identification data and the corresponding interaction intention data for different entry contents are matched with the entry contents, and the entry contents in the database are dynamically updated using the version control mechanism.

7. The method for acquiring RAG data and generating questions and answers based on an encyclopedia website according to claim 4, characterized in that: The method of receiving a user query request and generating a content response according to the knowledge graph includes: Identify user identification data, and call the deep learning model to segment user query requests and identify keywords and intents; Convert keywords into semantic vectors and use similarity calculation methods to query matching nodes in the knowledge graph; Based on the edge expansion relationship of the knowledge graph, the adjacent nodes most relevant to the query semantic vector are obtained; Filter the content that matches the user identification data in the matched nodes, edges and adjacent nodes, index the corresponding entry content, sort the entry content from high to low according to the interaction willingness score, and output the structured information of the entry content with the highest ranking; If there is no content matching the user identification data, then the structured information of the matched nodes, edges and adjacent nodes is output; The generative model is called to generate content replies based on structured information.

8. A RAG data acquisition and question-answer generation system based on an encyclopedia website, characterized in that: include: The crawling module is used to crawl the entry content in the encyclopedia website in real time, encrypt and store the entry content in the database after pre-processing, and create an index; Semantic recognition module, used to extract semantic information from entry content and build a semantic network to generate a knowledge graph; The interaction analysis module is used to obtain the user's interaction data on the encyclopedia website for the content of the entries, and to perform interaction intention analysis on the interaction data to identify the user's interaction intention data for each entry; Dynamic update module, used to match user data and their interaction intention data with the entry content and dynamically update the database; A generation module is used to receive user query requests and generate content responses based on the knowledge graph.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images