Material extraction and generation method based on large model and multi-storage technology

Abstract

The present invention provides a material extraction and generation method based on a large model and multiple storage technologies, and belongs to the technical field of material extraction and generation. The method uses an object database, a vector database, and a knowledge graph to store the collected original material data; when the user inputs text, relevant data are obtained from the object database, the vector database, and the knowledge graph, respectively, to achieve full-text accurate matching of the data, semantic similarity retrieval, and deep relationship mining; the text similarity and semantic similarity are calculated for the obtained data respectively; the obtained data are input into a large language model to generate corresponding materials, and the generated materials are scored using a scoring model; the multi-objective optimization value of each material is calculated by comprehensively considering the text similarity, semantic similarity, and scoring; the generated materials are sorted according to the multi-objective optimization value, and the highest ranking is taken as the final generated material. The materials generated by the present invention are safe and controllable, and have a unified style, which is more in line with user preferences.

Description

Technical Field

[0001] The present invention relates to the technical field of material extraction and generation, and in particular to a material extraction and generation method based on a large model and multiple storage technologies. Background Art

[0002] With the continuous advancement of artificial intelligence and big model technology, intelligent generation applications have penetrated into various fields. In terms of material generation, based on the deep understanding and analysis of massive text data based on big model technology, it can assist writers, editors and content creators to automatically generate diverse, high-quality paragraphs or even entire articles based on the keywords, topics or summaries entered by users, imitate different styles and writing styles, and help users quickly build article frameworks and fill in content details while maintaining logical coherence and language accuracy. As the basis for various generation applications, the quality of material extraction and generation by big models directly affects the effectiveness of the application.

[0003] However, the existing material generation technology has the following problems and difficulties:

[0004] 1. The relevant information comes from a wide range of sources, is large in quantity, has different formats and complex structures. In the process of material extraction, how to fully screen out useful, relevant and high-quality information from the huge amount of information has become a huge challenge.

[0005] 2. Due to the wide range of sources of material information, there are significant differences in writing styles and expressions between different authors, different platforms, and different fields, making it difficult to generate personalized material.

[0006] 3. The output of the model needs to be strict, accurate, efficient and consistent with social values. The lack of an effective output control mechanism may cause the model to produce inappropriate or erroneous responses, thereby posing potential risks. Summary of the invention

[0007] In view of this, an embodiment of the present invention provides a material extraction and generation method based on a large model and multiple storage technologies, so as to solve the problem that existing material generation and extraction technologies are difficult to generate personalized high-quality materials.

[0008] The technical solution adopted by the present invention is:

[0009] The present invention provides a material extraction and generation method based on a large model and multiple storage technologies, comprising:

[0010] Obtaining the original material data and preprocessing the data, and storing the preprocessed original material data in the object database;

[0011] A vector database is constructed based on the preprocessed original material data, and a material knowledge graph is constructed using the preprocessed original material data, and the material knowledge graph is stored in the graph database;

[0012] Filter the user's input text for sensitive words according to the preset sensitive word list to obtain the filtered input text;

[0013] Searching the object database, the vector database, and the graph database according to the filtered input text to obtain a first search result, a second search result, and a third search result, respectively;

[0014] Calculate the similarity between the first search result, the second search result and the third search result and the filtered input text to obtain the text similarity value Sim t and semantic similarity value Sim v ;

[0015] The filtered input text, the first search result, the second search result, and the third search result are concatenated and input into the first language model to generate personalized material, and the personalized material is input into the second model for scoring to obtain the material score value Sco ;

[0016] Based on text similarity value Sim t , semantic similarity value Sim v and the rating value of the material Sco Calculate multiple optimization target values ​​for each personalized material Obj ;

[0017] Multiple optimization target values ​​for each personalized material Obj Sort them and take the top k ranked materials as the final generated materials.

[0018] Furthermore, the obtaining of the original material data and preprocessing the data, and storing the preprocessed original material data in the object database, includes:

[0019] Collect data related to various materials, extract text information from the materials, and convert the text information into a unified text format to obtain the original data of the materials;

[0020] Perform data cleaning and standardization on the original material data to obtain the pre-processed original material data;

[0021] Keywords are extracted from the preprocessed original material data, and an index table is created for the extracted keywords. The preprocessed original material data and the index table are stored in the object database.

[0022] Furthermore, constructing a vector database based on the preprocessed original material data includes:

[0023] The pre-processed original material is divided into blocks according to semantics using a block model to obtain multiple text blocks;

[0024] Multiple text blocks are encoded through a fine-tuned BGE model to obtain corresponding embedding vectors, and a vector database is built based on the embedding vectors, and the embedded vectors are stored in the vector database.

[0025] Furthermore, the fine-tuning process of the BGE model includes:

[0026] Obtain historical question-answer pair data, use the data of correct answers to questions in the historical question-answer pair data as positive samples, and find similar answer data from the rest of the question-answer pair data as negative samples, and use the question-answer pair data, positive samples, and negative samples as training data;

[0027] The semantic relevance task is regarded as a binary classification task, and the BGE model is fine-tuned using the questions, positive samples and multiple negative samples in the training data. During the fine-tuning process, the cross entropy loss function is used as the optimization objective, and the parameters of the BGE model are iteratively updated to obtain the fine-tuned BGE model.

[0028] Furthermore, the method of constructing a material knowledge graph using the preprocessed material original data and storing the material knowledge graph in a graph database includes:

[0029] The pre-processed original material is divided into blocks according to semantics using a block model to obtain multiple text blocks;

[0030] Entity relationship extraction is performed on multiple text blocks to obtain entities and entity relationships in each text block;

[0031] The entities in multiple text blocks are merged, and the duplicate entities in the merging process are eliminated through the entity linking method. After the material knowledge graph is formed, the material knowledge graph is stored in the graph database.

[0032] Furthermore, the sensitive word filtering of the user's input text according to the preset sensitive word list to obtain the filtered input text includes:

[0033] According to the preset sensitive word list configured by the user, a DFA algorithm is used to construct a DFA for each sensitive word in the preset sensitive word list to obtain a sensitive word filter;

[0034] The user's input text is filtered for sensitive words through a sensitive word filter to obtain the input text with sensitive words filtered out.

[0035] Furthermore, the filtered input text is searched in the object database, the vector database and the graph database to obtain a first search result, a second search result and a third search result, respectively, including:

[0036] Based on the words in the filtered input text, a full-text search is performed in the index table of the object database, and the retrieved original material is used as the first search result;

[0037] The filtered input text is converted into a search vector using a fine-tuned BGE model, the search vector is compared with the embedded vector in the vector database for semantic similarity, and the embedded vector with a semantic similarity greater than a preset search threshold is taken as the second search result;

[0038] Perform community detection on the material knowledge graph in the graph database to obtain the community structure in the knowledge graph;

[0039] Entity extraction is performed on the filtered input text to obtain the question entity, and the question entity is entity linked with the entity in the material knowledge graph through the Pairwise algorithm, and the knowledge of the community structure to which the linked entity belongs is extracted as the third retrieval result.

[0040] Furthermore, the first search result, the second search result and the third search result are respectively similarly calculated with the filtered input text to obtain a text similarity value. Sim t and semantic similarity value Sim v ,include:

[0041] According to the frequency of occurrence of each word in the filtered input text and the importance of the word, the text similarity value between the filtered input text and the first search result, the second search result and the third search result is calculated by the BM25 algorithm Sim t ;

[0042] The filtered input text, the first search result, the second search result, and the third search result are encoded into vectors through the Word2Vec model, and the cosine similarity between the filtered input text and the first search result, the second search result, and the third search result is calculated to obtain the semantic similarity value. Sim v .

[0043] Furthermore, the step of splicing the filtered input text, the first search result, the second search result and the third search result and inputting them into the first language model to generate personalized material includes:

[0044] Concatenate the filtered input text with the first search result, the second search result, and the third search result into a complete prompt;

[0045] The complete prompt is input into the first large language model, and the large language model determines whether the user input is relevant to the first, second and third search results. If so, the large language model generates personalized material based on the first, second and third search results. If not, the large language model generates personalized material based on its own knowledge reserves.

[0046] Furthermore, the multiple optimization objective values Obj The calculation method is:

[0047] ;

[0048] in, are weight values, which respectively represent the importance of text similarity, semantic similarity and score value, and satisfy .

[0049] In summary, the beneficial effects of the present invention are as follows:

[0050] The present invention provides a material extraction and generation method based on a large model and multiple storage technologies. The method first collects material-related original data through multiple channels; pre-processes the collected original data and converts them into a unified text format; uses an object database, a vector database, and a knowledge graph to store the collected original data; when a user inputs text, obtains relevant data from the object database, the vector database, and the knowledge graph through full-text retrieval, vector retrieval, and community retrieval methods, respectively, to achieve full-text accurate matching of the data, semantic similarity retrieval, and deep mining of relationships, and can fully and accurately extract relevant data generated by the material. For the acquired data, the text similarity and semantic similarity are calculated respectively; the generated data is input into a large language model as a reference, and the large language model generates corresponding materials according to the data, and the generated materials are scored using a second large model (scoring model); based on multi-objective technology, the multi-objective optimization value of each personalized material is calculated by comprehensively considering the text similarity, semantic similarity, and scoring; the generated materials are sorted according to the multi-objective optimization value, and the highest ranking is taken as the final generated material. The present invention comprehensively considers the quality of material extraction and the quality of large model generation. On the one hand, it fully mines relevant information as a reference for large model generation through various data storage technologies, so that the generated material is safe and controllable; on the other hand, it uses the ability of the large model to ignore misleading retrieval content and generate materials with a unified style that are more in line with user preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0052] Figure 1 This is a schematic diagram of the principle of a material extraction and generation method based on a large model and multiple storage technologies according to the present invention;

[0053] Figure 2 A schematic diagram of the process of constructing a vector knowledge base in the present invention;

[0054] Figure 3 Schematic diagram of the process of constructing the knowledge graph in the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0056] With the continuous progress of artificial intelligence and big model technology, intelligent generation applications have penetrated into various fields. For example, in terms of intelligent question and answer, big model technology provides strong support for intelligence and efficiency, so that the question and answer system has now been applied to many fields such as government affairs, propaganda, medical treatment, and teaching, greatly improving the convenience of information acquisition and communication. In order to achieve efficient material extraction and generation, the present invention provides a multi-storage technology material extraction and generation method based on a big model, which is customized for personalized field question and answer needs, based on object database, vector database and knowledge graph, through a big model to achieve material extraction and generation, and combined with multi-objective optimization technology to optimize the generated material, while ensuring the safety and controllability of the material, misleading retrieval content is ignored, and a unified style of material is generated that is more in line with the user's personalized preferences, providing high-quality material for applications such as intelligent question and answer, auxiliary writing, and copywriting generation. The detailed implementation process of the present invention is shown in the following embodiments.

[0057] Embodiment 1:

[0058] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing the principle of a material extraction and generation method based on a large model and multiple storage technologies in Embodiment 1 of the present invention. Figure 1 As shown, the method provided by the present invention comprises:

[0059] Obtaining the original material data and preprocessing the data, and storing the preprocessed original material data in the object database;

[0060] A vector database is constructed based on the preprocessed original material data, and a material knowledge graph is constructed using the preprocessed original material data, and the material knowledge graph is stored in the graph database;

[0061] Filter the user's input text for sensitive words according to the preset sensitive word list to obtain the filtered input text;

[0062] Searching the object database, the vector database, and the graph database according to the filtered input text to obtain a first search result, a second search result, and a third search result, respectively;

[0063] Calculate the similarity between the first search result, the second search result and the third search result and the filtered input text to obtain the text similarity value Sim t and semantic similarity value Sim v ;

[0064] The filtered input text, the first search result, the second search result, and the third search result are concatenated and input into the first language model to generate personalized material, and the personalized material is input into the second model for scoring to obtain the material score value Sco ;

[0065] Based on text similarity value Sim t , semantic similarity value Sim v and the rating value of the material Sco Calculate multiple optimization target values ​​for each personalized material Obj ;

[0066] Multiple optimization target values ​​for each personalized material Obj Sort them and take the top k ranked materials as the final generated materials.

[0067] Furthermore, in an embodiment of the present invention, obtaining the original material data and preprocessing the data, and storing the preprocessed original material data in the object database specifically includes the following steps:

[0068] S11: Collect data such as books, reports, and reviews related to various materials through libraries, archives, and the Internet, and extract text information in combination with the user's internal personalized related data, extract text information from the materials, and convert the text information into a unified text format to obtain the original material. Among them, the data formats of the materials include txt, docx, pdf, jpg, etc. The text information in the materials is extracted and integrated into a unified text format, thereby improving the efficiency and accuracy of data processing.

[0069] Specifically, the format conversion method of text information is to open the docx format document through Microsoft word or WPS Office and save it as txt format to ensure the text content is correct. For pdf documents and jpg pictures, use OCR (Optical Character Recognition) technology to perform text recognition and save the text in the picture as txt format for subsequent processing.

[0070] S12: Perform data cleaning and standardization on the original material data to obtain the preprocessed original material data. The main operations of data cleaning and standardization include text verification and correction, cleaning unnecessary characters, deleting duplicate content, and standardizing and formatting time information. By writing automated scripts, existing text processing tools and libraries (such as NLTK, spaCy, etc.) are used to assist in completing preprocessing.

[0071] Specifically, the text verification and correction operations are as follows: check the converted txt files one by one, verify the accuracy of the text content, manually correct errors that may occur during the OCR recognition process (such as typos, garbled characters, etc.), and adjust paragraphs with confusing formats to ensure the coherence and readability of the text.

[0072] Specifically, the operation of cleaning unnecessary characters is as follows: Use the regular expression function in text editing tools or programming languages ​​(such as Python, Perl, etc.) to automatically delete extra spaces and unnecessary line breaks in the text. Ensure that the format between each paragraph is unified to facilitate subsequent analysis and processing. Use regular expressions to match the common formats of headers and footers, identify and delete headers and footers in the document and other information that is irrelevant to the text content.

[0073] Furthermore, the operation of deleting duplicate content is specifically: using a text comparison tool or writing a script to deduplicate the entire document set and delete completely duplicate files or text paragraphs.

[0074] Furthermore, the time standardization and formatting operations are specifically as follows: identifying all time information in the text, including year, month, date, etc. Converting irregular time expressions (such as "08", "same year", "next year", etc.) into standard date formats (such as "2008").

[0075] S13: extract keywords from the preprocessed original material data, create an index table for the extracted keywords, and store the preprocessed original material data and the index table in an object database. An object database is a database used to store and manage object data. Compared with a traditional relational database, an object database is more suitable for storing complex data structures and object-oriented data models, such as various common unstructured data (such as pictures, videos, log files, etc.).

[0076] The embodiment of the present invention stores the cleaned and standardized data in an object database, which helps to quickly retrieve and locate the original location of the data and facilitates accurate matching of the data.

[0077] In the embodiment of the present invention, a vector database is constructed based on cleaned and standardized data, and storing data in the vector database helps to quickly obtain semantically related data based on the similarity between vectors.

[0078] Among them, the vector database is a database specifically used to store and query vectors. The vectors it stores come from the vectorization of text, speech, images, videos, etc. Compared with traditional databases, vector databases can handle and search data more easily by semantic similarity. In the material generation scenario, the vector database can help quickly retrieve and integrate information related to the semantics of user input, providing strong data support for subsequent material generation. The vector database is optimized for the storage and retrieval of vector data, and can efficiently save and process large-scale vector data. The vector database not only saves the embedded vectors themselves, but also saves the original text data indexes corresponding to these vectors, so that relevant information can be quickly found in subsequent retrieval.

[0079] Specifically, in the embodiments of the present invention, referring to Figure 2 As shown, constructing a vector database based on the pre-processed original material data includes the following steps:

[0080] S21: Use the chunking model to chunk the preprocessed raw material according to semantics to obtain multiple text chunks. Among them, the cleaned and standardized data is divided into sentences or paragraphs to form multiple text chunks. The purpose of this step is to process the data more finely so that the subsequent knowledge embedding can capture the information in the text more accurately.

[0081] Furthermore, the block model is a model based on the BERT model and fine-tuned using the original data of the material. Through model fine-tuning, the model can better understand the semantics of the material text knowledge, learn the vocabulary, expressions and context information unique to the target field, and thus improve the accuracy of the model in text block segmentation. During the fine-tuning process, a variety of optimization strategies are adopted, such as adjusting the learning rate, increasing the number of training rounds, and using regularization technology to prevent overfitting, to ensure that the model can fully learn the characteristics of the data and maintain good generalization capabilities.

[0082] S22: Multiple text blocks are encoded through a fine-tuned BGE model to obtain corresponding embedding vectors (i.e., embedding), and a vector database is built based on the embedding vectors. The embedding vectors are stored in the vector database to facilitate efficient retrieval to find semantically related data.

[0083] Among them, the BGE model is a representation model based on Transformer, which can effectively transform text into embedding vectors in high-dimensional vector space, and then obtain the similarity between vectors through cosine similarity, so as to build an index for the vector database. In order to improve the representation ability of the BGE model, the BGE model is fine-tuned using historical material question and answer data. The fine-tuning process specifically includes:

[0084] Obtain historical question-answer pair data, use the data of correct answers to questions in the historical question-answer pair data as positive samples, and find similar answer data from the rest of the question-answer pair data as negative samples, and use the question-answer pair data, positive samples, and negative samples as training data;

[0085] The semantic relevance task is regarded as a binary classification task, and the BGE model is fine-tuned using the questions, positive samples and multiple negative samples in the training data. The fine-tuning dataset format is [query, positive sample set, negative sample set]. During the fine-tuning process, the cross entropy loss function (BCE) is used as the optimization target to iteratively update the parameters of the BGE model to better capture the semantic relevance, and finally obtain the fine-tuned BGE model.

[0086] Specifically, the cross entropy loss function is commonly used in the training of vector retrieval models, and its formula is as follows:

[0087] ;

[0088] in, Indicates the classification category, and its value is 2 in the binary classification task; Indicates whether a sample is actually a positive / negative sample, represented by 1 / 0; The probability calculated for the model that the sample is a positive / negative sample.

[0089] Furthermore, in an embodiment of the present invention, a knowledge graph is constructed based on cleaned and standardized data, and storing data in the knowledge graph helps to deeply explore the correlation between data, query the complex relationships between data, and discover hidden patterns and associations in the data.

[0090] Knowledge graphs represent data as entities (nodes) and their relationships (edges). They are good at modeling complex, interrelated data, such as semantic relationships between concepts, entities and their attributes. They are also very suitable for representing intricate relationships between data fragments.

[0091] Specifically, Figure 3 As shown, the material knowledge graph is constructed using the preprocessed material original data, and the material knowledge graph is stored in the graph database, including the following steps:

[0092] S31: Use the block model to block the preprocessed original material according to semantics to obtain multiple text blocks. The purpose of this step is to process the data more finely so that the subsequent entity relationship extraction can capture the information in the text more accurately.

[0093] S32: Extract entity relationships from multiple text blocks to obtain entities and entity relationships in each text block.

[0094] Among them, entities and relationships are the most basic units of knowledge graphs. Entities can be people, organizations, time, places, etc., and have clear boundaries and semantic meanings in the text. Through entity extraction, key information can be quickly and accurately obtained from massive data, providing basic data support for subsequent knowledge graph construction. Relationships are connections or links between entities, expressing the interaction or association between entities. Through relationship extraction, the connections and patterns between entities can be revealed. Through relationships, reasoning and queries can be performed in the knowledge graph to discover hidden connections and patterns between entities.

[0095] Furthermore, the bert-base-NER model is used to implement entity and relation recognition and extraction. It is a fine-tuned BERT model that contains 12 layers (Encoder layers), 12 self-attention heads (Attention heads) and 768 hidden layer sizes (Hidden size), with a total of about 110M parameters.

[0096] In order to improve the accuracy of the BERT-base-NER model in extracting entities and relations, question-answer pair data is used to fine-tune the BERT-base-NER model. The loss function is calculated based on the difference between the extracted entity relations and the true labels, and the parameters of the model are updated through the back-propagation algorithm to minimize the prediction error.

[0097] S33: Merge entities in multiple text blocks, eliminate duplicate entities in the merging process through entity linking, and solve the problem of representing the same real-world entity but with different entity names. After forming the material knowledge graph, store the material knowledge graph in the graph database, thereby improving the query speed of knowledge and providing reference materials for material generation.

[0098] Specifically, the entity linking method uses the Pairwise algorithm to determine whether different entities refer to the same real-world entity by calculating the semantic similarity between entities, thereby normalizing the same entity into one entity. The Pairwise algorithm calculates the relative ranking of candidate entities rather than their absolute scores by calculating the semantic similarity between any two target entities and the semantic similarity between the entity referent and the target entity. Therefore, the calculation of the loss depends on paired samples.

[0099] Among them, the graph database is a new type of NoSQL database based on graph theory. The basic meaning is to store and query data in a data structure called "graph", rather than a database that stores images. Its data storage structure and data query method are based on graph theory. The basic elements of a graph in graph theory are nodes and edges, which correspond to nodes and relationships in a graph database. In addition, graph databases can also process key-value pairs, and their advantage is that they can quickly solve complex relationship problems. In a graph database, the relationship between data and data forms a graph structure through nodes and edges, and all the characteristics of the database are implemented on this structure, such as the ability to create, read, update, and delete (Create, Read, Update, Delete, referred to as CRUD) graph data objects, as well as the ability to handle transactions and high availability. A graph mainly contains two data types: Nodes and Relationships. They each contain attributes in the form of key-value, and then the nodes are connected through relationships, forming a relational mesh structure.

[0100] Furthermore, in an embodiment of the present invention, filtering the user's input text for sensitive words according to a preset sensitive word list to obtain the filtered input text specifically includes:

[0101] S41: According to the preset sensitive word list configured by the user, a DFA algorithm is used to construct a DFA for each sensitive word in the preset sensitive word list to obtain a sensitive word filter.

[0102] Specifically, the embodiment of the present invention configures a filter to filter the questions input by the user according to a predefined sensitive word list to shield sensitive information. At the same time, a sensitive word configuration page is provided to support users to configure sensitive words according to their needs. If sensitive words are detected, appropriate measures are taken, such as replacing them with asterisks (*), deleting sensitive parts, returning replies that refuse to answer, etc., thereby achieving control over the output of the large model.

[0103] Among them, the DFA (Deterministic Finite Automaton) algorithm is a classic algorithm for string matching and text search, especially for sensitive word filtering. First, build a DFA for each sensitive word. This process starts from an initial state. For each character in the sensitive word, if there is no transition corresponding to the character in the current state, create a new state and add a transition from the current state to the new state, which is triggered by the character. If there is already a transition corresponding to the character, follow this transition to reach the next state. Repeat this process until every character of the sensitive word is processed. The state corresponding to the last character is marked as an accepting state, which means that reaching this state indicates that a sensitive word has been found.

[0104] S42: Finally, the user's input text is filtered for sensitive words through a sensitive word filter to obtain the input text with sensitive words filtered out.

[0105] Further, in an embodiment of the present invention, searching the object database, the vector database, and the graph database based on the filtered input text to obtain a first search result, a second search result, and a third search result, respectively, includes the following steps:

[0106] S51: Based on the words in the filtered input text, a full-text search is performed in the index table of the object database, and the retrieved original material is used as the first search result. The relevant information in the object database is searched through the full-text search, so as to quickly and accurately obtain the precise matching information.

[0107] Among them, full-text search creates an index for the words in the data, indicating the number and position of the words in the article. When receiving user input, the search program performs precise matching based on the pre-established index and feedbacks the found data.

[0108] Specifically, the full-text search uses the BM25 algorithm to calculate the similarity between the question and the original text based on the frequency of each word in the text and the importance of the word, and obtain the ranking of the relevance between the question and the text.

[0109] S52: Use a fine-tuned BGE model to convert the filtered input text into a retrieval vector, compare the semantic similarity between the retrieval vector and the embedded vector in the vector database, use the embedded vector with a semantic similarity greater than a preset retrieval threshold as the second retrieval result, and search for relevant information through the semantic retrieval vector database to quickly obtain semantically related information.

[0110] Among them, semantic retrieval first converts the user question into a retrieval vector using a fine-tuned BGE model, and then retrieves the top k text blocks most relevant to the user from the vector database based on semantic similarity. A retrieval threshold is set, and retrieval results are returned when the similarity is higher than the threshold.

[0111] S53: Perform community detection on the material knowledge graph in the graph database to obtain the community structure in the knowledge graph;

[0112] S54: Entity extraction is performed on the filtered input text to obtain the question entity, and the question entity is entity-linked with the entity in the material knowledge graph through the Pairwise algorithm, and the knowledge of the community structure to which the linked entity belongs is extracted as the third search result. The embodiment of the present invention searches the knowledge graph to find relevant information, thereby fully exploring the relationship between the data, expanding the search scope, and obtaining more information related to the user input.

[0113] Specifically, the community detection algorithm uses the Leiden algorithm, which can discover the community structure in the graph, that is, the group of nodes that are closely connected in the graph, which helps to understand how the different parts of the graph are organized and grouped. The algorithm recursively divides the graph into communities until a certain community size threshold is reached. The Leiden algorithm separates nodes into disjoint communities, thereby maximizing the modularity score of each community. The Leiden algorithm is a hierarchical clustering algorithm that recursively merges communities into single nodes by greedily optimizing modularity and repeating the process in a compressed graph.

[0114] Furthermore, in the embodiment of the present invention, the first search result, the second search result and the third search result are respectively similarly calculated with the filtered input text to obtain a text similarity value. Sim t and semantic similarity value Sim v , including the following steps:

[0115] S61: Based on the frequency of occurrence of each word in the filtered input text and the importance of the word, the text similarity values ​​between the filtered input text and the first search result, the second search result, and the third search result are calculated using the BM25 algorithm. Sim t This similarity calculation method is relatively direct, but it often ignores the semantic information of the text, that is, two texts may not be similar in terms of literal meaning, but they are highly related in terms of semantics.

[0116] S62: Encode the filtered input text, the first search result, the second search result, and the third search result into vectors respectively through the Word2Vec model, and calculate the cosine similarity of the vectors between the filtered input text and the first search result, the second search result, and the third search result to obtain a semantic similarity value Sim v This method can more accurately reflect the actual relationship between semantics, thereby improving the accuracy and efficiency of retrieval.

[0117] Among them, the cosine similarity calculation formula is:

[0118] ;

[0119] in, A sequence of vectors representing user input and data respectively.

[0120] Furthermore, in an embodiment of the present invention, user input and information obtained through object database, vector database, and knowledge graph retrieval are respectively spliced ​​together and input into the first largest language model, and the first largest language model generates material corresponding to the user input based on the input information.

[0121] The first language model and the second language model (i.e., the scoring model) can be open source models such as qwen, baichuan, chatGLM, and commercial models such as ChatGPT4 and Wenxinyiyan.

[0122] Furthermore, in an embodiment of the present invention, the filtered input text, the first search result, the second search result and the third search result are spliced ​​and input into the first language model to generate personalized material, including:

[0123] The filtered input text is concatenated with the first search result, the second search result and the third search result into a complete prompt.

[0124] Considering the errors in retrieval, the large language model is required to judge whether the user input is relevant to the acquired information when constructing the prompt. Therefore, after the complete prompt is input into the first large language model, the large language model will judge whether the user input is relevant to the first search result, the second search result, and the third search result. If relevant, the first large language model will answer and generate personalized materials based on the first search result, the second search result, and the third search result; if not relevant, the first large language model will answer and generate personalized materials based on its own knowledge reserves.

[0125] The main method of splicing the user input and the retrieved data in the embodiment of the present invention is to splice the user input with the retrieved data to construct a complete prompt, and then input it into the large language model to generate the corresponding answer. The key to this function is to intelligently handle the relevance of the data and the user input to ensure that the final generated answer is both accurate and relevant. Among them, prompt is the text input into Demo, which can be an entity.

[0126] Furthermore, the embodiment of the present invention provides a prompt example, specifically:

[0127]

[0128] Thinking process:

[0129] 1. Determine whether the problem is related to <data>< / data> Related to the content in.

[0130] 2. If relevant, please <data>< / data> Answer based on the content in .

[0131] 3. If not relevant, please answer on your own.

[0132] For the generated materials, the generated materials are scored by the second largest language model (also referred to as a scoring model in the present invention) to obtain corresponding scoring values .

[0133] The method for scoring the generated material by the scoring model is to input the material generated by the first language model into the scoring model for scoring, and obtain the corresponding scoring value. .

[0134] In the embodiment of the present invention, based on the idea of ​​multi-objective optimization, the text similarity of the data is comprehensively considered. , semantic similarity And the rating value of the corresponding material , calculate multiple optimization objective values .

[0135] Among them, the optimization target value The calculation method is:

[0136] ;

[0137] in, are weight values, which represent the importance of text similarity, semantic similarity and score value respectively, and are specified by the user to meet .

[0138] For the generated materials, the generated materials are scored by the second largest language model (also referred to as a scoring model in the present invention) to obtain corresponding scoring values .

[0139] Generate high-quality materials based on multiple optimization target values, and optimize the performance of each personalized material based on multiple optimization target values. Sort the generated materials and take the top ranking The personalized materials are used as the final generated materials to support applications such as intelligent question and answer, assisted writing, and copywriting generation.

[0140] The embodiments of the present invention have the following technical effects:

[0141] 1. Use a variety of data storage technologies, including object database, vector database, and knowledge graph, to achieve full-text accurate matching of data, semantic similarity retrieval, and deep relationship mining to fully extract relevant data;

[0142] 2. Use big model technology to summarize and generate materials based on the extracted relevant information, unify the material style, and generate high-quality materials, which can be applied to multiple applications such as intelligent question answering, assisted writing, and copywriting generation;

[0143] 3. Based on multi-objective optimization technology, the quality of material extraction and the quality of large model generation are comprehensively considered. On the one hand, various data storage technologies are used to fully mine relevant information as a reference for large model generation, making the generated materials safe and controllable; on the other hand, the ability of the large model is used to ignore misleading search content and generate materials with a unified style that better meets user preferences;

[0144] 4. Consider the special requirements of material generation, and precisely control the material generation through sensitive word filtering and replacement to ensure that the material generation is safe and controllable.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A material extraction and generation method based on large model and multi-storage technology, characterized in that: include: Obtaining the original material data and preprocessing the data, and storing the preprocessed original material data in the object database; A vector database is constructed based on the preprocessed original material data, and a material knowledge graph is constructed using the preprocessed original material data, and the material knowledge graph is stored in the graph database; Filter the user's input text for sensitive words according to the preset sensitive word list to obtain the filtered input text; Searching the object database, the vector database, and the graph database according to the filtered input text to obtain a first search result, a second search result, and a third search result, respectively; Calculate the similarity between the first search result, the second search result and the third search result and the filtered input text to obtain the text similarity value Sim t and semantic similarity value Sim v； The filtered input text, the first search result, the second search result, and the third search result are concatenated and input into the first language model to generate personalized material, and the personalized material is input into the second model for scoring to obtain the material score value Sco ; Based on text similarity value Sim t , semantic similarity value Sim v and the rating value of the material Sco Calculate multiple optimization target values ​​for each personalized material Obj ; Multiple optimization target values ​​for each personalized material Obj Sort them and take the top k ranked materials as the final generated materials.

2. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The obtaining of the original material data and preprocessing the data, and storing the preprocessed original material data in the object database, includes: Collect data related to various materials, extract text information from the materials, and convert the text information into a unified text format to obtain the original data of the materials; Perform data cleaning and standardization on the original material data to obtain the pre-processed original material data; Keywords are extracted from the preprocessed original material data, and an index table is created for the extracted keywords. The preprocessed original material data and the index table are stored in the object database.

3. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The constructing of a vector database based on the pre-processed original material data includes: The pre-processed original material is divided into blocks according to semantics using a block model to obtain multiple text blocks; Multiple text blocks are encoded through a fine-tuned BGE model to obtain corresponding embedding vectors, and a vector database is built based on the embedding vectors, and the embedded vectors are stored in the vector database.

4. The material extraction and generation method based on large model and multi-storage technology according to claim 3 is characterized in that: The fine-tuning process of the BGE model includes: Obtain historical question-answer pair data, use the data of correct answers to questions in the historical question-answer pair data as positive samples, and find similar answer data from the rest of the question-answer pair data as negative samples, and use the question-answer pair data, positive samples, and negative samples as training data; The semantic relevance task is regarded as a binary classification task, and the BGE model is fine-tuned using the questions, positive samples and multiple negative samples in the training data. During the fine-tuning process, the cross entropy loss function is used as the optimization objective, and the parameters of the BGE model are iteratively updated to obtain the fine-tuned BGE model.

5. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The method of constructing a material knowledge graph using the preprocessed material original data and storing the material knowledge graph in a graph database includes: The pre-processed original material is divided into blocks according to semantics using a block model to obtain multiple text blocks; Entity relationship extraction is performed on multiple text blocks to obtain entities and entity relationships in each text block; The entities in multiple text blocks are merged, and the duplicate entities in the merging process are eliminated through the entity linking method. After the material knowledge graph is formed, the material knowledge graph is stored in the graph database.

6. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The step of filtering the user's input text for sensitive words according to a preset sensitive word list to obtain the filtered input text includes: According to the preset sensitive word list configured by the user, a DFA algorithm is used to construct a DFA for each sensitive word in the preset sensitive word list to obtain a sensitive word filter; The user's input text is filtered for sensitive words through a sensitive word filter to obtain the input text with sensitive words filtered out.

7. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The filtered input text is searched in the object database, the vector database and the graph database to obtain a first search result, a second search result and a third search result, respectively, including: Based on the words in the filtered input text, a full-text search is performed in the index table of the object database, and the retrieved original material is used as the first search result; The filtered input text is converted into a search vector using a fine-tuned BGE model, the search vector is compared with the embedded vector in the vector database for semantic similarity, and the embedded vector with a semantic similarity greater than a preset search threshold is taken as the second search result; Perform community detection on the material knowledge graph in the graph database to obtain the community structure in the knowledge graph; Entity extraction is performed on the filtered input text to obtain the question entity, and the question entity is entity linked with the entity in the material knowledge graph through the Pairwise algorithm, and the knowledge of the community structure to which the linked entity belongs is extracted as the third retrieval result.

8. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The first search result, the second search result and the third search result are respectively calculated with the filtered input text to obtain a text similarity value. Sim t and semantic similarity value Sim v ,include: According to the frequency of occurrence of each word in the filtered input text and the importance of the word, the text similarity value between the filtered input text and the first search result, the second search result and the third search result is calculated by the BM25 algorithm Sim t ; The filtered input text, the first search result, the second search result, and the third search result are encoded into vectors through the Word2Vec model, and the cosine similarity between the filtered input text and the first search result, the second search result, and the third search result is calculated to obtain the semantic similarity value. Sim v .

9. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The step of splicing the filtered input text, the first search result, the second search result, and the third search result and inputting them into the first language model to generate personalized material includes: Concatenate the filtered input text with the first search result, the second search result, and the third search result into a complete prompt; The complete prompt is input into the first large language model, and the large language model determines whether the user input is relevant to the first, second and third search results. If so, the large language model generates personalized material based on the first, second and third search results. If not, the large language model generates personalized material based on its own knowledge reserves.

10. The material extraction and generation method based on large model and multi-storage technology according to claim 1 is characterized in that: The multiple optimization objective values Obj The calculation method is: ； in, are weight values, which respectively represent the importance of text similarity, semantic similarity and score value, and satisfy .

Images