A large model-based knowledge base management method, device, equipment and storage medium

Abstract

This application discloses a knowledge base management method, apparatus, device, and storage medium based on a large model, relating to the field of artificial intelligence technology. The method includes: identifying text paragraphs in the headings of each level in each document to be saved; processing each text paragraph using a large model; determining the first similarity of each first text by generating a list of thought-summary text paragraphs based on a summary generation task; determining a first community set using a first graph network established based on a first preset similarity threshold and the first similarity, along with a community detection algorithm; performing knowledge cleaning on the first community set to obtain a list of text paragraphs to be processed; determining a second community set using a second graph network established based on a second preset similarity threshold and the second similarity of each second text in the list of text paragraphs to be processed, along with a community detection algorithm; and performing knowledge aggregation on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list. This improves the efficiency of managing data in the knowledge base.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a knowledge base management method, apparatus, device, and storage medium based on a large model. Background Technology

[0002] While large models have achieved remarkable success, they still face many significant limitations, especially when dealing with specific domains or knowledge-intensive tasks. When faced with queries that exceed the scope of their training data or require current information, they often exhibit an "illusion" phenomenon, generating inaccurate or fictitious content.

[0003] Currently, while RAG (Retrieval-Augmented Generation) technology can cleverly incorporate external knowledge bases into the generation process, selecting the most relevant information for large models by retrieving relevant document fragments and calculating semantic similarity, existing technologies still fall short when it comes to processing unstructured data. Traditional methods often rely on specific tools for file reading and text segmentation, but this process struggles to guarantee the coherence and completeness of information, especially at the level of extracting macro-level knowledge, where it has significant shortcomings.

[0004] As can be seen from the above, how to improve the efficiency of managing data in the knowledge base during the management of a knowledge base based on a large model is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for knowledge base management based on a large model, which can improve the efficiency of managing data in the knowledge base during the knowledge base management process based on a large model, thereby improving the efficiency of the production process. The specific solution is as follows:

[0006] Firstly, this application provides a knowledge base management method based on a large model, including:

[0007] The text paragraphs in the headings of each level in each document to be saved are identified, and the large model is used to process each text paragraph to obtain the corresponding summary generation task, so as to generate a list of summary text paragraphs including several hierarchical relationships using the summary generation task.

[0008] The first similarity of each first text in the summary text paragraph list is determined by using a preset similarity determination rule, and a first graph network is established based on a first preset similarity threshold and each first similarity. Then, the first graph network and a community detection algorithm are used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a list of text paragraphs to be processed.

[0009] A second graph network is established based on the second similarity threshold and the second similarity corresponding to each second text in the list of text segments to be processed. Then, the second graph network and the community detection algorithm are used to determine the second community set. The text information of each community in the second community set is aggregated to obtain a knowledge-integrated paragraph list, and the knowledge-integrated paragraph list is saved to a preset knowledge base.

[0010] Optionally, the step of determining the text paragraphs in each level of headings in each document to be saved, and processing each text paragraph using the large model to obtain the corresponding summary generation task, so as to generate a list of summary text paragraphs including several hierarchical relationships using the summary generation task, includes:

[0011] Identify the headings at each level in each document to be saved, and determine the corresponding set of text paragraphs based on each heading level;

[0012] The large model is used to process the instructions of each set of text segments to obtain the corresponding summary task generation instructions. A summary generation task is then generated based on the summary task generation instructions. The summary generation task is then used to generate summary text segments that include several hierarchical relationships. The hierarchical relationships corresponding to each summary text segment are different.

[0013] The abstract text paragraphs are recursively processed according to a preset hierarchical order to obtain a list of abstract text paragraphs.

[0014] Optionally, the step of generating a list of summary text paragraphs including several hierarchical relationships using the summary generation task includes:

[0015] The document to be saved is segmented using preset natural text delimiters and preset length segmentation thresholds to obtain segmented documents. Then, a preset community detection algorithm is used to perform cluster analysis on the segmented documents to obtain clustering results. The preset natural text delimiters include line breaks, periods, exclamation marks, and question marks. The preset length segmentation threshold is a threshold set based on semantic integrity and a preset length threshold.

[0016] The large model is used to determine whether there is redundant information in the clustering results. If there is redundant information in the clustering results, the clustering results are processed using preset knowledge cleaning rules to obtain a list of summary text paragraphs including several hierarchical relationships.

[0017] Optionally, the step of determining the first similarity corresponding to each first text in the list of abstract text paragraphs using a preset similarity determination rule, and establishing a first graph network based on a first preset similarity threshold and each first similarity, includes:

[0018] The first similarity corresponding to each first text in the list of abstract text paragraphs is determined using a preset similarity determination rule, and it is determined whether each first similarity is greater than a first preset similarity threshold; the preset similarity determination rule is a numerical value determined based on the distance between each first text in the semantic vector space;

[0019] If each of the first similarities is greater than the first preset similarity threshold, then text nodes and text relationships corresponding to the first similarity are generated in the initial graph network to obtain the current graph network, and the current text judgment count is updated; the current text judgment count is the number of texts corresponding to the first similarity of each of the first texts that have been determined using the preset similarity determination rule.

[0020] If the number of current text judgments is less than the total number of texts corresponding to the first text in the summary text paragraph list, then proceed to the step of determining the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and then set the current graph network as the first graph network.

[0021] Optionally, the step of determining a first community set using the first graph network and community detection algorithm, and performing knowledge cleaning on the first community set to obtain a list of text paragraphs to be processed includes:

[0022] Perform a node traversal operation on each neighbor node corresponding to the node to be processed in the first graph network to obtain the modularity gain; the modularity gain is the modularity gain corresponding to the node to be processed being assigned to an adjacent community.

[0023] The community corresponding to the largest modularity gain among all the modularity gains is assigned to an adjusted community to obtain the assignment adjustment result.

[0024] The attribution adjustment results are processed using a community detection algorithm to obtain a first community set. Knowledge cleaning and redundancy removal operations are then performed on each text segment in the first community set to obtain a list of text segments to be processed.

[0025] Optionally, the step of performing knowledge cleaning and redundancy removal operations on each text segment in the first community set to obtain a list of text segments to be processed includes:

[0026] Key information extraction operations are performed on each text segment in the same community using preset key information extraction rules and a large model to obtain information extraction results. Then, the information extraction results are subjected to duplicate expression elimination operations using preset duplicate expression elimination rules and the large model to obtain cleaned text. The cleaned text includes the core semantic features of the document to be saved.

[0027] By constructing and dividing communities across documents, knowledge is integrated from similar text segments from different documents to generate aggregated text with temporal relevance and logical coherence.

[0028] Optionally, the step of performing knowledge aggregation on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list, and saving the knowledge-integrated paragraph list to a preset knowledge base, includes:

[0029] Using preset joint analysis rules, the text information of each community in the second community set is jointly analyzed to obtain joint analysis results; the joint analysis results include text content describing the same topic in different documents to be saved;

[0030] The joint analysis results are processed using the large model to obtain an initial knowledge integration segment with temporal evolution characteristics. The initial knowledge integration segment is then processed using preset structured rules to obtain a target knowledge integration segment with a structured form.

[0031] A knowledge integration paragraph list is generated based on each of the target knowledge integration paragraphs, and the knowledge integration paragraph list is saved to a preset knowledge base to generate a knowledge graph.

[0032] Secondly, this application provides a knowledge base management device based on a large model, comprising:

[0033] The first paragraph list generation module is used to determine the text paragraphs in the headings of each level in each document to be saved, and to process each text paragraph using the large model to obtain the corresponding summary generation task, so as to generate a summary text paragraph list including several hierarchical relationships using the summary generation task.

[0034] The second paragraph list generation module is used to determine the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and to establish a first graph network based on a first preset similarity threshold and each first similarity. Then, the first graph network and community detection algorithm are used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a text paragraph list to be processed.

[0035] The text information aggregation module is used to establish a second graph network based on a second preset similarity threshold and the second similarity corresponding to each second text in the list of text paragraphs to be processed. Then, the second graph network and the community detection algorithm are used to determine a second community set, and the text information of each community in the second community set is aggregated to obtain a knowledge-integrated paragraph list. The knowledge-integrated paragraph list is then saved to a preset knowledge base.

[0036] Thirdly, this application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the aforementioned knowledge base management method based on a large model.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned knowledge base management method based on a large model.

[0040] As can be seen from the above, before performing knowledge base management based on a large model, this application needs to determine the text paragraphs in the headings of each level in each document to be saved, and use the large model to process each text paragraph to obtain the corresponding summary generation task, so as to generate a list of summary text paragraphs including several hierarchical relationships using the summary generation task; determine the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and establish a first graph network based on the first preset similarity threshold and each first similarity, then use the first graph network and community detection algorithm to determine the first community set, and perform knowledge cleaning on the first community set to obtain a list of text paragraphs to be processed; establish a second graph network based on the second preset similarity threshold and the second similarity corresponding to each second text in the list of text paragraphs to be processed, then use the second graph network and community detection algorithm to determine the second community set, and perform knowledge aggregation on the text information of each community in the second community set to obtain a knowledge integration paragraph list, and save the knowledge integration paragraph list to the preset knowledge base.

[0041] Therefore, this application first needs to identify the text paragraphs in the headings of each level in each document to be saved, and then process each text paragraph using a large model to obtain the corresponding summary generation task. This task generates a list of summary text paragraphs including several hierarchical relationships. Subsequently, a first similarity is determined for each first text in the summary text paragraph list using a preset similarity determination rule. A first graph network is then established based on the first preset similarity threshold and each first similarity. The first graph network and a community detection algorithm are then used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a list of text paragraphs to be processed. Finally, a second graph network is established based on a second preset similarity threshold and the second similarity for each second text in the list of text paragraphs to be processed. The second graph network and a community detection algorithm are then used to determine a second community set, and knowledge aggregation is performed on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list. This knowledge-integrated paragraph list is then saved to a preset knowledge base. This improves the efficiency of data management in the knowledge base during the large model-based knowledge base management process, thereby improving the efficiency of the production process. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This application discloses a flowchart of a knowledge base management method based on a large model.

[0044] Figure 2 This is a schematic diagram illustrating a specific method for generating a list of abstract text paragraphs as disclosed in this application;

[0045] Figure 3 This is a schematic diagram illustrating a specific method of knowledge cleaning of text disclosed in this application;

[0046] Figure 4 This is a schematic diagram illustrating a specific information aggregation operation on different documents disclosed in this application;

[0047] Figure 5 This is a schematic diagram of the structure of a knowledge base management device based on a large model disclosed in this application;

[0048] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] While large-scale knowledge base models have achieved remarkable success, they often exhibit a "phantom" phenomenon when handling specific domains or knowledge-intensive tasks, generating inaccurate or fabricated content when faced with queries that exceed the scope of their training data or require immediate information. Currently, while RAG (Resource Aggregation and Query) technology cleverly incorporates external knowledge bases into the generation process, retrieving relevant document fragments and calculating semantic similarity to select the most relevant information for the large-scale model, existing technologies still fall short in processing unstructured data. Traditional methods often rely on specific tools for file reading and text segmentation, but this process struggles to guarantee the coherence and completeness of information, especially at the level of extracting macro-level knowledge, where it has significant shortcomings. Therefore, this application provides a knowledge base management method based on large-scale models, which can improve the efficiency of managing data in the knowledge base during the large-scale model-based knowledge base management process, thereby improving the efficiency of the production process.

[0051] See Figure 1 As shown in the figure, this invention discloses a knowledge base management method based on a large model, including:

[0052] Step S11: Determine the text paragraphs in each level heading of each document to be saved, and process each text paragraph using the large model to obtain the corresponding summary generation task, so as to generate a list of summary text paragraphs including several hierarchical relationships using the summary generation task.

[0053] In this embodiment, in the first specific implementation, the large model can be the large model used in the financial advisor robot to generate investment strategy explanations after retrieving real-time market data; in the second specific implementation, the large model can be the large model relied upon by the hotel booking assistant, and is used to answer users' questions about cancellation policies based on data in the hotel policy database; in the third specific implementation, the large model can be the large model used to retrieve stock data for the day and automatically generate financial news; in the fourth specific implementation, the large model can be the large model used to generate personalized advertising copy based on the product database.

[0054] It is understood that, in the process of managing the knowledge base based on a large model, the embodiments of this application first need to identify the hierarchical headings of the documents to be saved, and then use the large model and a specific prompt to perform a summary generation task on the text paragraphs under the same hierarchical heading, obtaining the corresponding summary generation task. Then, the summary generation task is used to generate a list of summary text paragraphs including several hierarchical relationships, ensuring the hierarchical structure and logical coherence of the information. A schematic diagram of generating the summary text paragraph list is shown below. Figure 2 As shown.

[0055] Specifically, the process involves identifying text paragraphs within the headings of each level in each document to be saved, processing these paragraphs using a large model to obtain corresponding summary generation tasks, and then using these tasks to generate a list of summary text paragraphs with several hierarchical relationships. This process may include: identifying the headings of each level in each document to be saved, and determining the corresponding set of text paragraphs based on each heading; using the large model to process the text paragraph sets to obtain corresponding summary task generation instructions, generating summary tasks based on these instructions, and then using these tasks to generate summary text paragraphs with several hierarchical relationships; each summary text paragraph corresponds to a different hierarchical relationship; and recursively processing each summary text paragraph according to a preset hierarchical recursive order to obtain a list of summary text paragraphs.

[0056] In one specific implementation, this application embodiment uses pseudocode to implement the operation of generating a list of summary text paragraphs, and the pseudocode is as follows:

[0057] Input: A list of text paragraphs P_D of document D;

[0058] process:

[0059] For each level in the hierarchical structure of document D:

[0060] summary_level = []

[0061] For each paragraph P in the set of paragraphs at level:

[0062] Summary text_summary = large_model.generate_summary(P, prompt_for_level)

[0063] summaries_level.append(text_summary)

[0064] # Recursively build higher-level summaries (if needed)

[0065] Output: A list containing summaries at different levels.

[0066] In this embodiment, after obtaining the list of abstract text paragraphs, knowledge cleaning and deduplication operations need to be performed on each document in the list of abstract text paragraphs: First, read the text in the list of abstract text paragraphs. Then, the text in the summary text segment list is segmented using preset natural text delimiters (such as newline characters, periods, exclamation marks, question marks, etc.) to obtain a series of text segments no larger than max_len. Where n represents the document The text was divided into n segments. Then, a community detection algorithm was used to cluster the segmented text segments, and a large language model was used to perform knowledge cleaning on texts clustered in the same community, removing duplicate information. The diagram is shown below. Figure 3 As shown.

[0067] It is worth mentioning that for different documents in the same knowledge base, such as text segments and text segment The corresponding text segmentation information is shown below: and This application embodiment requires the use of community detection algorithms to cluster different text segments to obtain similar text segments in different documents, and then performing information aggregation operations on the similar text segments. In one specific implementation, by performing information aggregation operations on different documents within the same knowledge base, such as company annual reports from different years, annual revenue information can be obtained, thereby achieving cross-document knowledge integration and expansion. This not only enriches the diversity of the knowledge base but also enhances the coherence and timeliness of information. A schematic diagram of information aggregation operations on different documents is shown below. Figure 4 As shown.

[0068] Specifically, the task of generating a summary text paragraph list containing several hierarchical relationships can include: segmenting the document to be saved using preset natural text delimiters and preset length segmentation thresholds to obtain segmented documents; performing cluster analysis on the segmented documents using a preset community detection algorithm to obtain clustering results; preset natural text delimiters include line breaks, periods, exclamation marks, and question marks; the preset length segmentation threshold is a threshold set based on semantic integrity and the preset length threshold; using a large model to determine whether there is redundant information in the clustering results; if there is redundant information in the clustering results, processing the clustering results using preset knowledge cleaning rules to obtain a summary text paragraph list containing several hierarchical relationships.

[0069] Step S12: Determine the first similarity corresponding to each first text in the summary text paragraph list using the preset similarity determination rules, and establish a first graph network based on the first preset similarity threshold and each first similarity. Then, use the first graph network and community detection algorithm to determine the first community set, and perform knowledge cleaning on the first community set to obtain the text paragraph list to be processed.

[0070] In this embodiment, after obtaining the list of summary text paragraphs, it is necessary to determine the similarity of each text in the list of summary text paragraphs in order to determine the graph network based on the obtained similarity. Specifically, determining the first similarity corresponding to each first text in the abstract text paragraph list using a preset similarity determination rule, and establishing a first graph network based on a first preset similarity threshold and each first similarity, may include: determining the first similarity corresponding to each first text in the abstract text paragraph list using a preset similarity determination rule, and determining whether each first similarity is greater than the first preset similarity threshold; the preset similarity determination rule is a numerical value determined based on the distance between each first text in the semantic vector space; if each first similarity is greater than the first preset similarity threshold, then generating text nodes and text relationships corresponding to the first similarity in the initial graph network to obtain the current graph network, and updating the current text judgment count; the current text judgment count is the number of texts corresponding to the first similarity of each first text determined using the preset similarity determination rule; determining whether the current text judgment count is less than the total number of texts corresponding to the first texts in the abstract text paragraph list; if the current text judgment count is less than the total number of texts, then jumping to the step of determining the first similarity corresponding to each first text in the abstract text paragraph list using the preset similarity determination rule, and then setting the current graph network as the first graph network.

[0071] In one specific implementation, the pseudocode for knowledge cleaning of documents is shown below:

[0072] Input: First preset similarity threshold and documents P_D_cut is a list of all text paragraphs of length K:

[0073] ;

[0074] in, For document The nth paragraph, For document The nth summary.

[0075] Step A: Construct the graph network G:

[0076] For each paragraph P1, P2 in P_D_cut:

[0077] if sim(P1, P2) > :

[0078] Add nodes v_P1, v_P2 and edge e_{P1P2} to graph G.

[0079] Furthermore, after obtaining the first graph network, this embodiment of the application needs to use the first graph network to perform knowledge cleaning on the first community set, thereby obtaining a list of text segments to be processed. Specifically, the first community set is determined using the first graph network and the community detection algorithm, and knowledge cleaning is performed on the first community set to obtain a list of text segments to be processed. This may include: performing node traversal operations on each neighbor node corresponding to the node to be processed in the first graph network to obtain the modularity gain; the modularity gain is the modularity gain corresponding to the node to be processed being assigned to an adjacent community; adjusting the community corresponding to the modularity gain with the largest value among all modularity gains to obtain the adjustment result; processing the adjustment result using the community detection algorithm to obtain the first community set, and performing knowledge cleaning and redundant information deduplication operations on each text segment in the first community set to obtain a list of text segments to be processed.

[0080] In one specific implementation, the pseudocode for executing the community detection algorithm is shown below:

[0081] for each node in G:

[0082] best neighbor node_best = None

[0083] Maximum module gain gain_max = 0

[0084] for each Neighboring nodes:

[0085] # Calculate the gain of this node belonging to the current community

[0086] gain = calculate_modularity_gain(v, node)

[0087] if gain > gain_max:

[0088] gain_max = gain

[0089] node_best = node

[0090] if node_best:

[0091] Will Join the community where node_best is located

[0092] Until the community affiliation of all nodes is stable and no longer changes.

[0093] in, For each node in the first graph network, This represents the number of documents.

[0094] In this embodiment, after obtaining the first community set, the present application embodiment needs to perform knowledge cleaning and redundant information deduplication operations on each text segment in the first community set to obtain a list of text segments to be processed. Specifically, performing knowledge cleaning and redundant information deduplication operations on each text segment in the first community set to obtain a list of text segments to be processed may include: using preset key information extraction rules and a large model to perform key information extraction operations on each text segment in the same community to obtain information extraction results, and then using preset duplicate expression elimination rules and a large model to perform duplicate expression elimination operations on the information extraction results to obtain cleaned text; the cleaned text includes the core semantic features of the document to be saved; through cross-document graph network construction and community division, knowledge integration is performed on similar text segments from different documents to generate aggregated text with temporal correlation and logical coherence.

[0095] In one specific implementation, the pseudocode for knowledge cleaning of documents is shown below:

[0096] For each community C in the set of communities G:

[0097] The cleaned text list L_C = []

[0098] for each paragraph P in C:

[0099] L_C.append(large_model.clean_text(P))

[0100] Output L_C.

[0101] Furthermore, the pseudocode for knowledge aggregation from different documents is shown below:

[0102] Input: Second preset similarity threshold A list of text paragraphs corresponding to z documents. ;in, .

[0103] process:

[0104] Similar to the graph network construction and community detection algorithm execution operations described above, but spanning different documents...

[0105] # Build a cross-document graph network and perform community discovery

[0106] # Knowledge aggregation of text paragraphs within each community

[0107] Output: A list of knowledge-integrated paragraphs across documents.

[0108] Step S13: Based on the second preset similarity threshold and the second similarity corresponding to each second text in the list of text segments to be processed, establish a second graph network. Then, use the second graph network and the community detection algorithm to determine the second community set, and perform knowledge aggregation on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list. Save the knowledge-integrated paragraph list to a preset knowledge base.

[0109] In this embodiment, after obtaining the list of text segments to be processed, the present application embodiment first needs to determine the second similarity corresponding to each second text in the list of text segments to be processed, so as to establish a second graph network based on each second similarity and a second preset similarity threshold. Subsequently, the second graph network and community detection algorithm are used to determine a second community set, and the text information of each community in the second community set is aggregated to obtain a knowledge-integrated segment list, so that the knowledge-integrated segment list can be finally saved to a preset knowledge base.

[0110] Specifically, knowledge aggregation is performed on the text information of each community in the second community set to obtain a knowledge integration paragraph list, which is then saved to a preset knowledge base. This process may include: using preset joint analysis rules to perform joint analysis on the text information of each community in the second community set to obtain joint analysis results; the joint analysis results include text content describing the same topic in different documents to be saved; using a large model to process the joint analysis results to obtain initial knowledge integration paragraphs with temporal evolution characteristics, and using preset structuring rules to process the initial knowledge integration paragraphs to obtain target knowledge integration paragraphs with a structured form; generating a knowledge integration paragraph list based on each target knowledge integration paragraph, and saving the knowledge integration paragraph list to a preset knowledge base to generate a knowledge graph.

[0111] Therefore, the embodiments of this application first need to determine the text paragraphs in the headings of each level in each document to be saved, and then process each text paragraph using a large model to obtain the corresponding summary generation task, thereby generating a list of summary text paragraphs including several hierarchical relationships. Subsequently, a first similarity is determined for each first text in the summary text paragraph list using a preset similarity determination rule, and a first graph network is established based on a first preset similarity threshold and each first similarity. Then, a first community set is determined using the first graph network and a community detection algorithm, and knowledge cleaning is performed on the first community set to obtain a list of text paragraphs to be processed. Finally, a second graph network is established based on a second preset similarity threshold and a second similarity for each second text in the list of text paragraphs to be processed. Then, a second community set is determined using the second graph network and a community detection algorithm, and knowledge aggregation is performed on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list, which is then saved to a preset knowledge base. In this way, the efficiency of managing data in the knowledge base is improved during the knowledge base management process based on the large model, thereby improving the efficiency of the production process.

[0112] Accordingly, see Figure 5 As shown, this application also provides a knowledge base management device based on a large model, including:

[0113] The first paragraph list generation module 11 is used to determine the text paragraphs in the headings of each level in each document to be saved, and to process each text paragraph using the large model to obtain the corresponding summary generation task, so as to generate a summary text paragraph list including several hierarchical relationships using the summary generation task.

[0114] The second paragraph list generation module 12 is used to determine the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and to establish a first graph network based on a first preset similarity threshold and each first similarity. Then, the first graph network and community detection algorithm are used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a text paragraph list to be processed.

[0115] The text information aggregation module 13 is used to establish a second graph network based on a second preset similarity threshold and the second similarity corresponding to each second text in the list of text paragraphs to be processed. Then, the second graph network and the community detection algorithm are used to determine a second community set, and the text information of each community in the second community set is aggregated to obtain a knowledge-integrated paragraph list. The knowledge-integrated paragraph list is then saved to a preset knowledge base.

[0116] As can be seen from the above, before performing knowledge base management based on a large model, this embodiment first needs to determine the text paragraphs in the headings of each level in each document to be saved, and then process each text paragraph using the large model to obtain the corresponding summary generation task. This summary generation task generates a list of summary text paragraphs including several hierarchical relationships. Subsequently, a first similarity is determined for each first text in the summary text paragraph list using a preset similarity determination rule. A first graph network is then established based on a first preset similarity threshold and each first similarity. The first graph network and a community detection algorithm are then used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a list of text paragraphs to be processed. Finally, a second graph network is established based on a second preset similarity threshold and a second similarity for each second text in the list of text paragraphs to be processed. The second graph network and a community detection algorithm are then used to determine a second community set, and knowledge aggregation is performed on the text information of each community in the second community set to obtain a knowledge-integrated paragraph list. This knowledge-integrated paragraph list is then saved to a preset knowledge base. In this way, the efficiency of managing data in the knowledge base is improved during the knowledge base management process based on the large model, thereby improving the efficiency of the production process.

[0117] In some specific embodiments, the first paragraph list generation module 11 may specifically include:

[0118] The text paragraph set determination unit is used to determine the headings at each level in each document to be saved, and to determine the corresponding text paragraph sets based on each heading at each level.

[0119] The abstract text paragraph generation unit is used to perform instruction generation processing on each of the text paragraph sets using the large model to obtain the corresponding abstract task generation instruction, and generate an abstract generation task based on the abstract task generation instruction. Then, the abstract generation task is used to generate abstract text paragraphs including several hierarchical relationships; the hierarchical relationships corresponding to each of the abstract text paragraphs are different.

[0120] The abstract text paragraph recursion unit is used to recursively process each of the abstract text paragraphs according to a preset hierarchical recursive order to obtain a list of abstract text paragraphs.

[0121] In some specific embodiments, the first paragraph list generation module 11 may specifically include:

[0122] The clustering result determination unit is used to segment the document to be saved using a preset natural text delimiter and a preset length segmentation threshold to obtain segmented documents, and to perform clustering analysis on the segmented documents using a preset community detection algorithm to obtain clustering results; the preset natural text delimiter includes line breaks, periods, exclamation marks, and question marks; the preset length segmentation threshold is a threshold set based on semantic integrity and a preset length threshold;

[0123] The clustering result processing unit is used to determine whether there is redundant information in the clustering result using the large model. If there is redundant information in the clustering result, the clustering result is processed using preset knowledge cleaning rules to obtain a list of summary text paragraphs including several hierarchical relationships.

[0124] In some specific embodiments, the second paragraph list generation module 12 may specifically include:

[0125] A similarity determination unit is used to determine the first similarity corresponding to each first text in the list of abstract text paragraphs using a preset similarity determination rule, and to determine whether each first similarity is greater than a first preset similarity threshold; the preset similarity determination rule is a numerical value determined based on the distance between each first text in the semantic vector space;

[0126] The graph network generation unit is used to generate text nodes and text relationships corresponding to the first similarity in the initial graph network if each of the first similarities is greater than the first preset similarity threshold, thereby obtaining the current graph network and updating the current text judgment count; the current text judgment count is the number of texts corresponding to the first similarity of each of the first texts that have been determined using the preset similarity determination rule.

[0127] The graph network processing unit is used to determine whether the current text judgment count is less than the total number of texts corresponding to the first text in the summary text paragraph list. If the current text judgment count is less than the total number of texts, the process jumps to the step of determining the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and then sets the current graph network as the first graph network.

[0128] In some specific embodiments, the second paragraph list generation module 12 may specifically include:

[0129] The modularity gain determination unit is used to perform a node traversal operation on each neighbor node corresponding to the node to be processed in the first graph network to obtain the modularity gain; the modularity gain is the modularity gain corresponding to the node to be processed being assigned to an adjacent community.

[0130] The attribution adjustment result determination unit is used to adjust the attribution of the community corresponding to the largest modularity gain among the modularity gains, and obtain the attribution adjustment result.

[0131] The text segment cleaning unit is used to process the attribution adjustment results using a community discovery algorithm to obtain a first community set, and to perform knowledge cleaning and redundant information removal operations on each text segment in the first community set to obtain a list of text segments to be processed.

[0132] In some specific embodiments, the second paragraph list generation module 12 may specifically include:

[0133] The post-cleaning text determination unit is used to perform key information extraction operations on each text segment in the same community using preset key information extraction rules and a large model to obtain information extraction results. Then, it uses preset duplicate expression elimination rules and the large model to perform duplicate expression elimination operations on the information extraction results to obtain cleaned text. The cleaned text includes the core semantic features of the document to be saved.

[0134] The aggregated text determination unit is used to integrate knowledge from similar text segments from different documents through cross-document graph network construction and community segmentation, generating aggregated text with temporal relevance and logical coherence.

[0135] In some specific embodiments, the text information aggregation module 13 may specifically include:

[0136] The text content determination unit is used to perform joint analysis on the text information of each community in the second community set using preset joint analysis rules to obtain joint analysis results; the joint analysis results include text content describing the same topic in different documents to be saved;

[0137] The knowledge integration paragraph generation unit is used to process the joint analysis results using the large model to obtain an initial knowledge integration paragraph with temporal evolution characteristics, and to process the initial knowledge integration paragraph using preset structured rules to obtain a target knowledge integration paragraph with a structured form.

[0138] The knowledge graph determination unit is used to generate a knowledge integration paragraph list based on each of the target knowledge integration paragraphs, and save the knowledge integration paragraph list to a preset knowledge base to generate a knowledge graph.

[0139] Furthermore, embodiments of this application also disclose an electronic device, Figure 6This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the knowledge base management method based on a large model disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0140] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0141] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0142] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the large-model-based knowledge base management method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0143] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned knowledge base management method based on a large model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0145] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0147] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0148] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A knowledge base management method based on a large model, characterized in that, include: The text paragraphs in the headings of each level in each document to be saved are identified, and the large model is used to process each text paragraph to obtain the corresponding summary generation task, so as to generate a list of summary text paragraphs including several hierarchical relationships using the summary generation task. The first similarity of each first text in the summary text paragraph list is determined by using a preset similarity determination rule, and a first graph network is established based on a first preset similarity threshold and each first similarity. Then, the first graph network and a community detection algorithm are used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a list of text paragraphs to be processed. A second graph network is established based on the second preset similarity threshold and the second similarity corresponding to each second text in each of the text paragraph lists to be processed. Then, the second graph network and the community detection algorithm are used to determine the second community set. The text information of each community in the second community set is aggregated to obtain a knowledge integration paragraph list. The knowledge integration paragraph list is then saved to a preset knowledge base. The step of determining a first community set using the first graph network and community detection algorithm, and performing knowledge cleaning on the first community set to obtain a list of text segments to be processed includes: performing node traversal operations on each neighbor node corresponding to the node to be processed in the first graph network to obtain a modularity gain; the modularity gain is the modularity gain corresponding to the node to be processed being assigned to an adjacent community; adjusting the community corresponding to the modularity gain with the largest value among the modularity gains to obtain an adjustment result; processing the adjustment result using the community detection algorithm to obtain a first community set, and performing knowledge cleaning and redundant information removal operations on each text segment in the first community set to obtain a list of text segments to be processed. The step of performing knowledge cleaning and redundant information removal operations on each text segment in the first community set to obtain a list of text segments to be processed includes: using preset key information extraction rules and a large model to perform key information extraction operations on each text segment in the same community to obtain information extraction results; then using preset duplicate expression elimination rules and the large model to perform duplicate expression elimination operations on the information extraction results to obtain cleaned text; the cleaned text includes the core semantic features of the document to be saved; and through cross-document graph network construction and community division, knowledge integration is performed on similar text segments from different documents to generate aggregated text with temporal correlation and logical coherence.

2. The knowledge base management method based on a large model according to claim 1, characterized in that, The process involves determining the text paragraphs in each level of headings within each document to be saved, processing each text paragraph using the large model, obtaining the corresponding summary generation task, and using the summary generation task to generate a list of summary text paragraphs including several hierarchical relationships, including: Identify the headings at each level in each document to be saved, and determine the corresponding set of text paragraphs based on each heading level; The large model is used to process the instructions of each set of text segments to obtain the corresponding summary task generation instructions. A summary generation task is then generated based on the summary task generation instructions. The summary generation task is then used to generate summary text segments that include several hierarchical relationships. The hierarchical relationships corresponding to each summary text segment are different. The abstract text paragraphs are recursively processed according to a preset hierarchical order to obtain a list of abstract text paragraphs.

3. The knowledge base management method based on a large model according to claim 1, characterized in that, The process of generating a list of summary text paragraphs with several hierarchical relationships using the summary generation task includes: The document to be saved is segmented using preset natural text delimiters and preset length segmentation thresholds to obtain segmented documents. Then, a preset community detection algorithm is used to perform cluster analysis on the segmented documents to obtain clustering results. The preset natural text delimiters include line breaks, periods, exclamation marks, and question marks. The preset length segmentation threshold is a threshold set based on semantic integrity and a preset length threshold. The large model is used to determine whether there is redundant information in the clustering results. If there is redundant information in the clustering results, the clustering results are processed using preset knowledge cleaning rules to obtain a list of summary text paragraphs including several hierarchical relationships.

4. The knowledge base management method based on a large model according to claim 1, characterized in that, The step of determining the first similarity corresponding to each first text in the list of abstract text paragraphs using a preset similarity determination rule, and establishing a first graph network based on a first preset similarity threshold and each first similarity, includes: The first similarity corresponding to each first text in the list of abstract text paragraphs is determined using a preset similarity determination rule, and it is determined whether each first similarity is greater than a first preset similarity threshold; the preset similarity determination rule is a numerical value determined based on the distance between each first text in the semantic vector space; If each of the first similarities is greater than the first preset similarity threshold, then text nodes and text relationships corresponding to the first similarity are generated in the initial graph network to obtain the current graph network, and the current text judgment count is updated; the current text judgment count is the number of texts corresponding to the first similarity of each of the first texts that have been determined using the preset similarity determination rule. If the number of current text judgments is less than the total number of texts corresponding to the first text in the summary text paragraph list, then proceed to the step of determining the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and then set the current graph network as the first graph network.

5. The knowledge base management method based on a large model according to any one of claims 1 to 4, characterized in that, The step of aggregating the text information of each community in the second community set to obtain a knowledge-integrated paragraph list, and saving the knowledge-integrated paragraph list to a preset knowledge base, includes: Using preset joint analysis rules, the text information of each community in the second community set is jointly analyzed to obtain joint analysis results; the joint analysis results include text content describing the same topic in different documents to be saved; The joint analysis results are processed using the large model to obtain an initial knowledge integration segment with temporal evolution characteristics. The initial knowledge integration segment is then processed using preset structured rules to obtain a target knowledge integration segment with a structured form. A knowledge integration paragraph list is generated based on each of the target knowledge integration paragraphs, and the knowledge integration paragraph list is saved to a preset knowledge base to generate a knowledge graph.

6. A knowledge base management device based on a large model, characterized in that, include: The first paragraph list generation module is used to determine the text paragraphs in the headings of each level in each document to be saved, and to process each text paragraph using the large model to obtain the corresponding summary generation task, so as to generate a summary text paragraph list including several hierarchical relationships using the summary generation task. The second paragraph list generation module is used to determine the first similarity corresponding to each first text in the summary text paragraph list using a preset similarity determination rule, and to establish a first graph network based on a first preset similarity threshold and each first similarity. Then, the first graph network and community detection algorithm are used to determine a first community set, and knowledge cleaning is performed on the first community set to obtain a text paragraph list to be processed. The text information aggregation module is used to establish a second graph network based on a second preset similarity threshold and the second similarity corresponding to each second text in each of the text paragraph lists to be processed. Then, the second graph network and the community detection algorithm are used to determine a second community set, and the text information of each community in the second community set is aggregated to obtain a knowledge integration paragraph list. The knowledge integration paragraph list is then saved to a preset knowledge base. Specifically, the second paragraph list generation module may include: performing a node traversal operation on each neighbor node corresponding to the node to be processed in the first graph network to obtain a modularity gain; the modularity gain is the modularity gain corresponding to the node to be processed when it is assigned to an adjacent community; adjusting the community corresponding to the modularity gain with the largest value among the modularity gains to obtain an adjustment result; processing the adjustment result using a community discovery algorithm to obtain a first community set, and performing knowledge cleaning and redundant information deduplication operations on each text segment in the first community set to obtain a list of text paragraphs to be processed; Specifically, the second paragraph list generation module may further include: using preset key information extraction rules and a large model to perform key information extraction operations on each text segment in the same community to obtain information extraction results; then using preset duplicate expression elimination rules and the large model to perform duplicate expression elimination operations on the information extraction results to obtain cleaned text; the cleaned text includes the core semantic features of the document to be saved; through cross-document graph network construction and community division, knowledge integration is performed on similar text segments from different documents to generate aggregated text with temporal correlation and logical coherence.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the knowledge base management method based on a large model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer programs, wherein the computer programs, when executed by a processor, implement the knowledge base management method based on a large model as described in any one of claims 1 to 5.

Images