Knowledge base management method and device, equipment and storage medium

Abstract

The invention relates to the technical field of natural language processing, and provides a knowledge base management method and device, equipment and a storage medium, and the method comprises the steps that a to-be-processed original document is acquired, and the original document comprises text content; based on a semantic slicing algorithm, splitting the text content of the original document to obtain one or more semantic segments; clustering the semantic segments corresponding to the plurality of original documents to obtain a plurality of knowledge units; clustering the plurality of knowledge units to obtain a plurality of knowledge topics; according to a corresponding relationship among the knowledge theme, the knowledge unit and the semantic fragment, storing the semantic fragment into a knowledge base; and managing the knowledge themes and / or knowledge units in the knowledge base. Taking the field of medical health as an example, a document of nursing knowledge is split into semantic segments, and then the semantic segments are stored in a medical knowledge base according to corresponding relations with knowledge themes and knowledge units. Taking the financial field as an example, a medical insurance document is split into semantic segments, and the semantic segments are stored in an insurance knowledge base.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to knowledge base management methods, apparatus, devices and storage media. Background Technology

[0002] With the rapid development of LLM (Large Language Models), their powerful capabilities in natural language processing and knowledge retrieval have greatly facilitated enterprise knowledge management. Large models can process and understand massive amounts of data in knowledge bases, uncovering potential semantic relationships and thus providing more intelligent knowledge services for enterprise knowledge base management. However, in fields with more complex business data, such as healthcare and finance, the increasing demands and growing complexity of data stored in enterprise knowledge bases (e.g., medical, insurance) pose significant challenges to knowledge base management.

[0003] In scenarios with large and complex business data, the relevant technologies cannot effectively manage the data in the knowledge base, thus failing to meet the enterprise's knowledge management needs. Summary of the Invention

[0004] The main objective of this application is to provide a knowledge base management method, apparatus, device, and storage medium that enables multi-level aggregation of a large number of complex documents, ensuring the structural order of the knowledge base and facilitating its management.

[0005] Firstly, this application provides a knowledge base management method, the method comprising: Obtain the original document to be processed, the original document including text content; Based on the semantic slicing algorithm, the text content of the original document is split to obtain one or more semantic segments; Clustering the semantic segments corresponding to each of the multiple original documents yields multiple knowledge units, and at least one knowledge unit includes multiple semantic segments with the same semantics. Clustering the multiple knowledge units yields multiple knowledge topics, each of which includes at least one knowledge unit. According to the correspondence between the knowledge topic, the knowledge unit, and the semantic fragment, the semantic fragment is stored in the knowledge base; Manage the knowledge topics and / or knowledge units in the knowledge base.

[0006] Secondly, this application also provides a knowledge base management device, the device comprising: The document acquisition module is used to acquire the original document to be processed, wherein the original document includes text content; The splitting module is used to split the text content of the original document based on a semantic slicing algorithm to obtain one or more semantic segments; The first clustering module is used to cluster the semantic segments corresponding to the original documents to obtain multiple knowledge units, and at least one knowledge unit includes multiple semantic segments with the same semantics. The second clustering module is used to cluster the multiple knowledge units to obtain multiple knowledge topics, and each knowledge topic includes at least one knowledge unit. The storage module is used to store the semantic fragments into a knowledge base according to the correspondence between the knowledge topic, the knowledge unit, and the semantic fragment; The management module is used to manage the knowledge topics and / or knowledge units in the knowledge base.

[0007] Thirdly, this application also provides a computer device, which includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and, when executing the computer program, implement the knowledge base management method as described above.

[0008] Fourthly, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the knowledge base management method described above.

[0009] This application provides a knowledge base management method, apparatus, device, and storage medium. The method includes: acquiring an original document to be processed, the original document including text content; splitting the text content of the original document into one or more semantic segments based on a semantic slicing algorithm; clustering the semantic segments corresponding to each of the multiple original documents to obtain multiple knowledge units, at least one of the knowledge units including multiple semantically identical semantic segments; clustering the multiple knowledge units to obtain multiple knowledge topics, each of the knowledge topics including at least one knowledge unit; storing the semantic segments in a knowledge base according to the correspondence between the knowledge topics, the knowledge units, and the semantic segments; and managing the knowledge topics and / or the knowledge units in the knowledge base. This application decomposes the text content of the original document into semantic fragments, then clusters these fragments to obtain multiple knowledge units; further, it clusters these knowledge units to obtain multiple knowledge topics; finally, according to the correspondence between knowledge topics, knowledge units, and semantic fragments, it stores the semantic fragments in a knowledge base. A hierarchical semantic modeling structure of topics and units is constructed in the knowledge base to achieve semantic abstraction and structured aggregation, enabling multi-level aggregation of a large number of complex documents and ensuring the structural orderliness of the knowledge base for easy management. For example, in the healthcare field, nursing knowledge documents are decomposed into semantic fragments and stored in a medical knowledge base according to their correspondence with knowledge topics and knowledge units. Similarly, in the financial field, medical insurance and commercial insurance documents are decomposed into semantic fragments and stored in an insurance knowledge base according to their correspondence with knowledge topics and knowledge units. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a knowledge base management method provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the connection between the server and the terminal device provided in an embodiment of this application; Figure 3 A schematic block diagram illustrating the construction of the knowledge base provided in the embodiments of this application; Figure 4 A schematic block diagram illustrating knowledge base retrieval provided for embodiments of this application; Figure 5 A schematic block diagram of a knowledge base management device provided in an embodiment of this application; Figure 6This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0012] With the rapid development of large language models, knowledge management faces higher requirements for structure, intelligence, and controllability of permissions. Current knowledge retrieval systems include full-text retrieval, question-answering matching systems, and RAG (Retrieval-Augmented Generation) based on slice embedding. However, these knowledge retrieval systems all have the following shortcomings: (1) lack of hierarchical organization of knowledge structure, resulting in weak semantic aggregation ability; (2) reliance on static construction and manual maintenance, resulting in high knowledge update costs; (3) retrieval results lack structured semantic indexes, have no related links, and do not support version tracking or evolution.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0015] This application provides a knowledge base management method, apparatus, device, and storage medium. The knowledge base management method can be applied to terminal devices, such as mobile phones, tablets, laptops, and desktop computers. It can also be applied to servers, which can be standalone servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0016] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] Please see Figure 1 , Figure 1This is a flowchart illustrating a knowledge base management method provided in an embodiment of this application. It should be noted that the knowledge base management method provided in this embodiment can be used on terminal devices, and of course, it can also be used on servers.

[0018] like Figure 2 As shown, the knowledge base management method is applied to the server, and the server and terminal devices communicate with each other. In specific implementation, the terminal devices include, but are not limited to, any of the following: mobile phones, tablets, laptops, and desktop computers; the server can be a single server, a server cluster, or a cloud server that provides cloud computing services.

[0019] like Figure 1 As shown, the knowledge base management method includes steps S101 to S106.

[0020] Step S101: Obtain the original document to be processed, which includes text content.

[0021] Taking the healthcare field as an example, the original documents to be processed may include nursing knowledge manuals, brochures of medical institutions, reports from different departments of the hospital, physical examination documents, and patient documents, etc.

[0022] Taking the financial sector as an example, the original documents to be processed may include financial reports of wealth management products, insurance product contracts, and prospectuses of fund products.

[0023] It is understood that the original document may include formats such as PDF, WORD, and HTML. After obtaining the original document to be processed, this embodiment of the application can preprocess the original document. Specifically, text cleaning is performed on the original document, such as noise removal and format standardization.

[0024] Step S102: Based on the semantic slicing algorithm, the text content of the original document is split to obtain one or more semantic segments.

[0025] After obtaining the original document, this application embodiment can segment the text content of the original document based on a semantic slicing algorithm to obtain one or more semantic segments. For example, semantic slicing algorithms may include the AntSK-FileChunk algorithm, K-means clustering algorithm, TextRank, etc.

[0026] Among them, the AntSK-FileChunk algorithm is an intelligent text slicing algorithm based on semantic understanding, aiming to solve the problems of semantic fragmentation, context loss, and format processing difficulties caused by traditional fixed-length slicing. The AntSK-FileChunk algorithm can uniformly process original documents of different formats, such as PDF, WORD, and HTML. Specifically, it first performs semantic analysis on the original document to capture the semantic information of the text content; then, it determines the slice boundaries by combining the character length limit and semantic similarity threshold; finally, it adjusts the slice boundaries to periods or paragraphs to obtain one or more semantic segments.

[0027] The K-means clustering algorithm calculates the semantic similarity of sentences or paragraphs in the original document, grouping continuous and semantically related text content into the same slice to obtain one or more semantic segments.

[0028] L-TextRank segments text content around specific keywords or topics in the original document to obtain one or more semantic segments, ensuring that each semantic segment focuses on a single topic.

[0029] Step S103: Cluster the semantic segments corresponding to each of the multiple original documents to obtain multiple knowledge units, at least one knowledge unit includes multiple semantic segments with the same semantics.

[0030] It is understood that a knowledge unit may include one or more semantic fragments with the same semantic meaning, and the embodiments of this application can configure a unit tag for each knowledge unit. The unit tag for each knowledge unit may include a summary, a representative statement, the original text fragment corresponding to the semantic fragment, and the contextual relationship, etc. This allows staff to easily understand the differences between different knowledge units through the unit tags.

[0031] After breaking down the text content of the original document into semantic fragments, multiple knowledge units can be obtained by clustering the corresponding semantic fragments from multiple original documents. Specifically, semantically similar fragments are clustered into the same knowledge unit, while semantically different fragments are assigned to different knowledge units. This ensures that multiple semantic fragments within the same knowledge unit have the same semantics, while semantic fragments in different knowledge units have different semantics.

[0032] Taking a medical scenario as an example, the text content of reports from different hospital departments is broken down into multiple semantic fragments. These semantic fragments include introductions of doctors, nurses, and instructions for using departmental facilities. This application's embodiments can cluster semantically similar fragments, such as introductions of different doctors, into knowledge units about doctors, and cluster introductions of different nurses into knowledge units about nurses.

[0033] Step S104: Cluster multiple knowledge units to obtain multiple knowledge topics, each of which includes at least one knowledge unit.

[0034] The knowledge topic in this application embodiment may include one knowledge unit or multiple knowledge units with the same semantic center vector. The semantic center vector of a knowledge unit is determined based on semantic fragments within that knowledge unit. By clustering multiple knowledge units, semantically identical knowledge units can be clustered into the same knowledge topic, while semantically different knowledge units can be clustered into different knowledge topics.

[0035] In the medical field, for example, knowledge units about doctors and nurses can be clustered into a knowledge topic about medical personnel. In the financial field, for example, knowledge units about commercial insurance and life insurance can be clustered into a knowledge topic about insurance; and knowledge units about funds and bonds can be clustered into a knowledge topic about investment and wealth management.

[0036] like Figure 3 As shown, in this embodiment of the application, after splitting the text content of the original document to obtain semantic fragments, the semantic fragments corresponding to each of the multiple original documents are clustered to obtain multiple knowledge units (knowledge unit 1...knowledge unit n), and then the multiple knowledge units (knowledge unit 1...knowledge unit n) are clustered to obtain multiple knowledge topics (knowledge topic 1, knowledge topic 2...knowledge topic m).

[0037] Step S105: Store the semantic fragments in the knowledge base according to the correspondence between knowledge topics, knowledge units, and semantic fragments.

[0038] For example, in this embodiment of the application, multiple knowledge units are obtained by clustering the semantic fragments corresponding to each of multiple original documents, and then multiple knowledge topics are obtained by clustering the multiple knowledge units. The semantic fragments can be stored in the knowledge base according to the correspondence between knowledge topics, knowledge units, and semantic fragments, so as to make the knowledge base structurally ordered and facilitate the management of the knowledge base.

[0039] For example, semantic fragments corresponding to a doctor's profile can be stored in the hospital's knowledge base according to the correspondence between medical staff's knowledge topics, knowledge units about doctors, and doctors' profiles. Similarly, semantic fragments corresponding to a nurse's profile can be stored in the hospital's knowledge base according to the correspondence between medical staff's knowledge topics, knowledge units about nurses, and nurses' profiles.

[0040] Step S106: Manage the knowledge topics and / or knowledge units in the knowledge base.

[0041] It is understood that the embodiments of this application can manage the entire lifecycle of knowledge topics and knowledge units in the knowledge base. Specifically, knowledge units and / or knowledge topics in the knowledge base can be re-clustered periodically; duplicate knowledge units can be merged or long-term inaccessible knowledge units can be deleted periodically based on the access frequency, update time, and clustering status of the knowledge units; and if a knowledge unit in a knowledge topic is transferred or deleted, and the knowledge topic has not been accessed for a long time, the knowledge topic can be marked as invalid and deregistered. This ensures that the knowledge base has an orderly structure, facilitating the retrieval of knowledge content within the knowledge base.

[0042] This application's embodiments introduce a knowledge aggregation and deletion / elimination mechanism to achieve dynamic evolution and optimal retention of knowledge content, thereby enhancing the adaptability of the knowledge base.

[0043] The knowledge base management method provided in the above embodiments includes: acquiring the original document to be processed, the original document including text content; splitting the text content of the original document based on a semantic slicing algorithm to obtain one or more semantic segments; clustering the semantic segments corresponding to each of the multiple original documents to obtain multiple knowledge units, at least one knowledge unit including multiple semantic segments with the same semantics; clustering the multiple knowledge units to obtain multiple knowledge topics, each knowledge topic including at least one knowledge unit; storing the semantic segments in the knowledge base according to the correspondence between knowledge topics, knowledge units, and semantic segments; and managing the knowledge topics and / or knowledge units in the knowledge base. This application splits the text content of the original document into semantic segments, clusters the semantic segments to obtain multiple knowledge units; then clusters the multiple knowledge units to obtain multiple knowledge topics; and then stores the semantic segments in the knowledge base according to the correspondence between knowledge topics, knowledge units, and semantic segments. A hierarchical semantic modeling structure of topics and units is constructed in the knowledge base to achieve semantic abstraction and structured aggregation, thereby realizing multi-level aggregation of a large number of complex documents, ensuring the structural order of the knowledge base, and facilitating the management of the knowledge base. Taking the healthcare field as an example, nursing knowledge documents are broken down into semantic fragments and then stored in a medical knowledge base according to their correspondence with knowledge topics and knowledge units. Similarly, in the financial field, medical insurance and commercial insurance documents are broken down into semantic fragments and then stored in an insurance knowledge base according to their correspondence with knowledge topics and knowledge units.

[0044] In one exemplary embodiment, step S103 includes steps S1031 and S1032.

[0045] Step S1031: Based on the preset semantic embedding model, determine the semantic similarity between different semantic segments in multiple semantic segments.

[0046] Step S1032: Based on the semantic similarity between different semantic segments in multiple semantic segments and the preset first similarity threshold, cluster the multiple semantic segments to obtain multiple knowledge units. The semantic similarity between different semantic segments in each knowledge unit is greater than or equal to the first similarity threshold.

[0047] For example, semantic embedding models can include BGE models and SimCSE models. BGE models can map semantic segments into low-dimensional, dense semantic embedding vectors. SimCSE models are unsupervised or supervised sentence vector representation methods based on contrastive learning. They train sentence embedding models through contrastive learning to map semantic segments into semantic embedding vectors.

[0048] In this embodiment, after mapping semantic segments to semantic embedding vectors, the semantic similarity between different semantic segments within a plurality of semantic segments is calculated. Specifically, the calculation process for the semantic similarity between different semantic segments is as follows: ; in, semantic fragments i With semantic fragments j Semantic similarity between them V i semantic fragments i The semantic embedding vector of the mapping, V j semantic fragments j The semantic embedding vector of the mapping, semantic embedding vector V i The length of the mold, semantic embedding vector V j The length of the module.

[0049] After calculating the semantic similarity between different semantic segments in multiple semantic segments, the semantic similarity between different semantic segments in multiple semantic segments can be compared with a preset first similarity threshold. When the semantic similarity between two semantic segments is greater than or equal to the first similarity threshold, the two semantic segments are clustered into the same knowledge unit, so as to achieve clustering of multiple semantic segments and obtain multiple knowledge units.

[0050] It is understandable that the semantic similarity between different semantic fragments in each knowledge unit obtained by clustering is greater than or equal to the first similarity threshold. Furthermore, embodiments of this application can configure unit labels for each knowledge unit obtained by clustering. The unit labels for each knowledge unit can include a summary, representative statement, the original text fragment corresponding to the semantic fragment, contextual relationships, etc. This allows staff to easily understand the differences between different knowledge units through the unit labels.

[0051] In one exemplary embodiment, step S104 includes steps S1041 and S1042.

[0052] Step S1041: Determine the unit semantic center vector corresponding to the knowledge unit based on the semantic embedding vector of the semantic fragment corresponding to the knowledge unit.

[0053] Step S1042: Based on a preset clustering algorithm, multiple knowledge units are clustered according to their respective semantic center vectors and a preset second similarity threshold to obtain multiple knowledge topics. The similarity of the semantic center vectors of different knowledge units in each knowledge topic is greater than or equal to the second similarity threshold.

[0054] It is understood that, in this application embodiment, the unit semantic center vector corresponding to the knowledge unit can be calculated based on the formula for calculating the unit semantic center vector, according to the semantic embedding vector of the semantic segment corresponding to the knowledge unit. For example, the formula for calculating the unit semantic center vector is as follows: ; in, V C This is the semantic center vector of the knowledge unit C; w i semantic fragments i The importance weight is determined based on the unit label of knowledge unit C; V i semantic fragments i The semantic embedding vector.

[0055] For example, the preset clustering algorithms may include hierarchical clustering algorithms and density-based clustering algorithms.

[0056] After calculating the semantic center vector corresponding to each knowledge unit, this embodiment of the application can use a hierarchical clustering algorithm or a density-based clustering algorithm to cluster the knowledge units corresponding to multiple semantic center vectors with a similarity greater than or equal to a second similarity threshold, thereby obtaining multiple knowledge topics. It is understood that the similarity of the semantic center vectors between knowledge units of different knowledge topics is less than the second similarity threshold; the similarity of the semantic center vectors between different knowledge units within each knowledge topic is greater than or equal to the second similarity threshold.

[0057] In one exemplary embodiment, step S106 includes steps S1061A to S1063A.

[0058] Step S1061A: Determine the topic semantic center vector corresponding to the knowledge topic based on the unit semantic center vector of the knowledge unit corresponding to the knowledge topic in the knowledge base.

[0059] Step S1062A: Determine the rate of change of the topic semantic center vector corresponding to the knowledge topic based on the topic semantic center vector of the knowledge topic and the historical topic semantic center vector of the knowledge topic.

[0060] Step S1063A involves re-clustering multiple knowledge topics when the rate of change of the topic semantic center vector corresponding to at least some knowledge topics is greater than or equal to a preset change threshold.

[0061] This application embodiment can determine the topic semantic center vector corresponding to a knowledge topic based on the unit semantic center vector of the knowledge unit corresponding to the knowledge topic in the knowledge base. For example, for a knowledge topic that includes only one knowledge unit, the unit semantic center vector corresponding to that knowledge unit can be used as the topic semantic center vector corresponding to the knowledge topic; for a knowledge topic that includes multiple knowledge units, the average value of the unit semantic center vectors of the multiple knowledge units in the knowledge topic can be calculated, and the average value of the unit semantic center vectors of the multiple knowledge units can be used as the topic semantic center vector corresponding to the knowledge topic.

[0062] It is understood that the embodiments of this application can calculate the rate of change of the topic semantic center vector corresponding to each knowledge topic based on the rate of change calculation formula. Specifically, the rate of change of the topic semantic center vector corresponding to the knowledge topic can be calculated based on the rate of change calculation formula, according to the topic semantic center vector of the knowledge topic and the historical topic semantic center vector of the knowledge topic.

[0063] The formula for calculating the rate of change in this application embodiment is as follows: ; in, The rate of change; The semantic center vector of the historical theme of the knowledge topic; The magnitude of the semantic center vector of the historical theme; The topic semantic center vector of the knowledge topic; The magnitude of the semantic center vector of the topic.

[0064] The embodiments of this application can monitor the rate of change of the topic semantic center vector corresponding to each knowledge topic in real time. When the rate of change of the topic semantic center vector corresponding to at least some, such as one-third, knowledge topics is greater than or equal to a preset change threshold, it can be determined that the knowledge topic has undergone semantic shift. At this time, multiple knowledge topics can be re-clustered.

[0065] For example, re-clustering multiple knowledge topics can refer to re-clustering them based on the similarity of the semantic center vectors of the knowledge units corresponding to the knowledge units in the multiple knowledge topics, and a second similarity threshold; alternatively, multiple knowledge topics whose semantic center vector similarity is greater than or equal to the threshold can be directly merged; or knowledge topics that have undergone semantic shift can be directly re-clustered to split the knowledge topic into multiple knowledge topics. This application embodiment can dynamically adjust the structure of knowledge topics in the knowledge base by monitoring the rate of change of the semantic center vectors corresponding to the knowledge topics, so that the content of knowledge topics and knowledge units in the knowledge base can be accurately classified.

[0066] In one exemplary embodiment, step S106 includes steps S1061B and S1062B.

[0067] Step S1061B: Detect the current load status of the knowledge base.

[0068] Step S1062B: When it is detected that the current load state of the knowledge base meets the preset load state, multiple knowledge units are re-clustered, and / or multiple knowledge topics are re-clustered.

[0069] For example, embodiments of this application can detect the current load status of the knowledge base in real time. When the current load status of the knowledge base meets the preset load status, such as when the CPU utilization of the knowledge base is lower than the threshold or when there is idle in the task queue of the knowledge base, multiple knowledge units are re-clustered and / or multiple knowledge topics are re-clustered.

[0070] It is understandable that re-clustering multiple knowledge units means that multiple new knowledge units can be obtained by re-clustering based on the semantic similarity and the first similarity threshold between different semantic segments in multiple semantic segments.

[0071] Re-clustering multiple knowledge topics refers to obtaining multiple new knowledge topics by re-clustering them based on the similarity between the semantic center vectors corresponding to the individual knowledge units and a second similarity threshold. Specifically, the similarity between the semantic center vectors corresponding to the individual knowledge units is calculated, and when the similarity between the semantic center vectors corresponding to the individual knowledge units is greater than or equal to the second similarity threshold, the multiple new knowledge units are re-clustered to obtain multiple new knowledge topics. Alternatively, when the semantic center vector of a new knowledge unit deviates from the topic semantic center vector of its corresponding knowledge topic, the new knowledge unit is clustered into a knowledge topic with a similar topic semantic center vector.

[0072] In this embodiment of the application, when the current load state of the knowledge base meets the preset load state, the knowledge units and / or knowledge topics are re-clustered to ensure the structural order of the knowledge base and enhance the flexibility of the knowledge base in storing data.

[0073] In one exemplary embodiment, the method further includes steps S201 and S202.

[0074] Step S201: Obtain the current text content input to the knowledge base.

[0075] Step S202: Based on the matching degree between the current text content and the semantic fragments corresponding to each of the multiple knowledge units, determine the target knowledge unit from the multiple knowledge units in the knowledge base and add the current text content to the target knowledge unit; or create a new knowledge unit in the knowledge base based on the current text content.

[0076] Understandably, upon receiving current text content input into the knowledge base, the semantic similarity between this current text content and the semantic segments corresponding to each of multiple knowledge units can be calculated to determine the matching degree between the current text content and the semantic segments corresponding to each of the multiple knowledge units. If the matching degree between the current text content and the semantic segments corresponding to each of the multiple knowledge units is greater than or equal to a preset matching degree, the knowledge unit with the highest matching degree among the knowledge units with matching degrees greater than or equal to the preset matching degree can be determined as the target knowledge unit, and the current text content can be added to the target knowledge unit, while simultaneously storing and updating the version number of the target knowledge unit. If the matching degree between the current text content and the semantic segments corresponding to each of the multiple knowledge units is less than the preset matching degree, it indicates that the current text content is new knowledge content. In this case, a new knowledge unit can be created in the knowledge base based on the current text content. This ensures the structural integrity of the knowledge base and expands the coverage of knowledge content within the knowledge base.

[0077] In one exemplary embodiment, the method further includes steps S301 to S305.

[0078] Step S301: Obtain the search instruction.

[0079] Step S302: Based on the search instruction, determine the target knowledge topic corresponding to the search instruction from multiple knowledge topics in the knowledge base.

[0080] Step S303: Determine the knowledge units in the target knowledge topic whose relevance to the search command is greater than or equal to the preset relevance as candidate knowledge units corresponding to the search command.

[0081] Step S304: Select semantic segments that meet the preset conditions from the candidate knowledge units, and use the semantic segments that meet the preset conditions as the retrieval results corresponding to the candidate knowledge units.

[0082] Step S305: Based on the matching degree between the retrieval results and retrieval instructions corresponding to the candidate knowledge units, determine the target knowledge unit from the candidate knowledge units, and take the semantic fragments in the target knowledge unit that meet the preset conditions as the target retrieval results.

[0083] like Figure 4 As shown in the embodiment of this application, after obtaining a search instruction, a search can first be performed on multiple knowledge topics in the knowledge base to retrieve the target knowledge topic corresponding to the search instruction. This initial coarse-grained search quickly narrows the search scope. Then, knowledge units within the target knowledge topic whose relevance to the search instruction is greater than or equal to a preset relevance level are identified as candidate knowledge units corresponding to the search instruction. This two-stage search process, "knowledge topic - knowledge unit," balances recall speed and semantic accuracy, effectively reducing redundancy and interference, thereby improving search accuracy.

[0084] For example, semantic fragments that meet preset conditions can be selected from candidate knowledge units, and these semantic fragments can be used as the search results corresponding to the candidate knowledge units. Preset conditions may include the position and scope of the semantic fragment in the original document. Specifically, related semantic fragments are first extracted from the candidate knowledge units. The start and end positions and context of the semantic fragment in the original document are determined through the index relationship of the semantic fragment in the knowledge base (i.e., semantic fragment-original document). Then, the related text content is obtained by expanding the scope (e.g., the three semantic fragments before and after the original document) to obtain the search results corresponding to the candidate knowledge units.

[0085] Finally, based on the formula for calculating the comprehensive ranking score, the comprehensive ranking score between the retrieval results and retrieval instructions corresponding to the candidate knowledge units is calculated to obtain the matching degree between the retrieval results and retrieval instructions corresponding to the candidate knowledge units, and the target knowledge unit is determined from the candidate knowledge units. The formula for calculating the comprehensive ranking score in this embodiment of the application is as follows: ; in, Candidate knowledge units i The overall ranking score; Semantic matching scores can be derived from search instructions and candidate knowledge units. i The cosine similarity is calculated. To score version timeliness, you can base your score on candidate knowledge units. i The update time and the frequency of access are calculated; The context relevance score can be calculated based on the co-occurrence frequency of the search command in the context and the weight of the citation relationship; , , All are weights, and The values ​​of the three weights can be updated based on the learning algorithm.

[0086] It is understood that, in this embodiment of the application, the candidate knowledge unit with the highest comprehensive ranking score is determined as the target knowledge unit from the comprehensive ranking scores corresponding to each of the multiple candidate knowledge units, and the semantic fragments in the target knowledge unit that meet the preset conditions are used as the target retrieval results, so as to improve the accuracy of knowledge base retrieval.

[0087] For example, when outputting the target search results, prompts can be generated based on the unit tags of candidate knowledge units, such as abstracts, representative statements, and contextual relationships. These prompts can include source information such as the ID number, version number, and original text citation location of the target knowledge unit. This improves the interpretability and verifiability of the knowledge base retrieval, thereby significantly reducing the risk of the target search results deviating from the facts.

[0088] In one exemplary implementation, the permission settings of the original document (such as public to the organization, shared by the team, privately visible, etc.) can be automatically inherited during the knowledge unit generation stage. During the retrieval and recall process, the identity, role, and access permissions of the current user can be compared in real time. Permission verification and retrieval and recall are performed simultaneously to ensure that content without permission does not appear in the candidate knowledge units. This can meet the needs of multi-role and multi-level permission management under complex organizational structures, prevent the leakage of sensitive information or unauthorized use, and thus meet the needs of users with multiple roles to access the knowledge base.

[0089] Please see Figure 5 , Figure 5 This is a schematic block diagram of a knowledge base management device provided in an embodiment of this application. The knowledge base management device can be configured in a server or terminal device to execute the aforementioned knowledge base management method.

[0090] like Figure 5As shown, the knowledge base management device includes: a document acquisition module 110, a splitting module 120, a first clustering module 130, a second clustering module 140, a storage module 150, and a management module 160.

[0091] Document acquisition module 110 is used to acquire the original document to be processed, which includes text content; The splitting module 120 is used to split the text content of the original document based on the semantic slicing algorithm to obtain one or more semantic segments; The first clustering module 130 is used to cluster the semantic fragments corresponding to multiple original documents to obtain multiple knowledge units, and at least one knowledge unit includes multiple semantic fragments with the same semantics. The second clustering module 140 is used to cluster multiple knowledge units to obtain multiple knowledge topics, each knowledge topic including at least one knowledge unit. Storage module 150 is used to store semantic fragments into a knowledge base according to the correspondence between knowledge topics, knowledge units, and semantic fragments; Management module 160 is used to manage knowledge topics and / or knowledge units in the knowledge base.

[0092] In one exemplary embodiment, the first clustering module 130 includes a first similarity determination submodule and a first clustering submodule.

[0093] The first similarity determination submodule is used to determine the semantic similarity between different semantic segments in multiple semantic segments based on a preset semantic embedding model.

[0094] The first clustering submodule is used to cluster multiple semantic segments based on the semantic similarity between different semantic segments in multiple semantic segments and a preset first similarity threshold to obtain multiple knowledge units. The semantic similarity between different semantic segments in each knowledge unit is greater than or equal to the first similarity threshold.

[0095] In one exemplary embodiment, the second clustering module 140 includes a first vector determination submodule and a second clustering submodule.

[0096] The first vector determination submodule is used to determine the unit semantic center vector corresponding to the knowledge unit based on the semantic embedding vector of the semantic fragment corresponding to the knowledge unit.

[0097] The second clustering submodule is used to cluster multiple knowledge units based on a preset clustering algorithm, according to the semantic center vector of each knowledge unit and a preset second similarity threshold, to obtain multiple knowledge topics. The similarity of the semantic center vectors of different knowledge units in each knowledge topic is greater than or equal to the second similarity threshold.

[0098] In one exemplary embodiment, the management module 160 includes a second vector determination submodule, a rate of change determination submodule, and a first re-clustering submodule.

[0099] The second vector determination submodule is used to determine the topic semantic center vector corresponding to the knowledge topic based on the unit semantic center vector of the knowledge unit corresponding to the knowledge topic in the knowledge base.

[0100] The rate of change determination submodule is used to determine the rate of change of the topic semantic center vector corresponding to the knowledge topic based on the topic semantic center vector of the knowledge topic and the historical topic semantic center vector of the knowledge topic.

[0101] The first re-clustering submodule is used to re-cluster multiple knowledge topics when the rate of change of the topic semantic center vector corresponding to at least some knowledge topics is greater than or equal to a preset change threshold.

[0102] In one exemplary embodiment, the management module 160 includes a detection submodule and a second re-clustering submodule.

[0103] The detection submodule is used to detect the current load status of the knowledge base.

[0104] The second re-clustering submodule is used to re-cluster multiple knowledge units and / or re-cluster multiple knowledge topics when the current load state of the knowledge base is detected to meet the preset load state.

[0105] In one exemplary embodiment, the apparatus further includes a text acquisition module and an update module.

[0106] The text acquisition module is used to acquire the current text content input into the knowledge base.

[0107] The update module is used to determine the target knowledge unit from multiple knowledge units in the knowledge base based on the matching degree between the current text content and the semantic fragments corresponding to each of the multiple knowledge units, and add the current text content to the target knowledge unit; or to create a new knowledge unit in the knowledge base based on the current text content.

[0108] In one exemplary embodiment, the device further includes an instruction acquisition module, a first retrieval module, a second retrieval module, a filtering module, and a result determination module.

[0109] The instruction acquisition module is used to acquire search instructions.

[0110] The first retrieval module is used to determine the target knowledge topic corresponding to the retrieval instruction from multiple knowledge topics in the knowledge base, based on the retrieval instruction.

[0111] The second retrieval module is used to identify knowledge units in the target knowledge topic whose relevance to the retrieval command is greater than or equal to a preset relevance level as candidate knowledge units corresponding to the retrieval command.

[0112] The filtering module is used to select semantic fragments that meet preset conditions from candidate knowledge units, and to use the semantic fragments that meet the preset conditions as the search results corresponding to the candidate knowledge units.

[0113] The result determination module is used to determine the target knowledge unit from the candidate knowledge units based on the matching degree between the search results and search instructions corresponding to the candidate knowledge units, and to take the semantic fragments in the target knowledge unit that meet the preset conditions as the target search results.

[0114] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0116] For example, the above-described methods and apparatus can be implemented as a computer program that can run on a computer device.

[0117] Please see Figure 6 , Figure 6 This is a schematic block diagram illustrating the structure of a computer device provided in an embodiment of this application. The computer device may be a server or a terminal device.

[0118] like Figure 6 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and internal memory.

[0119] The storage medium may store the operating system and computer programs. The computer programs include program instructions that, when executed, cause the processor to perform steps of any knowledge base management method.

[0120] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0121] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When these computer programs are executed by the processor, the processor can perform the steps of any knowledge base management method.

[0122] This network interface is used for network communication, such as sending assigned tasks.

[0123] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0125] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, may perform the following steps: Obtain the original document to be processed, which includes text content; Based on the semantic slicing algorithm, the text content of the original document is split into one or more semantic segments; Clustering the semantic fragments corresponding to each of the multiple original documents yields multiple knowledge units, with at least one knowledge unit comprising multiple semantically identical semantic fragments. Multiple knowledge units are clustered to obtain multiple knowledge topics, and each knowledge topic includes at least one knowledge unit. Based on the correspondence between knowledge topics, knowledge units, and semantic fragments, semantic fragments are stored in the knowledge base; Manage knowledge topics and / or knowledge units in the knowledge base.

[0126] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of knowledge base management described above can be referred to the corresponding process in the aforementioned embodiments of the knowledge base management method, and will not be repeated here.

[0127] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps: Obtain the original document to be processed, which includes text content; Based on the semantic slicing algorithm, the text content of the original document is split into one or more semantic segments; Clustering the semantic fragments corresponding to each of the multiple original documents yields multiple knowledge units, with at least one knowledge unit comprising multiple semantically identical semantic fragments. Multiple knowledge units are clustered to obtain multiple knowledge topics, and each knowledge topic includes at least one knowledge unit. Based on the correspondence between knowledge topics, knowledge units, and semantic fragments, semantic fragments are stored in the knowledge base; Manage knowledge topics and / or knowledge units in the knowledge base.

[0128] The computer-readable storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as a hard disk or memory of the computer device. Alternatively, it can be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the computer device.

[0129] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium described above can be referred to the embodiments of the aforementioned knowledge base management method.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A knowledge base management method, characterized in that, The method includes: Obtain the original document to be processed, the original document including text content; Based on the semantic slicing algorithm, the text content of the original document is split to obtain one or more semantic segments; Clustering the semantic segments corresponding to each of the multiple original documents yields multiple knowledge units, and at least one knowledge unit includes multiple semantic segments with the same semantics. Clustering the multiple knowledge units yields multiple knowledge topics, each of which includes at least one knowledge unit. According to the correspondence between the knowledge topic, the knowledge unit, and the semantic fragment, the semantic fragment is stored in the knowledge base; Manage the knowledge topics and / or knowledge units in the knowledge base.

2. The knowledge base management method according to claim 1, characterized in that, The process of clustering semantic fragments corresponding to each of the multiple original documents yields multiple knowledge units, including: Based on a preset semantic embedding model, the semantic similarity between different semantic segments in the plurality of semantic segments is determined; Based on the semantic similarity between different semantic segments in the plurality of semantic segments and a preset first similarity threshold, the plurality of semantic segments are clustered to obtain a plurality of knowledge units, wherein the semantic similarity between different semantic segments in each knowledge unit is greater than or equal to the first similarity threshold.

3. The knowledge base management method according to claim 1, characterized in that, The clustering of the multiple knowledge units yields multiple knowledge topics, including: Based on the semantic embedding vector of the semantic fragment corresponding to the knowledge unit, determine the unit semantic center vector corresponding to the knowledge unit; Based on a preset clustering algorithm, the multiple knowledge units are clustered according to their respective semantic center vectors and a preset second similarity threshold to obtain multiple knowledge topics. The similarity of the semantic center vectors of different knowledge units in each knowledge topic is greater than or equal to the second similarity threshold.

4. The knowledge base management method according to claim 1, characterized in that, The management of the knowledge topics in the knowledge base includes: Based on the unit semantic center vector of the knowledge unit corresponding to the knowledge topic in the knowledge base, determine the topic semantic center vector corresponding to the knowledge topic; Based on the topic semantic center vector of the knowledge topic and the historical topic semantic center vector of the knowledge topic, determine the rate of change of the topic semantic center vector corresponding to the knowledge topic; When the rate of change of the topic semantic center vector corresponding to at least some of the knowledge topics is greater than or equal to a preset change threshold, the multiple knowledge topics are re-clustered.

5. The knowledge base management method according to any one of claims 1 to 4, characterized in that, The management of the knowledge topics and / or knowledge units in the knowledge base includes: Detect the current load status of the knowledge base; When the current load state of the knowledge base is detected to meet the preset load state, the multiple knowledge units are re-clustered, and / or the multiple knowledge topics are re-clustered.

6. The knowledge base management method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the current text content input into the knowledge base; Based on the matching degree between the current text content and the semantic segments corresponding to each of the multiple knowledge units, a target knowledge unit is determined from the multiple knowledge units in the knowledge base and the current text content is added to the target knowledge unit; or a new knowledge unit is created in the knowledge base based on the current text content.

7. The knowledge base management method according to any one of claims 1 to 4, characterized in that, The method further includes: Get search instructions; Based on the search instruction, determine the target knowledge topic corresponding to the search instruction from the plurality of knowledge topics in the knowledge base; Knowledge units in the target knowledge topic whose relevance to the search command is greater than or equal to a preset relevance level are identified as candidate knowledge units corresponding to the search command; Semantic fragments that meet preset conditions are selected from the candidate knowledge units, and the semantic fragments that meet the preset conditions are used as the retrieval results corresponding to the candidate knowledge units; Based on the matching degree between the retrieval results corresponding to the candidate knowledge units and the retrieval instructions, the target knowledge unit is determined from the candidate knowledge units, and the semantic fragments in the target knowledge unit that meet the preset conditions are taken as the target retrieval results.

8. A knowledge base management device, characterized in that, The device includes: The document acquisition module is used to acquire the original document to be processed, wherein the original document includes text content; The splitting module is used to split the text content of the original document based on a semantic slicing algorithm to obtain one or more semantic segments; The first clustering module is used to cluster the semantic segments corresponding to the original documents to obtain multiple knowledge units, and at least one knowledge unit includes multiple semantic segments with the same semantics. The second clustering module is used to cluster the multiple knowledge units to obtain multiple knowledge topics, and each knowledge topic includes at least one knowledge unit. The storage module is used to store the semantic fragments into a knowledge base according to the correspondence between the knowledge topic, the knowledge unit, and the semantic fragment; The management module is used to manage the knowledge topics and / or knowledge units in the knowledge base.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the knowledge base management method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the knowledge base management method as described in any one of claims 1 to 7.

Images