A knowledge base processing method and electronic device

Abstract

The application discloses a knowledge base processing method and an electronic device, and relates to the technical field of artificial intelligence and natural language processing. The knowledge base processing method comprises the following steps: obtaining a target processing regulation currently used for guiding knowledge base construction; performing knowledge distillation on an original data file by using a large model based on the target processing regulation, so that a first knowledge page capable of being reused is obtained; storing the first knowledge page in a knowledge base and updating a global index used for knowledge page retrieval; determining user preference information in task processing application based on the knowledge base; and updating the target processing regulation based on the user preference information, so that knowledge distillation and storage processing are performed based on the updated processing regulation.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a knowledge base processing method and electronic device. Background Technology

[0002] Currently, Large Language Models (LLMs) have demonstrated outstanding capabilities in fields such as natural language processing and question answering, generating high-quality responses to user queries based on knowledge bases. With the rapid development of LLMs and agent technologies, "knowledge self-evolution" has gradually become a research hotspot. Therefore, how to design a self-evolving knowledge base processing scheme that enables the knowledge base to be continuously maintained and self-iterated, thereby effectively improving the overall performance of large language models, has become a key technical problem urgently needing to be solved in this field. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A knowledge base processing method, comprising:

[0005] Obtain the target processing specification currently used to guide the construction of the knowledge base;

[0006] Based on the target processing specification, knowledge distillation is performed on the original data file through a large model to obtain a reusable first knowledge page.

[0007] Store the first knowledge page in the knowledge base and update the global index used for knowledge page retrieval;

[0008] Determine user preference information for task processing applications based on the knowledge base;

[0009] The target processing specification is updated based on the user preference information, and knowledge distillation and storage are performed based on the updated processing specification.

[0010] Optionally, determining user preference information in task processing applications based on the knowledge base includes:

[0011] Obtain the interaction information corresponding to each knowledge page in the knowledge base, wherein the interaction information represents the interactive behavior features of the knowledge page that can be used to measure user preferences;

[0012] User preference information is determined based on the interaction information.

[0013] Optionally, the interactive behavior features include at least one of user feedback behavior features and system call behavior features;

[0014] The interactive information includes at least one of the following: explicit annotation information of the user on the knowledge page, link information of the system for the knowledge page, and reference information.

[0015] Optionally, updating the target processing specification based on the user preference information includes:

[0016] At least one candidate processing specification is determined, which is generated by adjusting the target processing specification through a large model;

[0017] Based on the aforementioned candidate processing specification, data file samples are distilled into knowledge page samples using a large model;

[0018] Determine the preference alignment between the knowledge page sample and user preferences;

[0019] In response to the preference alignment corresponding to the corresponding candidate processing specification satisfying the target condition, the target processing specification is updated based on the corresponding candidate processing specification.

[0020] Optionally, the method further includes:

[0021] Obtain the target task to be processed;

[0022] Determine index information that matches the target task, and locate a second knowledge page in the knowledge base that meets the association conditions with the target task based on the determined index information;

[0023] Based on the target task and the second knowledge page, task reasoning is performed through a large model to obtain the task processing result corresponding to the target task.

[0024] Optionally, the method further includes at least one of the following:

[0025] Update the interaction information corresponding to the second knowledge page, wherein the interaction information represents the interactive behavior characteristics of the corresponding knowledge page that can be used to measure user preferences;

[0026] In response to the task processing result satisfying the first quality condition, the knowledge page in the knowledge base is updated based on the task processing result.

[0027] Optionally, the method further includes:

[0028] In response to the fulfillment of inspection conditions, determine the knowledge pages in the knowledge base that have been updated;

[0029] Based on the updated knowledge page, a local subgraph to be inspected is determined in the knowledge graph represented by the knowledge base, and the local subgraph includes the updated knowledge page;

[0030] Determine the preference alignment between each third knowledge page in the local subgraph and the user preferences;

[0031] Based on the interaction information and preference alignment corresponding to each of the third knowledge pages, the quality of each of the third knowledge pages is evaluated; the interaction information represents the interactive behavior features that can be used to measure user preferences for the corresponding knowledge page.

[0032] Based on the quality assessment results corresponding to each of the aforementioned third knowledge pages, a matching quality optimization process is performed on each of the aforementioned third knowledge pages.

[0033] Optionally, based on the quality assessment results corresponding to each of the third knowledge pages, a matching quality optimization process is performed on each of the third knowledge pages, including:

[0034] In response to the corresponding quality assessment result indicating that the third knowledge page does not meet the second quality condition, at least one of the following is performed on the third knowledge page: downgrade, archive, or complete interactive information;

[0035] In response to the corresponding quality assessment result indicating that the third knowledge page meets the second quality condition, contradiction detection and knowledge accuracy improvement processing are performed on the third knowledge page.

[0036] Optionally, the method further includes:

[0037] The user preference information is updated based on the interaction information corresponding to each third knowledge page in the local subgraph.

[0038] An electronic device, comprising:

[0039] Memory, used to store at least one set of computer instructions;

[0040] A processor is configured to perform the following processing by executing a set of instructions stored in the memory:

[0041] Obtain the target processing specification currently used to guide the construction of the knowledge base;

[0042] Based on the target processing specification, knowledge distillation is performed on the original data file through a large model to obtain a reusable first knowledge page.

[0043] Store the first knowledge page in the knowledge base and update the global index used for knowledge page retrieval;

[0044] Determine user preference information for task processing applications based on the knowledge base;

[0045] The target processing specification is updated based on the user preference information, and knowledge distillation and storage are performed based on the updated processing specification.

[0046] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement any of the knowledge base processing methods provided above. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 This is a flowchart of the knowledge base processing method provided in this application;

[0049] Figure 2 This is a schematic diagram of the self-iterative process of target processing specification based on user preference awareness provided in this application;

[0050] Figure 3 This is a flowchart illustrating the process of achieving self-iteration of the target processing specification by generating candidate processing specifications, as provided in this application.

[0051] Figure 4 This is a process diagram provided in this application for optimizing the quality of the knowledge base through dynamic inspection and multi-dimensional quality assessment;

[0052] Figure 5 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] This application provides a knowledge base processing method and an electronic device for constructing and maintaining a preference-aware self-evolving knowledge base. The method can be applied to electronic devices in various general-purpose or special-purpose computing environments or configurations, such as personal computers, server computers, smart terminals, handheld or portable devices, tablet devices, etc.

[0055] In a macroscopic sense, the knowledge base in this application embodiment is a self-evolving knowledge base system that includes the following three-layer architecture.

[0056] I. Original Data Layer

[0057] The raw data layer provides a raw data storage area for storing unprocessed data source documents, whether manually curated or obtained from external sources. These documents serve only as content sources and do not contain any instructions to prevent tooltip injection attacks.

[0058] II. Knowledge Page Layer:

[0059] This layer is the core content layer of the knowledge base, and may include a series of knowledge pages, such as structured Markdown pages, which are distilled from the content of the original data layer by large models such as large language models according to preset processing / operational specifications. These pages may include, but are not limited to, entity pages, concept pages, and / or comprehensive pages.

[0060] In addition, in this embodiment, the knowledge page layer may also be associated with a global index for knowledge page retrieval, log files for related log recording and querying, and may also maintain interactive information such as explicit annotations of knowledge pages by users, cross-references between pages, and bidirectional links.

[0061] III. Operational Specification Layer

[0062] This layer stores and maintains global processing specifications / operation specifications that guide the operation of the entire system. Global processing specifications / operation specifications can be rule files that can be parsed and followed by large models such as large language models, such as skill files like AGENTS.md, also known as Skills. They can include, but are not limited to, related prompts, scripts / instructions, and other processing specification information. They define the specific logical patterns or behaviors that the LLM should follow in the Ingest, Query, and Lint phases, such as how to summarize content, which fields to retain, and how to establish links.

[0063] In a narrow sense, the knowledge base in this application mainly refers to the collection of knowledge pages in the knowledge page layer of the aforementioned three-layer architecture. This is a structured collection of knowledge pages generated by large models such as the large language model, which parses, extracts, and reorganizes source documents in the raw data layer according to the target processing specifications in the operation specification layer used to guide knowledge base construction. For example, it is a collection of knowledge pages in Markdown format. This knowledge base is user-, system-, or upper-layer application-oriented, providing knowledge services (such as retrieval and matching). Furthermore, the knowledge pages possess the characteristics of "compile once, maintain continuously, and reuse multiple times." Specifically, the knowledge pages are pre-distilled and stored during the data ingestion or linting stage, rather than being resynthesized for each query. This allows the system to directly call the pre-built knowledge pages to respond to retrieval requests, avoiding repeated calls to the large model for on-the-spot generation, thereby significantly improving the response speed and stability of the knowledge service.

[0064] Here, compilation refers to using the capabilities of large models to "distill" and process the messy, fragmented data in the original materials into structured knowledge pages, so that the knowledge can be reused efficiently and the computational power consumed by reprocessing the original data to obtain knowledge for each query is avoided.

[0065] The solution in this application embodiment is mainly used for the above-mentioned architecture of the knowledge base. By introducing user preference awareness and a user preference-based self-iteration mechanism, the construction, maintenance and processing specifications of the knowledge base can be dynamically adapted to user needs, so as to realize the intelligent and dynamic "self-evolution" of the knowledge base.

[0066] See Figure 1 The flowchart shown illustrates the knowledge base processing method provided in this application embodiment, which may include the following steps 101 to 105, which are described in detail below.

[0067] Step 101: Obtain the target processing specification currently used to guide the construction of the knowledge base.

[0068] Among them, the target processing specification is a part of the processing specification included in the current global processing specification of the operation specification layer, which is used to guide the construction of the knowledge base. It may include, but is not limited to, information such as prompt words, scripts / instructions, etc., and can be parsed and followed by large models such as large language models.

[0069] Step 102: Based on the target processing specification, perform knowledge distillation on the original data file using a large model to obtain a reusable first knowledge page.

[0070] Raw data files may be, but are not limited to, unprocessed data source documents that are manually curated or obtained from external sources.

[0071] Unprocessed data source documents, whether manually curated or obtained from external sources, can first be stored in the raw data layer. These source documents serve only as content sources and do not contain any instructions to prevent prompt injection attacks.

[0072] In practical applications, the raw, undistilled data files from the original data layer can be obtained through an LLM Agent, and these raw data files, along with the target processing specification, are input into the large model. Under the constraints and guidance of the target processing specification, the large model performs knowledge distillation on the raw data files, thereby generating a structured and reusable first knowledge page.

[0073] Large models can be, but are not limited to, large language models, visual language models, or multimodal large models. They possess the ability to build and maintain knowledge bases based on knowledge distillation, as well as reasoning capabilities for user tasks. In the knowledge base construction and maintenance phase, the large model primarily acts as a "knowledge compiler," performing deep semantic extraction, logical reorganization, and structured mapping on unstructured, multimodal data in the original data layer. This distills the data into reusable and easily searchable knowledge pages, and supports continuous optimization of page quality through a review mechanism.

[0074] Step 103: Store the first knowledge page in the knowledge base and update the global index used for knowledge page retrieval.

[0075] In this step, storing the first knowledge page into the knowledge base can refer to updating the knowledge page set in the knowledge page layer of the above three-layer architecture based on the first knowledge page. For example, it can be done by directly adding the first knowledge page to the knowledge page set, or by modifying the corresponding knowledge pages in the knowledge page layer that have a relationship with the first knowledge page.

[0076] The relationships mentioned here may include, but are not limited to, at least one of the following: overlapping content, related content, semantic similarity reaching a threshold, containing the same core entities, concepts, or belonging to the same knowledge topic cluster.

[0077] Simultaneously, based on the data entry processing of the first knowledge page, the global index corresponding to the knowledge page layer can be updated. The update operation for the global index may include, but is not limited to:

[0078] 1) Generate corresponding index information for newly added knowledge pages (such as the first knowledge page);

[0079] 2) Add new keywords, tags, summaries, or semantic vectors to the corresponding index entries on existing knowledge pages to reflect the updated page content;

[0080] 3) For knowledge pages that have been modified or merged, recalculate and update their corresponding vector representations and metadata to ensure that the global index and the actual content in the knowledge page layer are kept in real time synchronized.

[0081] 4) Based on the relationship between the first knowledge page and existing knowledge pages, update the cross-reference mapping or bidirectional link structure between knowledge pages so as to support multi-hop reasoning and context-related queries based on knowledge graphs during retrieval.

[0082] In addition, optionally, log files used for related logging and querying can be updated based on the data entry processing of the first knowledge page.

[0083] The update operation on the log file may include, but is not limited to, at least one of the following:

[0084] 1) Record the entry operation event information of the first knowledge page, such as operation time, operation type (add / modify / merge), operation object identifier, etc.

[0085] 2) Record details of changes to the global index, such as newly added or modified index entries, updated summaries of vector representations, and changes to cross-reference mappings.

[0086] 3) Record key intermediate states in the knowledge distillation process, such as the core fields extracted from the large model and the generated structured summary content.

[0087] Step 104: Determine user preference information for task processing applications based on the knowledge base.

[0088] User preference information is used to characterize the degree of personalized attention and demand tendency of users towards each component of knowledge content during interaction with the knowledge base and task processing.

[0089] Optionally, in this embodiment, user preference information is modeled or represented as a multi-dimensional vector, where each dimension can correspond to a content dimension, such as methodology, experimental details, conclusions, code, data, etc. The set of dimensions can be dynamically expanded as needed.

[0090] In practical applications, at least one of the explicit feedback and implicit behavior of users and / or systems during the interaction with the knowledge base can be captured and quantified, which can be used as the basic data for perceiving user preferences, and user preference information in task processing applications based on the knowledge base can be determined through preference modeling.

[0091] Step 105: Update the target processing specification based on the user preference information, and then perform knowledge distillation and storage processing based on the updated processing specification.

[0092] This step utilizes the user preference information determined in the preceding steps to dynamically adjust and self-evolve the target processing specifications (such as specification files formed by relevant prompts, scripts / instructions used to guide knowledge base construction) currently used in the operation specification layer to guide knowledge base construction. This ensures that the updated processing specifications can guide the subsequent knowledge distillation process to generate structured knowledge pages that better meet user needs and preferences.

[0093] Optionally, in one possible implementation, the system can map user preference information, such as a multidimensional user preference vector, to the corresponding components of the target processing specification, and update the specification content based on the mapping results by adjusting the weights or constraints of the corresponding components. For example, if the "experiment details" dimension in the user preference vector has a high weight, the generation instructions for "content summary" in the target processing specification can be modified to require the retention of experimental details such as experimental parameters or environmental configurations; if the "code" dimension in the user preference vector is significant, the corresponding knowledge distillation rules in the target processing specification can be modified so that when guiding a large model to perform knowledge distillation to generate Markdown and other knowledge pages, the text description must also be converted into corresponding code blocks, rather than plain text paragraphs.

[0094] In other possible implementations, candidate processing specifications that match user preference information can be generated to guide the construction of the knowledge base, and the target processing specification can be updated based on the candidate processing specifications. For example, the candidate processing specifications can be directly used to replace the target processing specification, or the target processing specification can be modified based on the candidate processing specifications to make it more compatible with user preferences.

[0095] After updating the target processing specification, the updated specification can be used to guide the large model in performing subsequent knowledge distillation tasks, distilling unstructured, fragmented raw data into structured, standardized knowledge pages. This process can then be iterated continuously to complete a full closed loop from "user preference awareness" to "knowledge base self-evolution".

[0096] See Figure 2 It provides a schematic diagram of the self-iterative process of target processing specifications (such as Skill) based on user preference awareness. For detailed processing procedures, please refer to [link / reference]. Figure 2 The description of this embodiment will not be repeated here.

[0097] In summary, the solution in this embodiment distills raw data into knowledge pages through a large model and introduces a feedback mechanism that dynamically updates the target processing specification based on user preference information. This allows users' personalized concerns during interaction with the knowledge base to be directly transformed into optimizations of the target processing specification used to guide the construction of the knowledge base (such as adjusting prompts or script constraints in the target processing specification). This achieves a leap from simple "personalized presentation of results" to a deeper "personalized adaptation of knowledge production logic," breaking through the limitations of existing Wiki structures and operation rules (Skills) that are "one-size-fits-all" and cannot adapt to the differences between individual users. At the same time, it overcomes the shortcomings of traditional personalization technology, which only limits user preferences to result sorting and display and cannot have a reverse effect on the knowledge base's underlying intake and maintenance rules. It can drive the knowledge base to complete a closed loop from user preference perception to self-evolution, thus significantly improving the adaptability, maintenance efficiency, and content reuse value of the knowledge base in different application scenarios.

[0098] In an optional embodiment, step 104 of the method of this application, namely determining user preference information in a task processing application based on a knowledge base, can be implemented as the following steps "1-1"-"1-2".

[0099] 1-1: Obtain the interaction information corresponding to each knowledge page in the knowledge base.

[0100] The interaction information is used to characterize interactive behavior features for the corresponding knowledge page, which can be used to measure user preferences. These interactive behavior features may include, but are not limited to, at least one of user feedback behavior features and system call behavior features.

[0101] Specifically, but not limited to, at least one of the following three types of information can be statistically analyzed based on the content dimension to serve as the interactive information:

[0102] a. Explicit annotations by users on knowledge pages.

[0103] This explicit annotation information represents the user's feedback to the knowledge page and signifies the user's explicit expectations. For example, it could include the type and frequency of user likes / dislikes on answers generated based on the knowledge page, the extent / frequency of manual modifications made to the knowledge page / answer, or the specific points of concern, corrections, or number of comments / annotations mentioned in the knowledge page.

[0104] b. Link information for knowledge pages.

[0105] This link information may include, but is not limited to, the number of bidirectional inbound links to the knowledge page.

[0106] c. The system's reference information for knowledge pages.

[0107] This citation information may include, but is not limited to, the number of times a knowledge page is cited, such as the number of times a knowledge page is cited by an Agent.

[0108] Among them, link information and reference information (such as the number of bidirectional inbound links to knowledge pages and the number of times they are referenced by the Agent) are implicit interaction information, requiring no additional user action.

[0109] The aforementioned interactive information is used to jointly characterize user feedback behavior and / or system call behavior, so as to provide a data foundation for subsequent accurate quantification of user preferences.

[0110] 1-2: Determine user preference information based on the interaction information.

[0111] After obtaining the interaction information corresponding to each knowledge page in the knowledge base, the interaction information can be used as the basic data for perceiving user preferences, and user preference information can be calculated through the preference calculation formula.

[0112] The following are exemplary formulas for calculating user preference information:

[0113] .

[0114] Wherein, P_d represents the user preference component in a knowledge-based task processing application, explicit_d represents the explicit annotation information of the user on the knowledge page, backlinks_d represents the number of bidirectional inbound links to the knowledge page, and agent_refs_d represents the number of times the knowledge page is referenced by the Agent; norm represents the normalization function, used to eliminate the dimensional differences between different features; α, β, and γ represent the weighting coefficients, used to adjust the degree of influence of different interaction signals on the final preference component.

[0115] In practical applications, for different content dimensions of a knowledge page, such as methodology, experimental details, conclusions, code, data, etc., the above formulas can be used to calculate user preference components based on the explicit annotations, links, references and other interactive information under that dimension, so as to obtain the user's preference components under each content dimension. The values ​​of user preference components can be, but are not limited to, expressed in the form of scores.

[0116] The system can combine (e.g., splice) the user preference components calculated under each content dimension to obtain a multidimensional user preference vector that represents the overall user preference. This multidimensional user preference vector can be used as the user preference information described in this application, so as to provide a personalized basis for the subsequent update of the target processing specification.

[0117] This embodiment integrates explicit user feedback (such as likes and modifications) with implicit system behaviors (such as bidirectional links and Agent references) to construct a multi-dimensional interactive information system. It also achieves deep perception and precise quantification of users' personalized needs for different content dimensions such as methodology and code. This provides data support and personalized basis for subsequent updates to the target processing specifications, and can significantly improve the intelligence level and user experience of the knowledge base in task processing applications.

[0118] In an optional embodiment, step 105 of the method of this application, namely, updating the target processing specification based on the user preference information, can be implemented as the following steps "2-1"-"2-4".

[0119] 2-1: Identify at least one candidate processing specification.

[0120] Among them, the candidate processing specification is used to explore operation rules that can better adapt to user preferences.

[0121] See also Figure 3 The provided process diagram shows the iterative process of generating a target processing specification (such as skill rules) by generating candidate processing specifications. Each candidate processing specification can be generated by adjusting the target processing specification through a large model.

[0122] For example, but not limited to, the target processing specification can be adjusted or mutated by a large language model to generate a candidate processing specification by adjusting or mutating the corresponding prompt words, scripts / instructions in the target processing specification, such as adjusting the summary granularity, fields, links and naming conventions, etc.

[0123] 2-2: Based on the aforementioned candidate processing specification, data file samples are distilled into knowledge page samples using a large model.

[0124] After obtaining at least one candidate processing specification, in conjunction with [see also...] Figure 3 Each candidate processing specification can be used as a rule and instruction to guide the construction of the knowledge base. A preset data file sample can be selected as the test input. The large model can be used to perform knowledge distillation on the data file sample according to the instructions of the candidate processing specification to obtain the corresponding knowledge page sample (such as candidate wiki fragments). This enables the trial ingestion processing of data file samples based on the candidate processing specification.

[0125] 2-3: Determine the preference alignment between the knowledge page sample and user preferences.

[0126] Among them, preference alignment is used to quantitatively evaluate the degree of matching between the knowledge page samples generated in the trial run and user preferences. It can be generated by calculating the similarity between the content features of the knowledge page samples and user preference information (such as multi-dimensional user preference vectors calculated based on interaction signals such as explicit annotation, bidirectional links, and Agent citation counts).

[0127] In practical applications, we can first extract the content features of the knowledge page samples (such as the level of detail of content granularity / data details, the types of fields included, the structured form, etc.), and then further calculate the similarity between the content features and user preference information through methods such as cosine distance, so as to use the preference alignment index.

[0128] The higher the preference alignment, the more the generated knowledge page samples match the user's personalized needs.

[0129] 2-4: In response to the preference alignment degree corresponding to the corresponding candidate processing specification satisfying the target condition, update the target processing specification based on the corresponding candidate processing specification.

[0130] The target condition may include, but is not limited to, at least one of the following:

[0131] Condition 1: The preference alignment between the knowledge page samples generated based on the corresponding candidate processing specification and user preferences is greater than a set threshold.

[0132] Condition 2: The preference alignment between the first knowledge page sample generated based on the corresponding candidate processing specification and the user preference is greater than the preference alignment between the second knowledge page sample generated based on the target processing specification and the user preference.

[0133] The first knowledge page sample and the second knowledge page sample are generated by knowledge distillation of the same data file sample.

[0134] If, among the candidate processing specifications, there exists a corresponding candidate processing specification whose preference alignment meets the target condition, then the target processing specification can be updated based on that corresponding candidate processing specification. For example, the target processing specification can be directly replaced by the corresponding candidate processing specification, or the target processing specification can be modified based on the corresponding candidate processing specification to make it more compatible with user preferences.

[0135] Conversely, if there is no corresponding candidate processing specification whose preference alignment meets the target condition, then there is no need to update the current target processing specification.

[0136] In practical applications, optionally, the above processing procedure can be executed repeatedly multiple times during a single update event that triggers the target processing specification, to iteratively update the target processing specification until the termination condition is met and the update process stops.

[0137] The termination conditions may include, but are not limited to, at least one of the following: the number of iterations reaches the set upper limit; the iteration time reaches the set upper limit; the update effect is not significantly improved (for example, before / after the target processing specification is updated, the alignment difference between the knowledge pages distilled from the same original data file and user preferences is less than a threshold).

[0138] This embodiment is based on the self-iterative loop idea of ​​"propose modification → evaluate → retain the best" and constructs a self-iterative update mechanism for target processing rules (knowledge distillation skills) based on user preferences. It realizes the automatic judgment and screening of the superiority and inferiority of the specification, and can continuously and adaptively optimize the target processing specification. This ensures that the construction logic of the knowledge base always evolves dynamically in the direction that best meets the real needs of users, and significantly improves the accuracy of knowledge intake and the personalized service level of the knowledge base.

[0139] In one optional embodiment, during the data ingestion stage (knowledge distillation and storage processing of the original data file), an incremental ingestion method based on content fingerprints can be adopted.

[0140] Specifically, a unique content hash value can be calculated for each raw data file, and a status manifest file (such as a Manifest) can be maintained. The status manifest file may, but is not limited to, recording the identifier of the processed raw data file and its corresponding content hash value.

[0141] When triggering an ingestion task, the content hash value of the raw data file to be ingested can be calculated first and compared with the historical hash value in the status list file. If the comparison result indicates that the content hash value of the file has changed or that the content hash value of the file is not included in the status list file, the file is determined to be a modified or newly added file, and the large model (such as the LLM in the LLM Agent) is triggered to read and distill the data of the file, and the status list file is updated after processing is completed. Otherwise, if the comparison result indicates that the content hash value of the file is consistent, the processing of the file can be skipped.

[0142] This embodiment, through the aforementioned incremental ingestion mechanism, can accurately identify and perform knowledge distillation and storage processing only on newly added or changed data sources, thereby effectively avoiding the waste of computing power and other resources caused by duplicate data processing.

[0143] In an optional embodiment, the knowledge base processing method provided in this application may further include the following processing steps "3-1"-"3-3".

[0144] 3-1: Obtain the target task to be processed.

[0145] Optionally, users can input the target task through natural language interaction.

[0146] This application does not limit the specific type of the target task in its embodiments, and it can be determined according to the actual application requirements. For example, the target task can be, but is not limited to, various types of tasks such as knowledge base-based question answering, content generation and creation, logical reasoning and calculation, and data analysis.

[0147] 3-2: Determine the index information that matches the target task, so as to locate the second knowledge page that meets the association conditions of the target task from the knowledge base based on the determined index information.

[0148] In this embodiment of the application, the knowledge base is associated with a corresponding global index (e.g., the directory index file index.md associated with the knowledge page collection of the knowledge page layer). The global index maintains metadata information such as the identifier, source (e.g., the identifier of the source file), topic, level, and summary of each knowledge page in the knowledge base, which can be used for navigation and retrieval of knowledge pages.

[0149] After obtaining the target task, it can first be matched against the global index of the knowledge base, such as semantic matching or keyword matching, to determine the index information that matches the target task. Then, based on this index information, the corresponding knowledge page can be located at the knowledge page level, thus obtaining a second knowledge page that satisfies the association conditions with the target task.

[0150] The association conditions here may include, but are not limited to, the semantic similarity between the target task and the metadata such as the summary or topic of the knowledge page reaching a preset threshold, and / or the target task and the metadata such as the identifier or summary of the knowledge page containing the same concepts or entities.

[0151] It should be noted that the knowledge base in this embodiment adopts an architecture of "compile once and maintain continuously," eliminating the need to resynthesize knowledge for each query. In this model, the original materials are distilled and compiled into structured knowledge pages by the large model upon entry into the database, and associated global indexes (e.g., the directory index file index.md) are automatically generated. This ensures that the knowledge is distilled, refined, and associated with the corresponding index before querying / retrieval.

[0152] Since the knowledge pages and their corresponding indexes have been established in advance, this step does not require re-parsing the original data files or temporarily reassembling the source data fragments of the knowledge during knowledge queries / retrievals. It also does not require re-synthesizing the knowledge that matches the requirements for each query / retrieval. Instead, the knowledge pages can be reused directly, which is "compile once, maintain continuously, and reuse multiple times". This avoids the computational power consumption and context breakage problems of traditional Retrieval-Augmented Generation (RAG) technology, which requires real-time synthesis of knowledge for each query.

[0153] 3-3: Based on the target task and the second knowledge page, task reasoning is performed through a large model to obtain the task processing result corresponding to the target task.

[0154] In this step, the target task and the second knowledge page can be contextually fused to construct structured prompts and input into large models such as large language models.

[0155] After obtaining the input information, the large model uses the knowledge provided in the second knowledge page as a basis and logical premise, combined with the large model's own inherent semantic understanding and logical deduction capabilities, to reason about the target task, thereby generating the task processing result corresponding to the output target task.

[0156] Optionally, when outputting task processing results, the large model can also include corresponding source reference information in the task processing results based on the source metadata (such as the identifier of the source file) associated with the second knowledge page, so as to clearly indicate the source data on which the task processing results are based.

[0157] In this embodiment, since the second knowledge page has undergone knowledge distillation and structured compilation during the database entry stage, the large model does not need to perform secondary parsing or information reorganization on the original fragmented documents during the query or task reasoning process. This effectively avoids the risks of context breakage and illusion caused by real-time knowledge synthesis in traditional retrieval enhancement generation technology, and can efficiently and accurately output task processing results that match the target task.

[0158] In an optional embodiment, the knowledge base processing method provided in this application may further include at least one of the following processing steps "4-1"-"4-2".

[0159] 4-1: Update the interactive information corresponding to the second knowledge page.

[0160] The interactive information represents interactive behavior features that can be used to measure user preferences for the corresponding knowledge page, and may include, but are not limited to, explicit annotation information of the user on the second knowledge page, the number of bidirectional inbound links to the second knowledge page, the number of times it is cited, etc.

[0161] In this step, when the second knowledge page is retrieved and the large model performs task inference based on the second knowledge page, the number of times the hit page, i.e. the second knowledge page, is cited can be accumulated. Based on the accumulated results, the interaction information of the number of citations corresponding to the second knowledge page can be updated.

[0162] 4-2: In response to the task processing result satisfying the first quality condition, update the knowledge page in the knowledge base based on the task processing result.

[0163] The first quality condition is a condition that characterizes the task processing result as a high-quality result, and may be set to at least one of the following:

[0164] Condition 1: The confidence level of the large model's output when generating task processing results is higher than a preset threshold;

[0165] Condition 2: The task processing results have been manually reviewed or verified by external experts;

[0166] Condition 3: The task processing result received positive feedback from the user during the interaction process, such as acceptance, likes, or confirmation.

[0167] When the task processing result corresponding to the target task meets the first quality condition, it is characterized as a high-quality result. In this case, a maintenance and self-evolution mechanism for the knowledge base based on the task processing result can be triggered. Optionally, the high-quality task processing result can be used as a new knowledge increment to supplement, correct, expand, update, or delete the relevant knowledge pages (such as second knowledge pages) in the knowledge base (e.g., if the knowledge represented by an existing relevant knowledge page contradicts the knowledge represented by the task processing result, the existing knowledge page can be deleted). Alternatively, the task processing result can be written back as a new knowledge page, and the global index can be updated synchronously to ensure that the global index matches the knowledge pages in the knowledge base. In addition, the log file can be updated to record the maintenance event for subsequent querying or monitoring.

[0168] This embodiment achieves automatic maintenance and continuous self-evolution of the knowledge base by introducing a dynamic interactive feedback and high-quality task processing result accumulation mechanism. On the one hand, by updating interactive information such as the number of citations of knowledge pages in real time, the popularity of knowledge can be accurately quantified, thereby accurately capturing user preferences and personalized needs, providing support for the continuous self-evolution and accurate content matching of the knowledge base. On the other hand, by transforming high-value task processing results into structured knowledge increments (such as supplementing or correcting existing knowledge pages or generating new knowledge pages), it not only ensures that the knowledge base can be continuously iterated and updated with business interactions, maintaining timeliness and accuracy, but also effectively avoids the high cost of relying on frequent manual reconstruction of the knowledge base in the traditional model, significantly improving the system's intelligence level and the efficiency of long-tail knowledge maintenance.

[0169] In an optional embodiment, the knowledge base processing method provided in this application may further include the following processing steps "5-1"-"5-5".

[0170] 5-1: In response to the fulfillment of the inspection conditions, determine the knowledge page in the knowledge base that has been updated.

[0171] The inspection conditions may include, but are not limited to, at least one of the following: reaching the set periodic inspection time, the number of knowledge pages updated or the frequency of changes in the knowledge base reaching a preset threshold, or the system's computing resource utilization or query request concurrency being lower than a preset threshold.

[0172] Updated knowledge pages in the knowledge base can include knowledge pages that have undergone updates such as knowledge supplementation, correction, expansion, or addition and deletion.

[0173] 5-2: Based on the updated knowledge page, determine the local subgraph to be inspected in the knowledge graph represented by the knowledge base, wherein the local subgraph includes the updated knowledge page.

[0174] Optionally, the local subgraph formed by the updated knowledge page and its neighboring knowledge pages with bidirectional links within n hops in the knowledge graph represented by the knowledge base can be identified as the local subgraph to be inspected. Here, n is an integer greater than or equal to 1.

[0175] For example, the local subgraph formed by the updated knowledge page and its neighboring knowledge pages with 1-2 hop bidirectional links can be identified as the local subgraph to be inspected in order to control resource consumption such as computing power.

[0176] 5-3: Determine the preference alignment between each third knowledge page and user preferences in the local sub-graph to be inspected.

[0177] The third knowledge page refers to the knowledge page contained in the local sub-map to be inspected.

[0178] The preference alignment here can be generated by calculating the similarity between the content features of the third knowledge page and user preference information (such as multi-dimensional user preference vectors calculated based on interaction signals such as explicit annotations, bidirectional links, and Agent citation counts).

[0179] In practical applications, we can first extract the content features of the third knowledge page (such as the level of detail of content granularity / data details, the types of fields included, the structured form, etc.), and then calculate the similarity between the content features and user preference information through methods such as cosine distance, and use this as a preference alignment index.

[0180] The higher the preference alignment, the more the third-party knowledge page meets the user's personalized needs.

[0181] 5-4: Based on the interaction information and preference alignment corresponding to each third knowledge page, the quality of each third knowledge page is evaluated; the interaction information represents the interactive behavior features that can be used to measure user preferences for the corresponding knowledge page.

[0182] The interactive information corresponding to the third knowledge page may include, but is not limited to, explicit annotation information, the number of bidirectional inbound links, and / or the number of times it is referenced by the Agent.

[0183] This step comprehensively evaluates the quality of the third-party knowledge page by integrating interactive information and preference alignment.

[0184] The following is an example quality assessment formula for third-party knowledge pages:

[0185]

[0186] Where score(page) represents the quality score of the third knowledge page, and w1, w2, and w3 represent the weight coefficients, respectively.

[0187] The bidirectional link centrality of a knowledge page characterizes its objective structural value. Optionally, this metric can be calculated based on, but not limited to, the number of bidirectional inbound links corresponding to the knowledge page, the quality of the knowledge page's neighboring nodes, and the knowledge page's pivotal position in the knowledge network topology. Generally, the more bidirectional inbound links a knowledge page has, the higher its bidirectional link centrality score.

[0188] 5-5: Based on the quality assessment results corresponding to each of the aforementioned third knowledge pages, perform matching quality optimization processing on each of the aforementioned third knowledge pages.

[0189] This step divides each third-party knowledge page into different quality levels (such as low pagination and high pagination) based on the quality assessment results (such as quality score), and performs targeted and differentiated quality optimization to achieve dynamic improvement of the overall quality of the knowledge base.

[0190] Specifically, if the corresponding quality assessment result indicates that the third knowledge page does not meet the second quality condition, then the third knowledge page is considered a low-level page. In this case, at least one of the following can be performed on the third knowledge page: downgrade, archive, or complete interactive information. If the corresponding quality assessment result indicates that the third knowledge page meets the second quality condition, then the third knowledge page is considered a high-level page. In this case, contradiction detection and knowledge accuracy improvement processing can be performed on the third knowledge page.

[0191] The second quality condition is a condition that characterizes the third knowledge page as a high-quality page. It can be, but is not limited to, being set to a quality score corresponding to the third knowledge page reaching a preset threshold.

[0192] In the high-level pagination conflict detection stage, the identified conflicts can be classified and processed: if it is determined to be a hard conflict, the blocking gate is activated, the process is suspended and transferred to manual review; if it is determined to be a soft conflict, it is recorded in the tracking list or logged for subsequent monitoring, without blocking the current business process.

[0193] Hard contradictions refer to absolute logical mutual exclusion between two or more pieces of knowledge, with no intermediate state or room for reconciliation, representing a fundamental conflict at the factual level. Soft contradictions refer to non-absolute conflicts between pieces of knowledge, which may manifest as flaws in expression, minor inconsistencies in non-core semantics, differences in information dimensions, or seemingly contradictory but actually multifaceted "pseudo-contradictions." These contradictions can be resolved, but are not limited to, by supplementing the context, clarifying preconditions, or integrating multiple perspectives.

[0194] See Figure 4 It provides a process diagram for optimizing knowledge base quality through dynamic inspection and multi-dimensional quality assessment. For detailed processing procedures, please refer to [link / reference]. Figure 4 The description of this embodiment will not be repeated here.

[0195] This embodiment achieves refined management and continuous improvement of knowledge base quality through a knowledge base self-optimization mechanism based on dynamic inspection and multi-dimensional quality assessment. It ensures the logical consistency and factual accuracy of core knowledge, and significantly improves the overall usability, accuracy, and matching degree of the knowledge base with user needs.

[0196] In an optional embodiment, the knowledge base processing method provided in this application may further include the following processing: updating the user preference information based on the interaction information corresponding to each third knowledge page in the local subgraph to be trained.

[0197] As mentioned above, the interactive information corresponding to the third knowledge page may include, but is not limited to, explicit annotation information, the number of bidirectional inbound links, and / or the number of times it is referenced by the Agent.

[0198] This embodiment constructs a data feedback mechanism, based on which the interactive information statistically collected from the third knowledge page is fed back to iteratively update the user preference information in the knowledge base task processing application.

[0199] This embodiment can accurately capture changes in users' actual needs and interests over time, significantly improving the timeliness and accuracy of user profiles, thereby enhancing the recommendation matching accuracy and intelligent service level of the knowledge base in subsequent task processing.

[0200] In practical applications, the knowledge base page content, processing rules (such as operation skills), and user preference information can be continuously updated based on the access of new data sources or the triggering of new query tasks, in order to achieve dynamic evolution and adaptive optimization of the knowledge base.

[0201] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 5 As shown, it may include:

[0202] Memory 10 is used to store the computer instruction set;

[0203] Computer instruction sets can be implemented in the form of computer programs.

[0204] Processor 20 is configured to implement the knowledge base processing method provided in any of the above method embodiments by executing a set of computer instructions in the memory.

[0205] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.

[0206] Optionally, the electronic device may also include a display device for displaying information.

[0207] Optionally, electronic devices may also include storage resources such as memory and cache.

[0208] Optionally, the electronic device may also include an image acquisition device.

[0209] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0210] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0211] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the knowledge base processing method provided in any of the above method embodiments.

[0212] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0213] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0214] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0215] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0216] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A knowledge base processing method, comprising: Obtain the target processing specification currently used to guide the construction of the knowledge base; Based on the target processing specification, knowledge distillation is performed on the original data file through a large model to obtain a reusable first knowledge page. Store the first knowledge page in the knowledge base and update the global index used for knowledge page retrieval; Determine user preference information for task processing applications based on the knowledge base; The target processing specification is updated based on the user preference information, and knowledge distillation and storage are performed based on the updated processing specification.

2. The knowledge base processing method according to claim 1, determining user preference information in task processing applications based on the knowledge base, includes: Obtain the interaction information corresponding to each knowledge page in the knowledge base, wherein the interaction information represents the interactive behavior features of the knowledge page that can be used to measure user preferences; User preference information is determined based on the interaction information.

3. The knowledge base processing method according to claim 2, wherein the interactive behavior features include at least one of user feedback behavior features and system call behavior features; The interactive information includes at least one of the following: explicit annotation information of the user on the knowledge page, link information of the system for the knowledge page, and reference information.

4. The knowledge base processing method according to claim 1, wherein updating the target processing specification based on the user preference information includes: At least one candidate processing specification is determined, which is generated by adjusting the target processing specification through a large model; Based on the aforementioned candidate processing specification, data file samples are distilled into knowledge page samples using a large model; Determine the preference alignment between the knowledge page sample and user preferences; In response to the preference alignment corresponding to the corresponding candidate processing specification satisfying the target condition, the target processing specification is updated based on the corresponding candidate processing specification.

5. The knowledge base processing method according to claim 1 further includes: Obtain the target task to be processed; Determine index information that matches the target task, and locate a second knowledge page in the knowledge base that meets the association conditions with the target task based on the determined index information; Based on the target task and the second knowledge page, task reasoning is performed through a large model to obtain the task processing result corresponding to the target task.

6. The knowledge base processing method according to claim 5, further comprising at least one of the following: Update the interaction information corresponding to the second knowledge page, wherein the interaction information represents the interactive behavior characteristics of the corresponding knowledge page that can be used to measure user preferences; In response to the task processing result satisfying the first quality condition, the knowledge page in the knowledge base is updated based on the task processing result.

7. The knowledge base processing method according to claim 1 further includes: In response to the fulfillment of inspection conditions, determine the knowledge pages in the knowledge base that have been updated; Based on the updated knowledge page, a local subgraph to be inspected is determined in the knowledge graph represented by the knowledge base, and the local subgraph includes the updated knowledge page; Determine the preference alignment between each third knowledge page in the local subgraph and the user preferences; Based on the interaction information and preference alignment corresponding to each of the third knowledge pages, a quality assessment is performed on each of the third knowledge pages. The interactive information represents the interactive behavior characteristics of the corresponding knowledge page that can be used to measure user preferences. Based on the quality assessment results corresponding to each of the third knowledge pages, a matching quality optimization process is performed on each of the third knowledge pages.

8. The knowledge base processing method according to claim 7, based on the quality assessment results corresponding to each of the third knowledge pages, performs matching quality optimization processing on each of the third knowledge pages, including: In response to the corresponding quality assessment result indicating that the third knowledge page does not meet the second quality condition, at least one of the following is performed on the third knowledge page: downgrade, archive, or complete interactive information; In response to the corresponding quality assessment result indicating that the third knowledge page meets the second quality condition, contradiction detection and knowledge accuracy improvement processing are performed on the third knowledge page.

9. The knowledge base processing method according to claim 7 further includes: The user preference information is updated based on the interaction information corresponding to each third knowledge page in the local subgraph.

10. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor is configured to perform the following processing by executing a set of instructions stored in the memory: Obtain the target processing specification currently used to guide the construction of the knowledge base; Based on the target processing specification, knowledge distillation is performed on the original data file through a large model to obtain a reusable first knowledge page. Store the first knowledge page in the knowledge base and update the global index used for knowledge page retrieval; Determine user preference information for task processing applications based on the knowledge base; The target processing specification is updated based on the user preference information, and knowledge distillation and storage are performed based on the updated processing specification.

Images