Knowledge base updating and deleting method and system based on industry large model

Abstract

The invention provides a knowledge base updating and deleting method and system based on an industry large model. According to the method, professional knowledge documents in different industry fields are uploaded to a knowledge base, data cleaning processing is carried out on the knowledge base, and id is appointed and stored in a vector database and a traditional database, so that construction of the knowledge base is completed. And matching knowledge most relevant to the target retrieval problem as a recall result through a specified knowledge base retrieval range and a specified retrieval mode. Partial knowledge existing in the knowledge base is updated or deleted through the specified id, and deletion of the whole knowledge base is also supported. In the document processing stage, due to different document formats provided by the user, the different document formats provided by the user need to be uniformly processed into the json data format, and user-defined expansion of the json data is also supported. Through the process, an efficient and rapid method for updating and deleting the knowledge base is provided, and the timeliness of the knowledge base can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for updating and deleting a knowledge base based on an industry large model. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of big models has penetrated into all aspects of all walks of life. From natural language processing, content generation to image recognition, big models have demonstrated amazing capabilities in various fields. However, when dealing with problems in specific fields, big models may lack the necessary background knowledge, resulting in inaccurate or insufficiently relevant output results. The core idea of ​​the knowledge base is to allow big models to utilize additional knowledge, which is not just a storage of data, but also contains verified and organized information.

[0003] At present, large model knowledge bases need to be frequently updated and deleted to ensure the timeliness of the knowledge base content to reflect the latest research results, industry trends, changes in laws and regulations, etc.

[0004] However, the existing large model knowledge base lacks updating and deletion methods, and is unable to frequently update and delete knowledge base content based on the latest research results, industry trends, and changes in laws and regulations, which reduces the timeliness of the knowledge base. Summary of the invention

[0005] The embodiment of the present invention provides a method and system for updating and deleting a knowledge base based on an industry large model, which can enhance the timeliness of the knowledge base. The method includes:

[0006] A1: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format;

[0007] A2: Clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID;

[0008] A3: Upload the at least one piece of data to be stored to a vector database and a traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequence ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequence ID;

[0009] A4: upon receiving a vector search instruction from an external source, determining a first matching degree between a target search question and the vector database index;

[0010] A5: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index;

[0011] A6: According to the sequence ID, update or delete operations are performed on the stored data of the vector database and the traditional database.

[0012] Preferably,

[0013] After A2 and before A3, further comprising:

[0014] D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3;

[0015] D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3;

[0016] D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

[0017] Preferably,

[0018] The A4 includes:

[0019] When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem;

[0020] According to the association relationship between the vector database index and the sequence id, a first matching degree between the vectorized target retrieval problem and the vector database index is determined.

[0021] Preferably,

[0022] The A5 includes:

[0023] When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

[0024] Preferably,

[0025] The A6 includes:

[0026] When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated;

[0027] Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector database index is used to represent the target vector database index of the input data to be updated;

[0028] Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index;

[0029] When the traditional database needs to be updated, determining a second current sequence id based on the local id record file;

[0030] Determine the corresponding current full amount of data based on the second current sequence id;

[0031] The current full data is replaced with the target full data and a second target sequence id is assigned, and the data is uploaded to the traditional database.

[0032] Preferably,

[0033] When it is necessary to perform a complete deletion operation on the vector database, the vector database index, the local maintained full data file, the local ID record file and the cached data in the memory are deleted;

[0034] When a complete deletion operation needs to be performed on the traditional database, all stored data corresponding to the traditional database is deleted based on the target deletion database name.

[0035] Preferably,

[0036] When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file;

[0037] Based on the third current sequence id, determining a corresponding target vector database index field;

[0038] Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file;

[0039] Based on the first stored data, deleting corresponding target cache data;

[0040] When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file;

[0041] Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

[0042] In a second aspect, an embodiment of the present invention provides a system for updating and deleting a knowledge base based on an industry big model, the system comprising: a knowledge base construction module, a knowledge base retrieval module, and a knowledge base update and deletion module;

[0043] The knowledge base construction module is used to perform:

[0044] A1: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format;

[0045] A2: Clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID;

[0046] A3: Upload the at least one piece of data to be stored to a vector database and a traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequence ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequence ID;

[0047] The knowledge base retrieval module is used to perform:

[0048] A4: upon receiving a vector search instruction from an external source, determining a first matching degree between a target search question and the vector database index;

[0049] A5: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index;

[0050] The knowledge base update and deletion module is used to execute:

[0051] A6: According to the sequence ID, update or delete operations are performed on the stored data of the vector database and the traditional database.

[0052] Preferably,

[0053] After the knowledge base construction module and before the knowledge base retrieval module, it further includes: a data processing module;

[0054] The data processing module is used to execute:

[0055] D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3;

[0056] D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3;

[0057] D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

[0058] Preferably,

[0059] The knowledge base update and deletion module is also used to execute:

[0060] When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem;

[0061] According to the association relationship between the vector database index and the sequence id, a first matching degree between the vectorized target retrieval problem and the vector database index is determined.

[0062] Preferably,

[0063] The A5 includes:

[0064] When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

[0065] Preferably,

[0066] The knowledge base update and deletion module is also used to execute:

[0067] When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated;

[0068] Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector database index is used to represent the target vector database index of the input data to be updated;

[0069] Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index;

[0070] When the traditional database needs to be updated, determining a second current sequence id based on the local id record file;

[0071] Determine the corresponding current full amount of data based on the second current sequence id;

[0072] The current full data is replaced with the target full data and a second target sequence id is assigned, and the data is uploaded to the traditional database.

[0073] Preferably,

[0074] When it is necessary to perform a complete deletion operation on the vector database, the vector database index, the local maintained full data file, the local ID record file and the cached data in the memory are deleted;

[0075] When a complete deletion operation needs to be performed on the traditional database, all stored data corresponding to the traditional database is deleted based on the target deletion database name.

[0076] Preferably,

[0077] The knowledge base update and deletion module is also used to execute:

[0078] When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file;

[0079] Based on the third current sequence id, determining a corresponding target vector database index field;

[0080] Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file;

[0081] Based on the first stored data, deleting corresponding target cache data;

[0082] When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file;

[0083] Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

[0084] The embodiment of the present invention provides a method and system for updating and deleting a knowledge base based on an industry big model. The method mainly uploads professional knowledge documents in different industry fields to the knowledge base, performs data cleaning on the knowledge base, and specifies an ID to store it in a vector database and a traditional database to complete the construction of the knowledge base. By specifying the knowledge base search range and the specified search method, the knowledge most relevant to the target search problem is matched as the recall result. By specifying an ID, part of the knowledge that already exists in the knowledge base is updated or deleted, and the entire knowledge base is also supported to be deleted. In the document processing stage, due to the different document formats provided by the user (such as: word documents and Excel documents), the different document formats provided by the user need to be uniformly processed into json data format (for example: extracting the document title into json data "title", and extracting the document content into json data "content"), and user-defined extended json data is also supported. In the data processing stage, since the processed json data may have invalid data formats such as redundant spaces, line breaks and useless symbols, it is also necessary to perform data cleaning on the processed json data to generate at least one piece of data to be stored, and upload it to the vector database and the traditional database. At the same time, it is also necessary to specify an ID for the data to be stored, so as to update or delete part or all of the knowledge in the knowledge base through the specified ID. Through the above process, an efficient and fast method for updating and deleting the knowledge base is provided, which can enhance the timeliness of the knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 It is a flow chart of a method for updating and deleting a knowledge base based on an industry big model provided by an embodiment of the present invention;

[0087] Figure 2 It is a flowchart of another method for updating and deleting a knowledge base based on an industry big model provided by an embodiment of the present invention;

[0088] Figure 3 It is a schematic diagram of a system for updating and deleting a knowledge base based on an industry big model provided by an embodiment of the present invention;

[0089] Figure 4 It is a schematic diagram of another system for updating and deleting a knowledge base based on an industry big model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] like Figure 1 As shown, an embodiment of the present invention provides a method for updating and deleting a knowledge base based on an industry large model, and the method may include the following steps:

[0092] Step 101: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format;

[0093] Step 102: clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes: at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID;

[0094] Step 103: Upload at least one piece of data to be stored to the vector database and the traditional database, wherein the vector database is used to reflect the association between the vector database index and the sequential ID, and the traditional database is used to reflect the association between the traditional database index and the sequential ID;

[0095] Step 104: upon receiving a vector search instruction from an external source, determining a first matching degree between the target search question and the vector database index;

[0096] Step 105: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index;

[0097] Step 106: According to the sequence ID, update or delete operations are performed on the input data of the vector database and the traditional database.

[0098] In an embodiment of the present invention, a method for updating and deleting a knowledge base based on an industry large model is provided. The method mainly uploads professional knowledge documents in different industry fields to the knowledge base, performs data cleaning on the knowledge base, and specifies an ID to store it in a vector database and a traditional database to complete the construction of the knowledge base. By specifying the knowledge base search scope and the specified search method, the knowledge most relevant to the target search problem is matched as the recall result. By specifying an ID, part of the knowledge that already exists in the knowledge base is updated or deleted, and the entire knowledge base is also supported to be deleted. In the document processing stage, due to the different document formats provided by the user (such as: word documents and Excel documents), the different document formats provided by the user need to be uniformly processed into json data format (for example: extracting the document title into json data "title", and extracting the document content into json data "content"), and user-defined extended json data is also supported. In the data processing stage, since the processed json data may have invalid data formats such as redundant spaces, line breaks and useless symbols, it is also necessary to perform data cleaning on the processed json data to generate at least one piece of data to be stored, and upload it to the vector database and the traditional database. At the same time, it is also necessary to specify an ID for the data to be stored, so as to update or delete part or all of the knowledge in the knowledge base through the specified ID. Through the above process, an efficient and fast method for updating and deleting the knowledge base is provided, which can enhance the timeliness of the knowledge base.

[0099] In order to process data with a large data length, in one embodiment of the present invention, after step 102 and before step 103 in the above embodiment, the following steps are further included:

[0100] D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3;

[0101] D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3;

[0102] D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

[0103] In an embodiment of the present invention, during the data processing stage, since the input json data may have a long data length, it is necessary to slice the data "content" in the data to be stored whose data length is greater than a preset data length threshold (for example, configure the slice length), and assign a unique sequential id to each processed slice data one by one. At the same time, it is necessary to record the sequential id of each data to be stored in the local id record file so as to facilitate subsequent knowledge deletion through the id.

[0104] Specifically, since the allocation of UUID does not require the intervention of a central management agency, it is unique and persistent, and is very suitable for use as a uniform resource name, and can generate new identifiers without a registration process. Therefore, the ID here can be represented by UUID, and the corresponding sequential ID is determined by UUID.

[0105] In order to perform vector search and keyword search, in one embodiment of the present invention, step 104 in the above embodiment may include the following steps:

[0106] When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem;

[0107] According to the association relationship between the vector database index and the sequence id, a first matching degree between the vectorized target retrieval problem and the vector database index is determined.

[0108] When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

[0109] In an embodiment of the present invention, the target search problem can be searched by three different search methods: vector search, keyword search, and multi-way recall. To support vector search and keyword search, when storing the data to be stored in the database, the data needs to be stored in the vector database and the traditional database at the same time, and the "content" field is selected as the database index field for creation, and other fields are in the form of local document records to reduce the overhead of vector search. For the vector database, the "content" field content can be vectorized using an embedded model, and the vectorized data is indexed by specifying a sequential id and stored in the vector database. After vectorization, the target search problem is matched with the vector database index, and the topk most similar results are returned by determining the first matching degree between the vectorized target search problem and the vector database index. The local maintenance of the full data file is searched by the position of the vector database index, and the full result is returned. In order to improve the speed of vector search, the file is no longer read every time, and the full data is cached in the memory, and the cached data in the memory is directly read. The traditional database can take the form of indexing the "content" field and storing the full field in the library. When storing, the sequential id is specified for the subsequent update and deletion functions of the specified id. Traditional databases use keyword retrieval to match keywords between the target retrieval question and the traditional database index field data, calculate the score to determine the second matching degree between the target retrieval question and the traditional database index, and return the topk results with the highest scores. The returned data is the full amount of data in the database inventory. In addition, you can also use a multi-way recall retrieval method to perform a preliminary recall through vector retrieval and keyword retrieval, deduplicate the recall results, and then sort them through the reranker rearrangement model to obtain the topk results with the highest scores and return them.

[0110] In order to update the vector database and the traditional database, in one embodiment of the present invention, step 106 in the above embodiment may include the following steps:

[0111] When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated;

[0112] Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector index is used to represent the target vector database index of the input data to be updated;

[0113] Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index;

[0114] When the traditional database needs to be updated, determining a second current sequence id based on the local id record file;

[0115] Determine the corresponding current full amount of data based on the second current sequence id;

[0116] The current full data is replaced with the target full data and a second target sequence id is assigned, and the data is uploaded to the traditional database.

[0117] In an embodiment of the present invention, when the vector database needs to be updated, the corresponding first current sequence id can be determined based on the uuid of the updated knowledge, and then the corresponding vector database index can be found and deleted based on the first current sequence id. After the new content is vectorized, the first target sequence id is specified to be re-entered into the database, and the local maintenance full data file and the cache data file in the memory are updated accordingly. When updating the traditional database, it is necessary to find the stored data, i.e. the current full data, through the specified second current sequence id in the local id record file, and re-assign the target full data to the second target sequence id for storage.

[0118] In order to perform a complete deletion operation on the vector database and the traditional database, in one embodiment of the present invention, step 106 in the above embodiment may include the following steps:

[0119] When it is necessary to perform a complete deletion operation on the vector database, the vector database index, the locally maintained full data file, the local ID record file and the cached data file in the memory of the vector database are deleted;

[0120] When a complete deletion operation needs to be performed on the traditional database, all stored data corresponding to the traditional database is deleted based on the target deletion database name.

[0121] In an embodiment of the present invention, when a complete deletion operation is required for the vector database and the traditional database, all the stored data in the vector database and the traditional database need to be deleted, and the locally maintained data files need to be deleted at the same time. The vector database first needs to delete the stored vector database index, and also needs to delete the locally maintained full data files and local id record files. In order to improve the vector retrieval speed, the full data is cached in the memory, and the cached data in the memory also needs to be deleted. The traditional database only needs to delete the database by specifying the database name.

[0122] In order to perform a partial deletion operation on the vector database and the traditional database, in one embodiment of the present invention, step 106 in the above embodiment may include the following steps:

[0123] When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file;

[0124] Based on the third current sequence id, determining a corresponding target vector database index field;

[0125] Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file;

[0126] Based on the first stored data, deleting corresponding target cache data;

[0127] When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file;

[0128] Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

[0129] In an embodiment of the present invention, since multiple pieces of stored data may be stored in a knowledge base, and each piece of stored data has its own unique ID, when deleting a piece of stored data, the specified sequential ID can be determined based on the uuid in the local id record file for deletion. For the vector database, the target vector database index field with the id of the third current sequential id in the vector database index can be deleted first, and then the data records in the locally maintained full data file and the cached data file in the memory can be deleted and re-read. For the traditional database, the local id record file can be searched first, and the fourth current sequential id can be determined based on the local id record file, that is, the sequential id corresponding to the uuid is found, and the target traditional database index and all fields of the stored data in the database, that is, the second stored data, can be deleted through the fourth current sequential id.

[0130] In an embodiment of the present invention, some knowledge may become outdated or no longer accurate over time. Regularly updating the knowledge base can ensure that the information contained therein is up to date, thereby improving overall accuracy. Updating the knowledge base can reflect the latest research results, industry trends, changes in laws and regulations, etc., to ensure the timeliness of the knowledge base content. As new knowledge is added, it may conflict with existing knowledge. These conflicts can be resolved through updates and deletions to maintain the consistency of the knowledge base. An accurate, timely, and structured knowledge base can provide a better user experience, allowing users to quickly find the information they need.

[0131] like Figure 2 As shown, in order to more clearly illustrate the technical solution and advantages of the present invention, a method for updating and deleting a knowledge base based on an industry large model is described in detail below in accordance with an embodiment of the present invention, which may specifically include the following steps:

[0132] Step 201: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format;

[0133] Step 202: clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes: at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID;

[0134] Step 203: determine whether the data length of the data to be stored is greater than a preset data length threshold, if yes, execute step 204, otherwise, execute step 206;

[0135] Step 204: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is a preset data length threshold, and execute step 205;

[0136] Step 205: assigning a sequence ID to each slice data, recording the sequence ID and storing it in a local ID record file, wherein the data to be stored includes at least one slice data;

[0137] Step 206: Upload at least one piece of data to be stored to the vector database and the traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequential ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequential ID;

[0138] Step 207: upon receiving a vector search instruction from an external source, performing a vectorization operation on the target search question to generate a vectorized target search question, and determining a first matching degree between the vectorized target search question and the vector database index according to an association relationship between the vector index and the sequence ID;

[0139] Step 208: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index according to the association relationship between the traditional database index and the sequence ID;

[0140] Step 209: when the vector database needs to be updated, the corresponding first current vector database index is searched in the vector database based on the first current sequence id, the first current vector database index is replaced with the first target vector database index, and the first target vector database index is assigned the corresponding first target sequence id, wherein the first current sequence id is used to represent the current sequence id of the input data to be updated, and the first target vector database index is used to represent the target vector database index of the input data to be updated; Step 210: updating the locally maintained full data file and the cached data file in the memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index;

[0141] Step 211: when the traditional database needs to be updated, the second current sequence ID is determined based on the local ID record file, the corresponding current full data is determined based on the second current sequence ID, the current full data is replaced with the target full data and the second target sequence ID is assigned, and uploaded to the traditional database;

[0142] Step 212: When it is necessary to perform a complete deletion operation on the vector database, the vector database index, the local maintenance full data file, and the local ID record file are deleted;

[0143] Step 213: when a complete deletion operation needs to be performed on the traditional database, all the stored data of the corresponding traditional database is deleted based on the target deletion database name;

[0144] Step 214: when a partial deletion operation needs to be performed on the vector database, a third current sequence id is determined based on the local id record file, a corresponding target vector database index field is determined based on the third current sequence id, and the corresponding first storage data in the local maintenance full data file is deleted based on the target vector database index field, and the corresponding target cache data is deleted based on the first storage data;

[0145] Step 215: When a partial deletion operation needs to be performed on the traditional database, the fourth current sequence id is determined based on the local id record file, and the target traditional database index and the corresponding second stored data in the traditional database are deleted based on the fourth current sequence id.

[0146] like Figure 3 As shown, the embodiment of the present invention provides a system for updating and deleting a knowledge base based on an industry big model, the system comprising: a knowledge base construction module 301, a knowledge base retrieval module 302 and a knowledge base update and deletion module 303;

[0147] The knowledge base construction module 301 is used to execute:

[0148] A1: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format;

[0149] A2: Clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID;

[0150] A3: Upload the at least one piece of data to be stored to a vector database and a traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequence ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequence ID;

[0151] The knowledge base retrieval module 302 is used to perform:

[0152] A4: upon receiving a vector search instruction from an external source, determining a first matching degree between a target search question and the vector database index;

[0153] A5: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index;

[0154] The knowledge base update and deletion module 303 is used to execute:

[0155] A6: According to the sequence ID, update or delete operations are performed on the stored data of the vector database and the traditional database.

[0156] based on Figure 3 A system for updating and deleting knowledge bases based on industry large models is shown in FIG. Figure 4 As shown, after the knowledge base construction module 301 and before the knowledge base retrieval module 302, it further includes: a data processing module 304;

[0157] The data processing module 304 is used to execute:

[0158] D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3;

[0159] D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3;

[0160] D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

[0161] like Figure 4 As shown, based on Figure 3 The system for updating and deleting a knowledge base based on an industry large model is shown, wherein the knowledge base updating and deleting module 303 is further used to execute:

[0162] When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem;

[0163] According to the association relationship between the vector database index and the sequence id, a first matching degree between the vectorized target retrieval problem and the vector database index is determined.

[0164] based on Figure 3 A system for updating and deleting knowledge bases based on industry large models is shown in FIG. Figure 4 As shown, the knowledge base retrieval module 302 is also used to execute:

[0165] When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

[0166] based on Figure 3 A system for updating and deleting knowledge bases based on industry large models is shown in FIG. Figure 4 As shown, the knowledge base update and deletion module 303 is also used to execute:

[0167] When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated;

[0168] Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector database index is used to represent the target vector database index of the input data to be updated;

[0169] Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index;

[0170] When the traditional database needs to be updated, determining a second current sequence id based on the local id record file;

[0171] Determine the corresponding current full amount of data based on the second current sequence id;

[0172] The current full data is replaced with the target full data and a second target sequence id is assigned, and the data is uploaded to the traditional database.

[0173] based on Figure 3 A system for updating and deleting knowledge bases based on industry large models is shown in FIG. Figure 4 As shown, the knowledge base update and deletion module 303 is also used to execute:

[0174] When it is necessary to perform a complete deletion operation on the vector database, delete the vector database index, the locally maintained full data file, the local ID record file and the cached data in the memory; when it is necessary to perform a complete deletion operation on the traditional database, delete all the stored data corresponding to the traditional database based on the target deletion database name.

[0175] based on Figure 3 A system for updating and deleting knowledge bases based on industry large models is shown in FIG. Figure 4 As shown, the knowledge base update and deletion module 303 is also used to execute:

[0176] When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file;

[0177] Based on the third current sequence id, determining a corresponding target vector database index field;

[0178] Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file;

[0179] Based on the first stored data, deleting corresponding target cache data;

[0180] When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file;

[0181] Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

[0182] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a system for updating and deleting a knowledge base based on an industry big model. In other embodiments of the present invention, a system for updating and deleting a knowledge base based on an industry big model may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0183] The information interaction, execution process and other contents between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0184] An embodiment of the present invention further provides a computer-readable medium on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes a method for updating and deleting a knowledge base based on an industry big model in any embodiment of the present invention.

[0185] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0186] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0187] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0188] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0189] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0190] Each embodiment of the present invention has at least the following beneficial effects:

[0191] 1. In an embodiment of the present invention, a method and system for updating and deleting a knowledge base based on an industry big model are provided. The method mainly completes the construction of the knowledge base by uploading professional knowledge documents in different industry fields to the knowledge base, performing data cleaning processing on the knowledge base, and storing the knowledge base in a vector database and a traditional database by specifying an ID. By specifying the knowledge base search scope and the search method, the knowledge most relevant to the target search question is matched as the recall result. By specifying an ID, part of the knowledge that already exists in the knowledge base is updated or deleted, and the entire knowledge base is also deleted. In the document processing stage, due to the different document formats provided by users (such as: word documents and Excel documents), the different document formats provided by users need to be uniformly processed into json data format (for example: extracting the document title into json data "title", and extracting the document content into json data "content"), and user-defined extended json data is also supported. In the data processing stage, since the processed json data may contain invalid data formats such as extra spaces, line breaks and useless symbols, it is also necessary to clean the processed json data to generate at least one piece of data to be stored and upload it to the vector database and the traditional database. At the same time, it is also necessary to specify an id for the data to be stored so as to update or delete part or all of the knowledge in the knowledge base through the specified id. Through the above process, an efficient and fast method for updating and deleting the knowledge base is provided, which can enhance the timeliness of the knowledge base;

[0192] 2. In the embodiment of the present invention, during the data processing stage, since the input json data may have a long data length, it is necessary to slice the data "content" whose data length in the data to be stored is greater than the preset data length threshold (for example, configure the slice length), and assign a unique id to each processed slice data one by one, and at the same time, it is necessary to sequentially record the id of each data to be stored in the local id record file, so as to delete the knowledge by id later;

[0193] 3. In the embodiment of the present invention, the target search problem can be searched by three different search methods: vector search, keyword search and multi-way recall. To support vector search and keyword search, when storing the data to be stored in the database, the data needs to be stored in the vector database and the traditional database at the same time, and the "content" field is selected as the database index field for creation, and other fields are in the form of local document records to reduce the overhead of vector search. For the vector database, the "content" field content can be vectorized using an embedded model, and the vectorized data is indexed by specifying a sequential id and stored in the vector database. After vectorization, the target search problem is matched with the vector database index, and the topk most similar results are returned by determining the first matching degree between the vectorized target search problem and the vector database index. The local maintenance full data file is searched by the position of the vector database index, and the full result is returned. In order to improve the speed of vector search, the file is no longer read every time, and the full data is cached in the memory, and the cached data in the memory is directly read. The traditional database can take the form of indexing the "content" field and storing the full field in the library. When storing, the sequential id is specified for the subsequent update and deletion functions of the specified id. Traditional databases use keyword retrieval to match keywords between the target retrieval question and the traditional database index field data, calculate the score to determine the second matching degree between the target retrieval question and the traditional database index, and return the topk results with the highest scores. The returned data is the full amount of data in the database inventory. In addition, you can also use a multi-way recall retrieval method to perform a preliminary recall through vector retrieval and keyword retrieval, deduplicate the recall results, and then sort them through the reranker rearrangement model to obtain the topk results with the highest scores and return them.

[0194] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0195] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on cost and time considerations.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for updating and deleting a knowledge base based on an industry large model, characterized in that: The method includes: A1: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format; A2: Clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID; A3: Upload the at least one piece of data to be stored to a vector database and a traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequence ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequence ID; A4: upon receiving a vector search instruction from an external source, determining a first matching degree between a target search question and the vector database index; A5: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index; A6: According to the sequence ID, update or delete operations are performed on the stored data of the vector database and the traditional database.

2. The method according to claim 1, characterized in that: After A2 and before A3, further comprising: D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3; D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3; D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

3. The method according to claim 1, characterized in that The A4 includes: When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem; Determining a first matching degree between the vectorized target retrieval problem and the vector database index according to an association relationship between the vector database index and the sequence id; and / or, The A5 includes: When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

4. The method according to claim 1, characterized in that The A6 includes: When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated; Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector database index is used to represent the target vector database index of the input data to be updated; Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index; when the traditional database needs to be updated, determining a second current sequence id based on the local id record file; Determine the corresponding current full amount of data based on the second current sequence id; Replacing the current full data with the target full data and assigning a second target sequence id, and uploading it to the traditional database; and / or, When it is necessary to perform a complete deletion operation on the vector database, delete the vector database index, the local maintenance full data file, the local ID record file and the cached data in the memory; when it is necessary to perform a complete deletion operation on the traditional database, delete all the stored data corresponding to the traditional database based on the target deletion database name; and / or, When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file; Based on the third current sequence id, determining a corresponding target vector database index field; Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file; Based on the first stored data, deleting corresponding target cache data; When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file; Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

5. A system for updating and deleting knowledge bases based on industry big models, characterized in that: The system includes: a knowledge base construction module, a knowledge base retrieval module and a knowledge base update and deletion module; The knowledge base construction module is used to perform: A1: converting the data format of at least one acquired knowledge document to be processed into a target data format to generate target data, wherein the target data format is a json data format; A2: Clean the target data to delete invalid data formats therein, and generate at least one piece of data to be stored, wherein the invalid data format includes at least one of redundant spaces, line breaks, and useless symbols, and each piece of data to be stored corresponds to a different sequence ID; A3: Upload the at least one piece of data to be stored to a vector database and a traditional database, wherein the vector database is used to reflect the association relationship between the vector database index and the sequence ID, and the traditional database is used to reflect the association relationship between the traditional database index and the sequence ID; The knowledge base retrieval module is used to perform: A4: upon receiving a vector search instruction from an external source, determining a first matching degree between a target search question and the vector database index; A5: upon receiving a keyword search instruction from an external source, determining a second matching degree between the target search question and the traditional database index; The knowledge base update and deletion module is used to execute: A6: According to the sequence ID, update or delete operations are performed on the stored data of the vector database and the traditional database.

6. The system according to claim 5, characterized in that After the knowledge base construction module and before the knowledge base retrieval module, it further includes: a data processing module; The data processing module is used to execute: D1: Determine whether the data length of the data to be stored is greater than a preset data length threshold. If yes, execute step D2; otherwise, execute step A3; D2: Divide the data to be stored into at least one slice data, wherein the maximum length of the slice data is the preset data length threshold, and execute step D3; D3: Allocate the sequence ID for each slice data, record the sequence ID and store it in a local ID record file, wherein the data to be stored includes the at least one slice data.

7. The system according to claim 5, characterized in that The knowledge base update and deletion module is also used to execute: When receiving the vector retrieval instruction from the outside, performing a vectorization operation on the target retrieval problem to generate a vectorized target retrieval problem; Determining a first matching degree between the vectorized target retrieval problem and the vector database index according to an association relationship between the vector database index and the sequence id; and / or, The A5 includes: When a keyword search instruction is received from outside, a second matching degree between the target search question and the traditional database index is determined according to the association relationship between the traditional database index and the sequence ID.

8. The system according to claim 5, characterized in that The knowledge base update and deletion module is also used to execute: When the vector database needs to be updated, a corresponding first current vector database index is searched in the vector database based on the first current sequence id, wherein the first current sequence id is used to represent the current sequence id of the stored data to be updated; Replacing the first current vector database index with a first target vector database index, and assigning a corresponding first target sequence id to the first target vector database index, wherein the first target vector database index is used to represent the target vector database index of the input data to be updated; Updating a locally maintained full data file and a cached data file in memory, wherein the locally maintained full data file is used to represent the full data corresponding to the vector database index; when the traditional database needs to be updated, determining a second current sequence id based on the local id record file; Determine the corresponding current full amount of data based on the second current sequence id; The current full data is replaced with the target full data and a second target sequence id is assigned, and the data is uploaded to the traditional database.

9. The system according to claim 5, characterized in that: The knowledge base update and deletion module is also used to execute: When it is necessary to perform a complete deletion operation on the vector database, delete the vector database index, the locally maintained full data file, the local ID record file and the cached data in the memory; when it is necessary to perform a complete deletion operation on the traditional database, delete all the stored data corresponding to the traditional database based on the target deletion database name.

10. The system according to claim 5, characterized in that The knowledge base update and deletion module is also used to execute: When a partial deletion operation needs to be performed on the vector database, determining a third current sequence id based on the local id record file; Based on the third current sequence id, determining a corresponding target vector database index field; Based on the target vector database index field, deleting the first stored data corresponding to the locally maintained full data file; Based on the first stored data, deleting corresponding target cache data; When a partial deletion operation needs to be performed on the traditional database, a fourth current sequence id is determined based on the local id record file; Based on the fourth current sequence id, the target traditional database index and the corresponding second stored data in the traditional database are deleted.

Images