A knowledge base updating and retrieving method, device, equipment, medium and product

Abstract

The application discloses a knowledge base updating and retrieving method, device, equipment, medium and product. The method comprises the following steps: obtaining a to-be-processed document, and performing block processing on the to-be-processed document in a preset mode to obtain at least one to-be-tested text block; performing four-segment processing on all the to-be-tested text blocks in a preset mode to obtain four basic segments corresponding to each to-be-tested text block; performing retrieval on a target knowledge base according to the similar hash values of the basic segments and the semantic vectors of the to-be-tested text blocks to obtain a library text block set; determining the storage mode of the to-be-tested text block according to the text similarity between the to-be-tested text block and each library text block in the library text block set, so as to update the target knowledge base; and performing retrieval in the updated target knowledge base in response to retrieval information and a cutoff time point provided by a user. The scheme can effectively save storage resources, prevent the problem of repeated storage, and improve the efficiency of knowledge base updating and the accuracy of retrieval.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a method, apparatus, device, medium and product for updating and retrieving a knowledge base. Background Technology

[0002] In existing technologies, knowledge question-answering systems based on retrieval feedback frameworks have been widely adopted. These systems retrieve documents within a knowledge base to provide feedback on user questions, thereby meeting user needs. However, when these systems are applied to fields where documents are constantly evolving, such as product specifications, policies and regulations, or technical standards, they generally face the significant challenge of dynamic knowledge base management.

[0003] Currently, the industry generally attempts to alleviate the above problems by attaching version metadata to documents and manually filtering them during retrieval. However, this method relies on users providing explicit version filtering conditions, which places high demands on users and fails to fundamentally solve the pain points of underlying knowledge redundancy and content conflicts. The overall intelligence and automation level of the system still has considerable room for improvement. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and product for updating and retrieving knowledge bases to solve the problem of redundant storage of duplicate content and to improve the accuracy of knowledge base-based retrieval.

[0005] According to one aspect of this application, a method for updating and retrieving a knowledge base is provided, characterized in that the method may include: Obtain the document to be processed and perform block processing on the document in a preset manner to obtain at least one text block to be tested; All text blocks to be tested are processed into four segments using a preset method to obtain four basic segments corresponding to each text block to be tested. Based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, a search is performed on the target knowledge base to obtain a set of library text blocks; Based on the text similarity between the text block to be tested and each text block in the library text block set, the method of storing the text block to be tested is determined in order to update the target knowledge base; In response to the search information and deadline provided by the user, a search is performed in the updated target knowledge base.

[0006] According to another aspect of this application, a knowledge base updating and retrieval apparatus is provided, characterized in that it may include: The document segmentation module is used to obtain the document to be processed and to segment the document into segments in a preset manner to obtain at least one text block to be tested. The text block segmentation module is used to perform four-segmentation processing on all text blocks to be tested in a preset manner, so as to obtain four basic segments corresponding to each text block to be tested. The library text block retrieval module is used to retrieve the library text block set by searching the target knowledge base based on the similarity hash value of each basic segment and the semantic vector of each text block to be tested. The knowledge base update module is used to determine the method of storing the text block to be tested in the database based on the text similarity between the text block to be tested and each text block in the database text block set, so as to update the target knowledge base. The knowledge base retrieval module is used to retrieve information from the target knowledge base in response to the search information and deadline provided by the user.

[0007] According to another aspect of this application, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the knowledge base update and retrieval method described in any embodiment of this application.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the knowledge base update and retrieval method described in any embodiment of this application.

[0009] According to another aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the knowledge base update and retrieval method according to any embodiment of this application.

[0010] In the technical solution of this application embodiment, a document to be processed is obtained, and the document to be processed is divided into blocks according to a preset method to obtain at least one text block to be tested. First, the document to be processed is divided into blocks, and the split text blocks to be tested are used as the unit for detection and storage. This allows for the granularization of the originally lengthy document, enabling updates at the text block level and improving the flexibility of knowledge base updates. All text blocks to be tested are processed into four segments according to a preset method to obtain four basic segments corresponding to each text block to be tested. The segmentation of the similarity hash values ​​of the text blocks to be tested provides a basis for subsequent similarity judgment based on the similarity hash values. Based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, a search is performed on the target knowledge base to obtain a set of library text blocks. Filtering the library text blocks based on both similarity hash values ​​and semantic vectors improves the efficiency and accuracy of the filtering process. Based on the text similarity between the text block to be tested and each text block in the library text block set, the method of storing the text block to be tested is determined to update the target knowledge base; in response to the search information and deadline provided by the user, a search is performed in the updated target knowledge base; differentiating the storage methods of different text blocks based on text similarity can effectively save storage resources, prevent the problem of duplicate storage, and improve the efficiency of knowledge base updates and the accuracy of retrieval.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a knowledge base updating and retrieval method provided according to Embodiment 1 of this application; Figure 2A This is an example diagram of document splitting provided according to Embodiment 2 of this application; Figure 2B This is a schematic diagram of the knowledge base construction stage according to Embodiment 2 of this application; Figure 2C This is a schematic diagram of the knowledge base retrieval stage according to Embodiment 2 of this application; Figure 3 This is a schematic diagram of the structure of a knowledge base updating and retrieval device according to Embodiment 3 of this application; Figure 4This is a schematic diagram of the structure of an electronic device that implements the knowledge base update and retrieval method of the embodiments of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This application provides a flowchart of a knowledge base update and retrieval method according to Embodiment 1. This embodiment is applicable to situations involving updating a knowledge base and retrieving data based on the updated knowledge base. This method can be executed by a knowledge base update and retrieval device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the document to be processed and perform block processing on the document to be processed in a preset manner to obtain at least one text block to be tested.

[0017] The document to be processed can be a document that urgently needs to be verified for inclusion in the knowledge base and is ready to be saved to the knowledge base. Chunking processing can involve splitting the document into different text blocks, which facilitates subsequent similarity identification. The preset method for chunking processing can be based on natural paragraphs, or on the number of sentences or words. This can be preset by relevant technical personnel based on actual conditions or human experience; this embodiment does not limit this. The number of text blocks to be split into can be selected according to different situations of the document. For example, if the document has a large amount of text, splitting it into more text blocks is helpful for subsequent processing; of course, if the document has a small amount of text, it is reasonable to process it directly as a single text block. It should be noted that if there is no other knowledge content in the knowledge base at the beginning of its construction, multiple text blocks of the document to be processed can be directly saved to the knowledge base. Synchronization can also include synchronizing the hash value, semantic vector, and metadata information of each text block. Metadata information may include, but is not limited to, the publication time and version number of the text block; these are not exhaustively listed here.

[0018] S120. Perform a four-segmentation process on all text blocks to be tested using a preset method to obtain four basic segments corresponding to each text block to be tested.

[0019] The four-segment processing of the text block to be tested can be achieved by segmenting the hash value of the text block into four basic segments. Each basic segment is a hash segment of the text block to be tested, and concatenating the four hash segments can reconstruct the hash segment of the text block to be tested. The hash value is preferably a similar hash (SimHash). Since similar hash is a locality-sensitive hash, it can map high-dimensional (such as text) data into a fixed-length fingerprint (hash value), and it has the characteristic that similar but different texts will produce similar similar hash values. Therefore, assuming the similar hash value of the text block to be tested is 64 bits, the four segments can be divided into four equal segments, with each basic segment being 16 bits. This processing is performed on all text blocks to be tested that need to be determined for inclusion in the database.

[0020] S130. Based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, retrieve the target knowledge base to obtain the library text block set.

[0021] The semantic vector of the text block to be tested is a representation that maps the text content of the text block to a dense, low-dimensional real vector space. The semantic vector of the text block to be tested can be obtained using any vector transformation method in the relevant field, such as converting the text content into a semantic vector through a preset text embedding model; this embodiment does not limit this method. The target knowledge base can be the database into which the text to be processed is stored. The library text block can be the result retrieved from the target knowledge base, that is, a set of text blocks that may be similar to the text block to be tested, retrieved from the target knowledge base based on the semantic vector of the text block to be tested and the similarity hash values ​​of each basic segment under the text block to be tested. Of course, the retrieval method can be to perform vector similarity matching in the target knowledge base based on the semantic vector of the text block to be tested, or to perform retrieval based on the similarity hash values ​​of each basic segment under the text block to be tested, or to perform a weighted retrieval using both methods; this embodiment does not limit this method. It should be added that there can be more than one retrieved library text block, therefore it exists in the form of a set.

[0022] S140. Based on the text similarity between the text block to be tested and each text block in the library text block set, determine the method of storing the text block to be tested in order to update the target knowledge base.

[0023] The text similarity can be the similarity between the text block to be tested and the text blocks in the library retrieved in the previous steps. This can be achieved by comparing the semantic vectors of the text block to be tested with those of the text blocks in the library, or by comparing the similarity hash values ​​of the text block to be tested with those of the text blocks in the library, or by weighting both methods. The storage method refers to how the text block to be tested is stored in the target knowledge base. It should be noted that multiple text blocks of the document to be processed can be checked separately before storage. This is understandable because the document to be processed may contain duplicate information already in the target knowledge base. Therefore, storing only some text blocks from the document to be processed prevents the accumulation of duplicate content, improves efficiency, and reduces the storage pressure on the knowledge base.

[0024] Ingestion methods can be mainly divided into incremental ingestion and metadata update ingestion. Incremental ingestion occurs when the target knowledge base does not contain text similar to the text block under test; in this case, the text block under test is stored as new content. Metadata update ingestion occurs when the target knowledge base contains text similar to or nearly identical to the text block under test. Instead of storing the text content of the text block under test (to prevent redundant content from occupying large storage resources), only the metadata of the text block under test needs to be updated in the metadata of the similar text. For example, information such as the publication time and version number in the metadata is synchronized to the metadata of the similar text. It's understandable that if the target knowledge base already contains text content highly similar to the text block under test, it's not necessary to store the content of the text block under test in the target knowledge base. However, if the metadata of the text block under test contains relevant publication time and version information, this information is helpful for subsequent retrieval based on time or version dimensions; therefore, this metadata can be updated in the target knowledge base.

[0025] S150. In response to the search information and deadline provided by the user, perform a search in the updated target knowledge base.

[0026] The search information can be the content the user requests to retrieve from the target knowledge base, such as a search question or keywords. The deadline is the latest time point the user expects to retrieve relevant content before that deadline. This deadline can be set by the user when entering the search information; otherwise, the current time of the search can be used. Therefore, based on the user's search information and the deadline, updated content is matched and provided to the user from the target knowledge base.

[0027] In the technical solution of this application embodiment, a document to be processed is obtained, and the document to be processed is divided into blocks according to a preset method to obtain at least one text block to be tested. First, the document to be processed is divided into blocks, and the split text blocks to be tested are used as the unit for detection and storage. This allows for the granularization of the originally lengthy document, enabling updates at the text block level and improving the flexibility of knowledge base updates. All text blocks to be tested are processed into four segments according to a preset method to obtain four basic segments corresponding to each text block to be tested. The segmentation of the similarity hash values ​​of the text blocks to be tested provides a basis for subsequent similarity judgment based on the similarity hash values. Based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, a search is performed on the target knowledge base to obtain a set of library text blocks. Filtering the library text blocks based on both similarity hash values ​​and semantic vectors improves the efficiency and accuracy of the filtering process. Based on the text similarity between the text block to be tested and each text block in the library text block set, the method of storing the text block to be tested is determined to update the target knowledge base; in response to the search information and deadline provided by the user, a search is performed in the updated target knowledge base; differentiating the storage methods of different text blocks based on text similarity can effectively save storage resources, prevent the problem of duplicate storage, and improve the efficiency of knowledge base updates and the accuracy of retrieval.

[0028] In an alternative implementation, the text similarity in S140 can be determined in the following manner: A1. For any text block to be tested, determine the semantic similarity based on the semantic vectors of the text block to be tested and the library text block.

[0029] Specifically, for any test text block from the segmented text to be processed, the semantic similarity between the test text block and the semantic vector of each library text block in the library text block set is calculated to obtain the semantic similarity between the test text block and the library text blocks. This semantic similarity can be the cosine similarity between semantic vectors, and can be determined using any pre-calculation method in related technologies.

[0030] A2. Calculate the Hamming distance between the text block to be tested and the library text block based on their similarity hash values.

[0031] The Hamming distance itself refers to the number of different characters (or symbols) at corresponding positions in two strings (or sequences), but it can also be used to measure the similarity between two similar hash fingerprints. For example, to determine whether the similar hash fingerprints of two documents are similar, instead of directly comparing the original text, the Hamming distance between the two binary fingerprint strings is calculated. The smaller the Hamming distance, the more identical the two fingerprint strings are in most positions, indicating that the two documents are highly similar, possibly being near-repeated or even completely identical documents. The larger the Hamming distance, the more different the two fingerprints are, and the less similar the original document content.

[0032] A3. Determine the hash similarity based on the Hamming distance.

[0033] Hash similarity can be a similarity metric between two text blocks determined by their similar hash values. The Hamming distance is calculated based on the similar hash values, and then the hash similarity between the text block to be tested and each library text block is calculated based on the Hamming distance determined in the preceding steps.

[0034] A4. Based on the preset weight allocation, the semantic similarity and hash similarity are weighted and calculated to obtain the text similarity.

[0035] Semantic similarity and hash similarity are assigned different weights, and the text similarity between the text block to be tested and the library text blocks is obtained through weighted summation. It should be noted that the library text block set contains multiple library text blocks, and the text block to be tested needs to perform the above text similarity calculation with each of these library text blocks to subsequently determine whether there are any similarities between the text block to be tested and the library text blocks. Of course, the weight allocation can be set by relevant technical personnel according to actual conditions, and this application embodiment does not limit this.

[0036] For example, hash similarity and semantic similarity are calculated separately, and then text similarity is further calculated, as follows: in, It is based on the similarity of Hamming distance (values ​​from 0 to 1). It is based on the cosine similarity of semantic vectors (values ​​range from 0 to 1). These are weighting coefficients. , It is the SimHash value of the text chunk (all four segments). , A semantic vector representing a text chunk; This represents text similarity.

[0037] In the above implementation, the text similarity between the text block to be tested and each library text block in the library text block set is obtained by weighted calculation of semantic similarity and hash similarity respectively, which provides basic support for the subsequent selection of the library entry method based on the text similarity.

[0038] In an optional implementation, the step S140, which determines the method of storing the text block to be tested in the database based on the text similarity between the text block to be tested and each text block in the database text block set, in order to update the target knowledge base, may include: S141. The library text block with the highest text similarity between each library text block and the text block to be tested is taken as the closest text block.

[0039] In the foregoing embodiments and implementations, text similarity was calculated between all library text blocks in the library text block set and the text block to be tested. The library text block with the highest text similarity to the text block to be tested was taken as the closest text block. It is conceivable that the probability that the closest text block and the text block to be tested are the same or similar text is the highest in the library text block set.

[0040] S142. In response to the text similarity between the closest text block and the text block to be tested being greater than a preset similarity threshold, the version release time of the text block to be tested is updated to the applicable version range of the closest text block, so as to update the earliest version release time and the latest version release time included in the applicable version range.

[0041] The similarity threshold can be a pre-defined criterion for determining whether a text block in the library is highly similar to the text block under test. The applicable version range can be a set of release times and version numbers of a text block in the target knowledge base. For example, it could include the earliest release time (i.e., the earliest version release time of the text block) and the latest release time (i.e., the latest version release time of the text block as of now), as well as the version number corresponding to each release time. The existence of the applicable version range helps users filter text by release time or version number. For example, if a text block in the library is highly similar to the text block under test, and the release time of the text block under test is earlier than the earliest version release time of the text block in the library, then the earliest version release time of the text block in the library is updated to the release time of the text block under test; similarly, if the release time of the text block under test is later than the latest version release time of the text block in the library, then the latest version release time of the text block in the library is updated to the release time of the text block under test.

[0042] When the text similarity exceeds this similarity threshold, it indicates that the closest text block is indeed highly similar to the text block under test. In this case, there is no need to add the text content of the text block under test to the target knowledge base; otherwise, a large amount of duplicate and redundant text information would appear. Only the metadata content of the text block under test, such as the version release time and version number, needs to be updated to the applicable version range of the closest text block. That is, different release times and version numbers of highly similar texts need to be recorded, while the text content itself, due to its high degree of repetition, does not need to be added to the database again.

[0043] S143. In response to the fact that the text similarity between the closest text block and the text block to be tested is not greater than the similarity threshold, the text block to be tested, as well as the semantic vector and metadata of the text block to be tested, are added to the target database.

[0044] In another scenario, if the text similarity between the closest text block and the text block to be tested does not exceed the similarity threshold, it indicates that even the closest text block with the highest similarity is not a duplicate of the text block to be tested. In this case, an incremental addition operation can be performed on the text block to be tested. That is, the text content of the text block to be tested, its corresponding semantic vector, and its metadata are added to the target database. Of course, this similarity threshold can be set by relevant technical personnel according to the actual situation, and this application embodiment does not limit it.

[0045] In the above implementation, a preset similarity threshold is used to determine whether the closest text block is a duplicate text block to be tested, thereby determining how to update the text block to be tested into the target knowledge base. This provides a reliable way for incremental updates of the knowledge base. In particular, the method of updating only the metadata for duplicate text can record different release times and version numbers while reducing storage resource consumption, which helps to improve the accuracy of subsequent searches.

[0046] In one optional implementation, the step S130, which involves retrieving data from the target knowledge base based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, to obtain a set of library text blocks, may include: S131. For any text block to be tested, in response to the fact that at least three of the four basic segments of the text block to be tested have similar hash values ​​that match any three segments of the similar hash values ​​of any text block in the target knowledge base with a preset difference, the arbitrary text block is included in the first text block set.

[0047] Understandably, all existing documents in the target knowledge base will also exist in the form of split text blocks to facilitate database entry verification and retrieval. Each split text block itself has a similar hash value, as well as four segments resulting from the similar hash value being divided into four parts.

[0048] The preset difference is used to determine whether the base segment (one of the four segments of hash similarity) of the text block under test is highly similar to or duplicates a certain segment of the hash similarity of a text block in the knowledge base. This preset difference can be pre-set by relevant technical personnel according to the actual situation. It is understood that if at least three of the four base segments of the text block under test are highly similar to or even duplicates at least three of the four segments of an existing text block in the knowledge base, then the text block under test can be considered highly similar to or even duplicated with that text block. Such text blocks in the target knowledge base are then filtered out and placed in the first text block set.

[0049] S132. Based on the semantic vector of the text block to be tested, recall at least one similar text block in the target knowledge base whose semantic similarity meets the preset conditions, and include each similar text block in the second text block set.

[0050] Based on the semantic vector of the text block to be tested, text blocks similar to the text block to be tested are searched from the target knowledge base through semantic similarity recall, and these similar text blocks are placed into a second text block set. The preset conditions for judging the degree of similarity can be a quantity condition or a similarity difference condition. For example, the first preset number of text blocks with vector similarity calculated by semantic vectors are selected to form the second text block set; or, text blocks that meet the preset similarity difference condition from the vector similarity calculated by semantic vectors are selected to form the second text block set.

[0051] S133. Take the intersection of the first set of text blocks and the second set of text blocks as the library text block set.

[0052] In the first and second text block sets determined in the aforementioned steps, text blocks that are similar to the text block to be tested are found from the target knowledge base from different dimensions. The intersection of these two sets (the unit of elements in the set is a text block, so the result of taking the intersection is also a text block) is taken as the library text block set.

[0053] In the above implementation, queries are performed on the target knowledge base from two dimensions: semantic vector and similarity hash value. The intersection of the two text block sets in the query results is then taken as the library text block set. This improves the accuracy of text block filtering in the target knowledge base and provides accurate evidence for subsequent verification of whether the text block to be tested already has the same content in the target knowledge base, thus helping to improve the efficiency and accuracy of document entry detection.

[0054] In another optional implementation, the step S150, which involves performing a search in the updated target knowledge base in response to the search information and deadline provided by the user, may include: S151. Based on the search information and the deadline, a set of text blocks is obtained from the target knowledge base.

[0055] First, a preliminary screening is performed in the target knowledge base based on the search information and the deadline. For example, text blocks with similar vectors are retrieved using semantic vectors based on the search information. Then, based on the deadline, metadata filtering is used to filter out text blocks whose earliest version release time recorded in the metadata is earlier than the deadline. The resulting filtering results form the text block filter set. Text blocks in the text block filter set can then participate in the subsequent sorting as candidate text blocks.

[0056] S152. Based on the preset reordering model, determine the similarity score between the retrieval information and each candidate text block in the text block filtering set according to the retrieval information and the text block filtering set.

[0057] The reranking model (reranker model) is applicable to reordering and optimizing search results. It can perform refined semantic understanding and ranking of the results obtained from the initial search and filtering, thereby improving the reliability of the final search results. The similarity score can be the similarity evaluation result calculated by the reranking model. This application's implementation can use any reranker model from related technologies to calculate this similarity score, and this application's implementation is not limited in this regard. The similarity score must be calculated for both the search information and each candidate text block for subsequent ranking.

[0058] S153. Determine the text block weight of each candidate text block based on the deadline and the latest version release time of each candidate text block.

[0059] Understandably, the metadata of all text blocks in the target knowledge base contains both the earliest and latest version release times. The closer the latest version release time is to the deadline, the newer and more credible the text content is in terms of time; conversely, the further the latest version release time is from the deadline, the older and less credible the text content is. Therefore, the text block weight can be considered an influence factor corresponding to the version release time. This text block weight can be calculated using a pre-set algorithm based on the deadline and the latest version release time of each candidate text block. Alternatively, it can be determined using a pre-trained machine learning model, whose inputs are the deadline and the latest version release time of a candidate text block, and whose output is the text block weight.

[0060] S154. Perform the search based on similarity scores and text block weights.

[0061] Based on similarity scores and text block weights, candidate text blocks are ranked, and this ranking is output as the search results for users to choose from. For example, similarity scores can be weighted according to text block weights. The similarity scores of different candidate text blocks change after being weighted by text block weights, and they can be ranked according to the weighted similarity scores.

[0062] In a further optional implementation, determining the text block weight of each candidate text block based on the deadline and the latest version release time of each candidate text block in S152 may include: B1. Determine the release time difference based on the deadline and the latest version release time.

[0063] The release time difference can be the time difference between the latest version release time and the deadline. It is calculated by subtracting the deadline from the latest version release time.

[0064] B2. Determine the text block weight based on the publication time difference and the preset timeliness factor.

[0065] The weight of text blocks is calculated based on the publication time difference and timeliness factor, as follows: In the above formula, For text block weights, The timeliness factor is defined as Δt, where Δt = max(deadline - latest version release time, 0). It's understood that the release time difference between the deadline and the latest version release time is not necessarily positive; that is, the latest version release time may be later than the user-set deadline. In this case, the release time difference is negative. Therefore, when the release time difference is negative, the time factor should be 0. Furthermore, the timeliness factor can be preset by relevant personnel, and this embodiment does not limit this.

[0066] Accordingly, the retrieval based on similarity scores and text block weights as described in S154 includes: C1. The product of the similarity score and the text block weight is used as the overall score of the candidate text block. Specifically: in, For the overall score, For text block weights, This represents the similarity score calculated based on the re-ranking model.

[0067] C2. Select the top-ranked candidate text blocks by comprehensive score as the search results.

[0068] The candidate text blocks are sorted from largest to smallest according to their comprehensive scores, and the first preset number of candidate text blocks are output as search results for the user to view.

[0069] In the above implementation, the similarity score between the retrieved information and each text block is calculated by the re-ranking model, and then adjusted according to the text block weight in the time dimension, so that the comprehensive score can take into account the impact of version release time on retrieval, thereby improving the retrieval accuracy based on the target knowledge base.

[0070] Example 2 Figure 2A This is an example diagram of document splitting provided for Embodiment 2 of this application. This embodiment is a specific example provided based on the foregoing embodiments and implementation methods. Specifically, it is as follows: During the knowledge base construction (or updating) phase, such as Figure 2A As shown, the document is first preprocessed. The document is divided into blocks (e.g., by chapters, articles, etc.), and the SimHash value of each text block is calculated. The 64-bit SimHash value is used for a four-segment index for fast retrieval. At the same time, the semantic vector of the text block is calculated based on the vector model. After preprocessing, the SimHash value, semantic vector, and corresponding metadata information such as publication time and version number of multiple text blocks are finally obtained.

[0071] Then, the documents are incrementally added to the knowledge base in units of text blocks. When the knowledge base is empty, after the above preprocessing, the documents do not need to be judged for similar content. The information of each text block is directly written into the knowledge base. The knowledge base fields include, but are not limited to, text block identification number, text block content, text block simhash (segment 1), text block simhash (segment 2), text block simhash (segment 3), text block simhash (segment 4), text block semantic vector, file name of the text block, applicable version range of the text block, earliest publication time of the text block, latest publication time of the text block, etc.

[0072] When the knowledge base is not empty, after the above preprocessing, for each text block of the new document, a decision is made on whether to incrementally add it to the knowledge base. The specific decision rule is as follows: based on the simhash value of the text block, the simhash segment index is used to recall a set A of text blocks in the knowledge base that satisfy at least three equal segments (equivalent to the aforementioned first text block set). Based on the semantic vector of the text block, the semantic similarity is used to recall a set B of the top-K similar text blocks in the knowledge base (equivalent to the aforementioned second text block set). The intersection C of the text blocks in sets A and B is used as a candidate set, and the text similarity between the text block and each text block in set C is calculated, as follows: in, It is based on the similarity of Hamming distance (values ​​from 0 to 1). It is based on the cosine similarity of semantic vectors (values ​​range from 0 to 1). These are weighting coefficients. , It is the SimHash value of the text chunk (all four segments). , A semantic vector representing a text chunk; This represents text similarity.

[0073] when If the value exceeds the threshold, the text block is determined to be common content already existing in the library. The applicable version range for the same text block in the library is updated; that is, the version number is added to the applicable version range for the same text block in the library, and the earliest version release time (if the text block's release time is earlier than the earliest version release time of the same text block in the library, then the earliest version release time of the same text block in the library is updated to the text block's release time) and the latest version release time (if the text block's release time is later than the latest version release time of the same text block in the library, then the latest version release time of the text block is updated to the text block's release time). If the value is less than or equal to the threshold, the text block is determined to be differentiated content that does not exist in the database. Then, the SimHash value (four segments), semantic vector, applicable version range (version number), earliest version release time (i.e., release time), and latest version release time (i.e., release time) of the text block are added to the database.

[0074] During the knowledge base retrieval phase, when a query request (as mentioned above) arrives, if the user does not specify a deadline for the query request, the current time is used as the default deadline. Metadata filtering methods are used to select text blocks whose earliest version publication time is earlier than the query deadline as a preliminary candidate set. This ensures that the terms, rules, and other content represented by the candidate text blocks were published before the query deadline. Then, relevant text block candidate sets are recalled from the preliminary candidate set based on semantic similarity. The similarity score between the query request and the relevant text blocks is calculated using the reranker model. A time decay function is then used to dynamically calculate the text block weights, i.e., the decay factor. This decay factor is combined with the similarity score obtained from the reranker model to reduce the weight of older documents over time. Finally, the final Top-K results are returned. The calculation formula is as follows: in, The similarity score calculated by the Reranker model. As the decay factor, Δt = max(deadline time - latest version release time, 0). Factors related to timeliness ( A value of 0.01 indicates weak timeliness. A value of 0.05 indicates medium timeliness. =0.1 indicates strong timeliness.

[0075] Example 3 Figure 3 This is a schematic diagram of the structure of a knowledge base updating and retrieval device provided in Embodiment 3 of this application. Figure 3 As shown, the device 300 includes: The document segmentation module 310 is used to acquire the document to be processed and perform segmentation processing on the document to be processed in a preset manner to obtain at least one text block to be tested. The text block segmentation module 320 is used to perform four-segmentation processing on all text blocks to be tested in a preset manner to obtain four basic segments corresponding to each text block to be tested. The library text block retrieval module 330 is used to retrieve the library text block set by searching the target knowledge base based on the similarity hash value of each basic segment and the semantic vector of each text block to be tested. The knowledge base update module 340 is used to determine the method of storing the text block to be tested in the database based on the text similarity between the text block to be tested and each text block in the database text block set, so as to update the target knowledge base. The knowledge base retrieval module 350 is used to perform retrieval in the updated target knowledge base in response to the retrieval information and deadline provided by the user.

[0076] In the technical solution of this application embodiment, a document to be processed is obtained, and the document to be processed is divided into blocks according to a preset method to obtain at least one text block to be tested. First, the document to be processed is divided into blocks, and the split text blocks to be tested are used as the unit for detection and storage. This allows for the granularization of the originally lengthy document, enabling updates at the text block level and improving the flexibility of knowledge base updates. All text blocks to be tested are processed into four segments according to a preset method to obtain four basic segments corresponding to each text block to be tested. The segmentation of the similarity hash values ​​of the text blocks to be tested provides a basis for subsequent similarity judgment based on the similarity hash values. Based on the similarity hash values ​​of each basic segment and the semantic vector of each text block to be tested, a search is performed on the target knowledge base to obtain a set of library text blocks. Filtering the library text blocks based on both similarity hash values ​​and semantic vectors improves the efficiency and accuracy of the filtering process. Based on the text similarity between the text block to be tested and each text block in the library text block set, the method of storing the text block to be tested is determined to update the target knowledge base; in response to the search information and deadline provided by the user, a search is performed in the updated target knowledge base; differentiating the storage methods of different text blocks based on text similarity can effectively save storage resources, prevent the problem of duplicate storage, and improve the efficiency of knowledge base updates and the accuracy of retrieval.

[0077] In one alternative embodiment, the device 300 may include a text similarity determination module, which may include: The semantic similarity determination unit is used to determine the semantic similarity between any text block to be tested and the text blocks in the library, based on their semantic vectors. The Hamming distance determination unit is used to calculate the Hamming distance between the text block to be tested and the library text block based on the similarity hash value of the text block to be tested and the library text block. The hash similarity determination unit is used to determine hash similarity based on Hamming distance. The text similarity determination unit is used to calculate the text similarity by weighting semantic similarity and hash similarity according to a preset weight allocation.

[0078] In one optional implementation, the knowledge base update module 340 may include: The closest text block determination unit is used to identify the text block with the highest text similarity between each library text block and the text block to be tested as the closest text block. The version release time update unit is used to update the version release time of the text block under test to the applicable version range of the closest text block in response to the text similarity between the closest text block and the text block under test being greater than a preset similarity threshold, so as to update the earliest version release time and the latest version release time included in the applicable version range. The text incremental update unit is used to add the text block to be tested, along with its semantic vector and metadata, to the target database in response to the text similarity between the closest text block and the text block to be tested not being greater than a similarity threshold.

[0079] In one optional implementation, the library text block retrieval module 330 may include: The first text block set determination unit is used to include any text block into the first text block set if, for any text block to be tested, at least three of the four basic segments of the text block to be tested have similar hash values ​​that match any three segments of the similar hash values ​​of any text block in the target knowledge base with a preset difference. The second text block set determination unit is used to recall at least one similar text block with semantic similarity meeting the preset conditions from the target knowledge base based on the semantic vector of the text block to be tested, and to include each similar text block into the second text block set. The library text block set determination unit is used to take the intersection of the first text block set and the second text block set as the library text block set.

[0080] In one optional implementation, the knowledge base retrieval module 350 may include: The text block filtering unit is used to filter the text blocks in the target knowledge base based on the search information and the deadline. The similarity score determination unit is used to determine the similarity score between the retrieval information and each candidate text block in the text block filtering set based on the preset re-ranking model. The text block weight determination unit is used to determine the text block weight of each candidate text block based on the deadline and the latest version release time of each candidate text block; The text block retrieval unit is used to perform retrieval based on similarity scores and text block weights.

[0081] In one optional implementation, the text block weight determination unit may include: There is a duration determination subunit, which is used to determine the release time difference based on the deadline and the latest version release time; The weight calculation subunit is used to determine the weight of text blocks based on the publication time difference and the preset timeliness factor. In response, the text block retrieval unit may include: The comprehensive score determines the sub-unit, which is used to multiply the similarity score and the text block weight as the comprehensive score of the candidate text block; The search results determine the sub-units, which are used to filter the top-ranked candidate text blocks by comprehensive score as search results.

[0082] The knowledge base updating and retrieval apparatus provided in this application embodiment can execute the knowledge base updating and retrieval method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each knowledge base updating and retrieval method.

[0083] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0084] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0085] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0086] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as knowledge base updating and retrieval methods.

[0087] In some embodiments, the knowledge base update and retrieval method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the knowledge base update and retrieval method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the knowledge base update and retrieval method by any other suitable means (e.g., by means of firmware).

[0088] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0092] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0093] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0094] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the knowledge base updating and retrieval method provided in any embodiment of this application. This program product shares the same inventive concept as the knowledge base updating and retrieval methods disclosed in the embodiments of this application, and therefore will not be described in detail here.

[0095] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A knowledge base updating and retrieval method, characterized in that, The method includes: Obtain the document to be processed, and perform block processing on the document to be processed in a preset manner to obtain at least one text block to be tested; All the text blocks to be tested are processed into four segments in a preset manner to obtain four basic segments corresponding to each text block to be tested; Based on the similarity hash values ​​of each of the basic segments and the semantic vectors of each of the text blocks to be tested, a search is performed on the target knowledge base to obtain a set of library text blocks; Based on the text similarity between the text block to be tested and each text block in the library text block set, the method of storing the text block to be tested is determined in order to update the target knowledge base; In response to the search information and deadline provided by the user, a search is performed in the updated target knowledge base.

2. The method according to claim 1, characterized in that, The text similarity is determined in the following ways: For any of the text blocks to be tested, the semantic similarity is determined based on the semantic vectors of the text blocks to be tested and the library text blocks; Calculate the Hamming distance between the text block to be tested and the library text block based on their similarity hash values. Determine the hash similarity based on the Hamming distance; The semantic similarity and the hash similarity are weighted and calculated according to a preset weight allocation to obtain the text similarity.

3. The method according to claim 1, characterized in that, The step of determining the method of storing the text block to be tested in the database based on the text similarity between the text block to be tested and each text block in the database text block set, in order to update the target knowledge base, includes: The library text block with the highest text similarity to the text block to be tested is taken as the closest text block; In response to the text similarity between the closest text block and the text block to be tested being greater than a preset similarity threshold, the version release time of the text block to be tested is updated to the applicable version range of the closest text block, so as to update the earliest version release time and the latest version release time included in the applicable version range. In response to the fact that the text similarity between the closest text block and the text block to be tested is not greater than the similarity threshold, the text block to be tested, along with its semantic vector and metadata, are added to the target database.

4. The method according to claim 1, characterized in that, The step involves retrieving data from a target knowledge base based on the similarity hash values ​​of each of the basic segments and the semantic vectors of each text block to be tested, to obtain a set of library text blocks, including: For any of the text blocks to be tested, in response to the fact that at least three of the four basic segments of the text block to be tested have similar hash values ​​that match any three segments of the similar hash values ​​of any text block in the target knowledge base with a preset difference, the arbitrary text block is included in the first text block set. Based on the semantic vector of the text block to be tested, at least one similar text block whose semantic similarity meets the preset conditions is recalled from the target knowledge base, and each of the similar text blocks is included in the second text block set; The intersection of the first set of text blocks and the second set of text blocks is taken as the library text block set.

5. The method according to any one of claims 1-4, characterized in that, The step of retrieving information and a deadline provided by the user in response to the updated target knowledge base includes: Based on the search information and the deadline, a set of text blocks is obtained by filtering in the target knowledge base; Based on a preset reordering model, the similarity score between the retrieval information and each candidate text block in the text block filtering set is determined according to the retrieval information and each candidate text block. The text block weight of each candidate text block is determined based on the deadline and the latest version release time of each candidate text block. The retrieval is performed based on the similarity score and the text block weight.

6. The method according to claim 5, characterized in that, The step of determining the text block weight of each candidate text block based on the deadline and the latest version release time of each candidate text block includes: The release time difference is determined based on the aforementioned deadline and the latest version release time; The weight of the text block is determined based on the publication time difference and the preset timeliness factor. Accordingly, the retrieval based on the similarity score and the text block weight includes: The product of the similarity score and the text block weight is used as the comprehensive score of the candidate text block; The candidate text blocks ranked by the comprehensive score are selected as the search results.

7. A knowledge base updating and retrieval device, characterized in that, include: The document segmentation module is used to acquire the document to be processed and to segment the document to be processed in a preset manner to obtain at least one text block to be tested. The text block segmentation module is used to perform four-segmentation processing on all the text blocks to be tested in a preset manner to obtain four basic segments corresponding to each text block to be tested. The library text block retrieval module is used to retrieve the library text block set by searching the target knowledge base based on the similarity hash value of each of the basic segments and the semantic vector of each of the text blocks to be tested. The knowledge base update module is used to determine the method of storing the text block to be tested in the database based on the text similarity between the text block to be tested and each database text block in the database text block set, so as to update the target knowledge base. The knowledge base retrieval module is used to perform a retrieval in the updated target knowledge base in response to the retrieval information and deadline provided by the user.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the knowledge base update and retrieval method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the knowledge base update and retrieval method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the knowledge base update and retrieval method according to any one of claims 1-6.

Images