Knowledge base processing method and device

Abstract

The invention discloses a knowledge base processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining a knowledge base to be processed; taking the text in the knowledge base to be processed as a clustering object, and circularly executing the step until a loop ending condition is met: based on a clustering algorithm, clustering the clustering object by adjusting and optimizing parameters of the clustering algorithm, selecting a target category from clustering results, and taking the remaining clustering results as clustering objects of next clustering; and selecting a specified number of target texts from the target category, and constructing a processed knowledge base according to the target texts. According to the implementation mode, accurate and efficient knowledge base processing is achieved, the texts in the knowledge base are classified and screened through the clustering algorithm based on the adjusted and optimized parameters, the accuracy and reasonability of knowledge base processing are guaranteed, and the processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for processing a knowledge base. Background Art

[0002] In the large-model question-answering business scenario, the quality of the knowledge base plays an important role in the performance of the large model, and the quality of the knowledge base mainly depends on two aspects: one is the richness of data information, and the other is the accuracy of data annotation. The richness of data information can be achieved by collecting a large number of knowledge question-answer pairs or data enhancement. The rich amount of data information will bring pressure to data annotation. In order to reduce the pressure of data annotation, the knowledge base is usually filtered and screened to filter out some poor quality texts. The existing knowledge base screening and processing methods mainly include manual filtering and filtering according to preset text filtering rules.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following problems:

[0004] The manual filtering method is inefficient and requires a lot of manpower and time costs; the rule filtering method is not flexible enough and has limited accuracy, and cannot meet actual needs well. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for processing a knowledge base, which realizes accurate and efficient knowledge base processing. Based on the tuned parameters, the text in the knowledge base is classified by a clustering algorithm, and then the target text encoding is selected from the target category, which not only ensures the accuracy and rationality of the knowledge base processing, but also improves the processing efficiency.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for processing a knowledge base is provided, comprising:

[0007] Obtain the knowledge base to be processed;

[0008] The text in the knowledge base to be processed is used as a clustering object, and this step is executed repeatedly until the loop end condition is met: based on the clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, and the target category is selected from the clustering results, and the remaining clustering results are used as the clustering objects for the next clustering;

[0009] A specified number of target texts are selected from the target category, and a processed knowledge base is constructed based on the target texts.

[0010] Optionally, obtaining the knowledge base to be processed includes: obtaining question-answering conversation information within a specified time period; inputting the question-answering conversation information into a preset knowledge base extraction model, and extracting question-answer pair text from the question-answering conversation information using the knowledge base extraction model; and using the question-answer pair text as the knowledge base to be processed.

[0011] Optionally, based on the clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, including: initializing the parameters of the clustering algorithm; looping the following steps until at least one category in the clustering results meets a preset stop condition; based on the parameters of the clustering algorithm, clustering the clustering objects to obtain corresponding clustering results; and adjusting and updating the parameters of the clustering algorithm according to the clustering results.

[0012] Optionally, the clustering algorithm is a density clustering algorithm; according to the clustering results, the parameters of the clustering algorithm are adjusted and updated, including: obtaining the number of texts included in each category in the clustering results, screening valid categories whose text quantity exceeds a preset quantity threshold from each category, and taking the total number of texts contained in the valid categories as the number of clusters; according to the number of clusters and the total number of clustering objects, using a heuristic algorithm to adjust the maximum distance parameter value of neighbor nodes in the density clustering algorithm.

[0013] Optionally, before using the question-answer pair text as the knowledge base to be processed, the method also includes: performing regular matching on the question-answer pair text according to preset filtering conditions, filtering out noise data in the question-answer pair text, and using the filtered question-answer pair text as the knowledge base to be processed.

[0014] Optionally, taking the text in the knowledge base to be processed as clustering objects includes: encoding the text to obtain a text encoding, and taking the text encoding as a clustering object; selecting a specified number of target texts from the target category and constructing a processed knowledge base based on the target texts includes: selecting a specified number of target text encodings from the target category, decoding each of the target text encodings, and obtaining a processed knowledge base.

[0015] Optionally, the text encoding is a text vector; the selecting a specified number of target text encodings from the target category includes: calculating the mean vector of the target category based on the text vectors contained in the target category; for each text vector in the target category, calculating the distance between the text vector and the mean vector; sorting and scoring each text vector according to the distance, and selecting the target text vectors corresponding to the specified number of scores according to the scoring scores.

[0016] Optionally, in response to receiving processing tasks for other knowledge bases to be processed that have a similar scale to the knowledge base to be processed, the method further includes: obtaining optimization parameters of the clustering algorithm when the stop condition is met, initializing parameters of the clustering algorithm using the optimization parameters, and then clustering the clustering objects corresponding to the processing tasks to realize processing of the other knowledge bases to be processed.

[0017] According to a second aspect of an embodiment of the present invention, there is provided a knowledge base processing device, comprising:

[0018] A knowledge base acquisition module is used to acquire the knowledge base to be processed;

[0019] A clustering module is used to take the text in the knowledge base to be processed as a clustering object, and execute this step repeatedly until the loop end condition is met: based on the clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, and the target category is selected from the clustering results, and the remaining clustering results are used as the clustering objects for the next clustering;

[0020] The screening module is used to select a specified number of target texts from the target category and construct a processed knowledge base based on the target texts.

[0021] Optionally, the knowledge base acquisition module is further used to: obtain question-answering conversation information within a specified time period; input the question-answering conversation information into a preset knowledge base extraction model, and use the knowledge base extraction model to extract question-answering pair text from the question-answering conversation information; and use the question-answering pair text as the knowledge base to be processed.

[0022] Optionally, the clustering module is also used to: initialize the parameters of the clustering algorithm; loop the following steps until at least one category in the clustering results meets a preset stop condition; cluster the clustering objects based on the parameters of the clustering algorithm to obtain corresponding clustering results; and adjust and update the parameters of the clustering algorithm based on the clustering results.

[0023] Optionally, the clustering algorithm is a density clustering algorithm; the clustering module is further used to: obtain the number of texts included in each category in the clustering results, screen out valid categories whose text quantity exceeds a preset quantity threshold from each category, and take the total number of texts contained in the valid categories as the number of clusters; and use a heuristic algorithm to adjust the maximum distance parameter value of neighbor nodes in the density clustering algorithm based on the number of clusters and the total number of clustering objects.

[0024] Optionally, the processing device of the knowledge base also includes a filtering module, which is used to: before using the question-answer pair text as the knowledge base to be processed, perform regular matching on the question-answer pair text according to preset filtering conditions, and filter out noise data in the question-answer pair text, so as to use the filtered question-answer pair text as the knowledge base to be processed.

[0025] Optionally, the clustering module is further used to: encode the text to obtain a text encoding, and use the text encoding as a clustering object; the screening module is further used to: select a specified number of target text encodings from the target category, decode each of the target text encodings, and obtain a processed knowledge base.

[0026] Optionally, the text is encoded as a text vector; the screening module is also used to: calculate the mean vector of the target category based on the text vectors contained in the target category; for each text vector in the target category, calculate the distance between the text vector and the mean vector; sort and score the text vectors according to the distance, and select the target text vectors corresponding to a specified number of scores according to the scoring values.

[0027] Optionally, in response to receiving processing tasks for other knowledge bases to be processed that have a similar scale to the knowledge base to be processed, the processing device of the knowledge base also includes a parameter sharing module, which is used to: obtain the optimization parameters of the clustering algorithm when the stop condition is met, initialize the parameters of the clustering algorithm using the optimization parameters, and then cluster the clustering objects corresponding to the processing tasks to realize the processing of the other knowledge bases to be processed.

[0028] According to a third aspect of an embodiment of the present invention, there is provided an electronic processing device for a knowledge base, comprising:

[0029] one or more processors;

[0030] a storage device for storing one or more programs,

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0032] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0033] According to a fifth aspect of an embodiment of the present invention, a computer program product includes a computer program, and when the computer program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0034] An embodiment of the invention has the following advantages or beneficial effects: by obtaining a knowledge base to be processed; taking the text in the knowledge base to be processed as a clustering object, and executing this step in a loop until the loop end condition is met: based on a clustering algorithm, clustering objects are clustered by tuning the parameters of the clustering algorithm, and a target category is selected from the clustering results, and the remaining clustering results are used as clustering objects for the next clustering; a specified number of target texts are selected from the target category, and a technical solution for constructing a processed knowledge base based on the target texts realizes an accurate and efficient knowledge base processing method. Based on the tuned parameters, the text in the knowledge base is classified by a clustering algorithm, and then the target text encoding is selected from the target category, which not only ensures the accuracy and rationality of the knowledge base processing, but also improves the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0036] Figure 1 is a schematic diagram of the main process of the method for processing a knowledge base according to an embodiment of the present invention;

[0037] Figure 2 is a main flow chart of a method for processing a knowledge base according to a reference embodiment of the present invention;

[0038] Figure 3 It is a complete flow chart of the method for processing the knowledge base of an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of main modules of a knowledge base processing device according to an embodiment of the present invention;

[0040] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that in the technical solution of the present invention, the collection / collection, update, analysis, use, transmission, storage and other aspects of user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside of these legal uses, and are subject to supervision and management by national regulatory authorities. Necessary measures should be taken for user personal information to selectively block the use or access to personal information data to prevent illegal access to such personal information data, ensure that persons who have access to personal information data comply with the provisions of relevant laws and regulations, and ensure the security of user personal information. In addition, once these user personal information data are no longer needed, the risks should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] Among the existing knowledge base processing methods, the manual filtering method is inefficient and requires a lot of manpower and time costs; the rule filtering method is not flexible enough and has limited accuracy, which cannot meet actual needs and cannot meet practical applications well.

[0045] In order to solve the above problems existing in the prior art, the present invention proposes a method for processing a knowledge base, performs multiple rounds of cyclic clustering on the text corresponding to the knowledge base, and obtains clustering results by tuning the parameters of the clustering algorithm, and then selects a target category from the clustering results, so as to realize the screening processing of the text in the knowledge base by selecting the target text from the target category, thereby realizing an accurate and efficient knowledge base processing method. Based on the tuned parameters, the text in the knowledge base is classified by a clustering algorithm, and then the target text is selected from the target category, which not only ensures the accuracy and rationality of the knowledge base processing, but also improves the processing efficiency.

[0046] Figure 1 is a schematic diagram of the main process of the method for processing the knowledge base according to an embodiment of the present invention, such as Figure 1 As shown, the knowledge base processing method of the embodiment of the present invention includes the following steps S101 to S103.

[0047] Step S101: Obtain a knowledge base to be processed.

[0048] Specifically, in the large-model question-answering business scenario, the support of the knowledge base for question-answering reasoning is indispensable, and the quality of the knowledge base directly determines the performance of the large model. In addition to continuously updating the knowledge base, the knowledge base management system also needs to regularly maintain and process the knowledge base to avoid too many redundant or poor-quality question-answer texts in the knowledge base. The knowledge base is essentially a collection of texts composed of various question-answer pairs. When the knowledge base management system receives a knowledge base processing task, it follows the preset extraction rules, such as random extraction, to randomly extract question-answer pairs that meet the quantity requirements from the basic data source to form the knowledge base to be processed; or layered extraction, to extract a specified number of question-answer pairs from different population layers or business domain layers to form the knowledge base to be processed.

[0049] According to one embodiment of the present invention, obtaining a knowledge base to be processed includes: obtaining question-answering conversation information within a specified time period; inputting the question-answering conversation information into a preset knowledge base extraction model, and extracting question-answer pair text from the question-answering conversation information using the knowledge base extraction model; and using the question-answer pair text as the knowledge base to be processed.

[0050] Specifically, in order to improve the efficiency of obtaining the knowledge base to be processed, and considering the universality of the demand for knowledge base processing, the embodiment of the present invention pre-constructs a knowledge base extraction model that meets the needs of the current scenario. When receiving a knowledge base processing task, all data in the current question-and-answer conversation database or data within a specified time period are input into the knowledge base extraction model, and the knowledge base extraction model is used to filter and extract target question-and-answer conversation data from the received question-and-answer conversation data, and these target question-and-answer conversation data are standardized to obtain question-and-answer pair texts. The question-and-answer pair texts output by the knowledge base extraction model are summarized to obtain the knowledge base to be processed.

[0051] According to another embodiment of the present invention, before using the question-answer pair text as the knowledge base to be processed, the method further includes: performing regular matching on the question-answer pair text according to preset filtering conditions, filtering out noise data in the question-answer pair text, and using the filtered question-answer pair text as the knowledge base to be processed.

[0052] Specifically, in order to reduce the pressure of subsequent knowledge base processing and improve processing efficiency, after obtaining the question-answer pair text, before using the question-answer pair text as the knowledge base to be processed, the question-answer pair text is subjected to regular matching according to preset filtering conditions, such as whether it contains grammatical part of speech requirements such as subject, predicate, and object. According to the matching results, the question-answer pair text with a matching rate that reaches or exceeds the matching threshold is selected, and the question-answer pair text with a matching rate lower than the matching threshold is filtered out as noise data, and the filtered question-answer pair text is used as the knowledge base to be processed.

[0053] Step S102, taking the text in the knowledge base to be processed as the clustering object, and executing this step repeatedly until the loop end condition is met: based on the clustering algorithm, clustering the clustering objects by tuning the parameters of the clustering algorithm, selecting the target category from the clustering results, and taking the remaining clustering results as the clustering objects for the next clustering.

[0054] Specifically, based on the acquired knowledge base to be processed, the embodiment of the present invention uses clustering to filter the text in the knowledge base. Taking the text as the clustering object, due to the huge number of question-answer texts in the knowledge base, in order to improve the accuracy of knowledge base screening, the embodiment of the present invention adopts multiple rounds of clustering, and selects the best clustering category from each round of clustering results as the target category. The remaining clustering results are clustered again for the next round, that is, double screening of multiple rounds and the best category is used to ensure accuracy. In each round of clustering, the parameters of the clustering algorithm are adaptively adjusted to achieve the application of the optimal parameters to clustering, so as to ensure the accuracy of each round of clustering results, and enhance the accuracy of subsequent screening of question-answer texts in the knowledge base based on clustering results.

[0055] The loop termination condition of the multi-round clustering in the embodiment of the present invention is preset in advance, which can be a specified number of loops, a specified loop duration, or other loop termination conditions. For example, based on the set number of target question-answer pair texts, the loop terminates when the sum of the number of question-answer pair texts in the target categories that have been cyclically selected exceeds 50% of the number of target question-answer pair texts.

[0056] Step S103: Select a specified number of target texts from the target category, and construct a processed knowledge base based on the target texts.

[0057] Specifically, for the target category selected above, which contains multiple question-answer pair texts, the text similarity comparison and scoring of each question-answer pair text in the target category can be performed. Specifically, each question-answer pair text can be segmented, and the similarity scores can be obtained by comparing the similarities between the segmented words of each question-answer pair text, and a specified number of high-scoring question-answer pair texts can be selected as target question-answer pair texts. Alternatively, a preset text matching model can be used to perform similarity matching on each question-answer pair text in the target category, and the target question-answer pair texts can be selected according to the similarity scores output by the model, and these target question-answer pair texts can be summarized into a knowledge base after screening.

[0058] Figure 2 is a main flow chart of a method for processing a knowledge base according to a reference embodiment of the present invention. As another embodiment of the present invention, Figure 2 As shown, the method for processing the knowledge base may include:

[0059] Step S201: Obtain the knowledge base to be processed.

[0060] Step S202, encode the text to obtain a text code, use the text code as a clustering object, and execute this step repeatedly until the loop end condition is met: based on the clustering algorithm, cluster the clustering objects by tuning the parameters of the clustering algorithm, select the target category from the clustering results, and use the remaining clustering results as the clustering objects for the next clustering.

[0061] Specifically, in order to facilitate the clustering algorithm to focus on clustering processing, the embodiment of the present invention first encodes the text in the knowledge base to be processed. Specifically, Unicode encoding, UTF-8 encoding or ACGE encoding can be used. The embodiment of the present invention selects ACGE encoding to encode the question-answer pairs in the knowledge base to be processed to obtain text encoding, and uses the text encoding as a clustering object. The specific encoding can be executed in the knowledge base processing system, or it can be executed by an external encoding service, and the encoded text encoding is sent to the knowledge base processing system. Then, the clustering algorithm is used to cluster the text encoding to obtain the target category.

[0062] According to one embodiment of the present invention, based on a clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, including: initializing the parameters of the clustering algorithm; looping the following steps until at least one category in the clustering results meets a preset stop condition; based on the parameters of the clustering algorithm, clustering the clustering objects to obtain corresponding clustering results; and adjusting and updating the parameters of the clustering algorithm according to the clustering results.

[0063] Specifically, when using a clustering algorithm to screen text encodings, first initialize the various parameters of the clustering algorithm. Use the initialized parameters to perform the first clustering on the encoded text encodings to obtain clustering results based on the initialized parameters. It can be understood that the clustering results are aggregations of text encodings of various categories, and texts with similar semantics are aggregated in one category. Then, perform an accuracy evaluation on each category in the clustering results to evaluate the accuracy between the text encodings contained in each category and the category to which the text encodings theoretically belong. The stopping condition of the embodiment of the present invention is the accuracy threshold. For example, if the stopping condition is an accuracy of 90%, then it means that as long as one category in the clustering results has an accuracy of 90%, this round of clustering tuning is terminated, and then the category with an accuracy that reaches the threshold is selected from the results as the target category.

[0064] If the stopping condition is not met, the parameters of the current clustering algorithm are adjusted, and the text encodings are clustered again using the adjusted parameters. The accuracy of the clustering results of this clustering is evaluated. If no category meets the stopping condition, the parameters of the clustering algorithm are continued to be adjusted until at least one category in a clustering result meets the stopping condition. At this time, this round of the cycle ends, and the parameters of the current clustering algorithm are used as the optimization parameters of the current clustering object.

[0065] According to another embodiment of the present invention, the clustering algorithm is a density clustering algorithm; according to the clustering results, the parameters of the clustering algorithm are adjusted and updated, including: obtaining the number of texts included in each category in the clustering results, screening valid categories whose text quantity exceeds a preset quantity threshold from each of the categories, and taking the total number of texts contained in the valid categories as the number of clusters; according to the number of clusters and the total number of clustering objects, using a heuristic algorithm to adjust the maximum distance parameter value of neighbor nodes in the density clustering algorithm.

[0066] Specifically, the embodiment of the present invention selects to use a density clustering algorithm, and based on the clustering results of each clustering of the density clustering algorithm, a heuristic algorithm is used to adjust and update the parameters of the clustering algorithm. Based on the text encoding obtained by encoding, each category in the clustering result is first obtained, and valid categories whose number of text encodings contained therein is greater than or equal to a preset number threshold are screened from each category. It can be understood that after clustering, each text encoding is clustered in different categories, and the number of text encodings contained in each category is also different. For example, category 1 contains 5 text encodings, category 2 contains 10 text encodings, and category 3 contains 1 text encoding. It can be understood that the 1 text encoding mentioned in the embodiment of the present invention is 1 question-answer text in the knowledge base. Then the situation in which category 3 contains only 1 text encoding is not the expected result from the clustering effect alone. The clustering effect is reflected in that at least 2 text encodings are contained in one category. The number threshold of the embodiment of the present invention is 2, that is, meaningful categories are screened from the clustering results, and these meaningful categories are used as valid categories.

[0067] Further, the sum of the number of text codes contained in all valid categories is counted to obtain the number of clusters. According to the number of clusters determined by the current clustering result and the total number of text codes of the clustering objects in this round of circulation, the difference between the two is calculated. According to the difference between the two, combined with business experience, the parameters in the density clustering algorithm are appropriately adjusted. Specifically in the density clustering algorithm, the key parameter values ​​affecting the density clustering algorithm are the maximum distance parameter value eps of the neighboring nodes and the number of samples min_samples of the cluster. The embodiment of the present invention sets min_samples=1, so it is only necessary to adjust the maximum distance parameter value eps of the neighboring nodes. According to the difference between the number of clusters and the total number of text codes of the clustering objects in this round of circulation, combined with business adjustment experience, eps is appropriately adjusted, and the clustering objects are clustered again using the adjusted eps. The obtained clustering results are compared with the clustering results before adjustment to determine whether the adjustment range is appropriate. If it is not appropriate, it is corrected by reducing eps. If it is appropriate, eps is continued to be increased, and so on, until the accuracy of at least one category in the clustering result reaches 90%. The eps obtained at this time is the optimization parameter corresponding to the current clustering object.

[0068] Step S203: Select a specified number of target text codes from the target category, decode each of the target text codes, and obtain a processed knowledge base.

[0069] Specifically, the clustering object based on the above clustering process is the text encoding corresponding to the question-answer pair text in the knowledge base, so the selected target category contains the text encoding corresponding to the question-answer pair text. Then, after selecting a specified number of target text encodings from the target category, it is necessary to decode the selected target text encodings to restore the target question-answer pair texts corresponding to each target text encoding, and finally summarize these target question-answer pair texts into the knowledge base after the screening process.

[0070] According to one embodiment of the present invention, the text encoding is a text vector; the selecting a specified number of target text encodings from the target category includes: calculating the mean vector of the target category based on the various text vectors contained in the target category; for each of the text vectors in the target category, calculating the distance between the text vector and the mean vector; sorting and scoring the various text vectors according to the distance, and selecting the target text vectors corresponding to the specified number of scores according to the scoring scores.

[0071] Specifically, after selecting the corresponding target category in each cycle, it is necessary to screen the target text code from the target category. The text code of the embodiment of the present invention is essentially a text vector. For each target category: calculate the mean vector of all text vectors contained in the target category; then traverse each text vector in the target category, calculate the distance between each text vector and the mean vector, and sort and score each calculated text vector according to the distance. The closest distance has the highest score, and the highest score is ranked in front, showing a descending order of scores; finally, select a specified number of text vectors corresponding to high scores from front to back in order as the target text vector.

[0072] Figure 3 It is a complete flow chart of the method for processing the knowledge base of an embodiment of the present invention. First, data extraction is performed to extract the required question-answer text, and the question-answer text is filtered according to the preset filtering conditions to filter out the noise data in the question-answer text to obtain the knowledge base to be processed. Then the knowledge base to be processed is encoded to obtain the text encoding to be processed. Initialize the density clustering algorithm, determine the parameters of the density clustering algorithm, cluster the text encoding, obtain the clustering result, and then tune the parameters of the clustering algorithm according to the clustering result. Use the updated parameters to cluster the text encoding again until at least one category in the clustering result meets the preset accuracy value, extract the target category that meets the accuracy value, and use the remaining clustering results as the clustering object for the next clustering. Clustering and tuning the parameters of the clustering algorithm and clustering are performed again. The tuning parameters of the target category for each round of cyclic screening correspond to the preferred parameters of the current clustering object until the cycle end condition is met and the cycle ends.

[0073] At the same time, after obtaining the target category selected in each round, the target category is used as clustered data, and the distance between the text vector and the mean vector in the target category is used to perform intra-class scoring and sorting. According to the sorting result, a specified number of high-scoring (TOPk) target text codes are selected from each text code of the target category, which is intra-class screening. After decoding the screened target text codes, the screened knowledge base is obtained.

[0074] The embodiment of the present invention also lists and illustrates a process record table of a knowledge base obtained through four rounds of cyclic screening, as shown in Table 1: The total number of samples in Table 1 is the number of clustering objects (the number of text encodings), which is equivalent to the number of question-answer texts in the knowledge base to be processed. It can be seen from Table 1 that with the increase of the number of cycles, more and more clustering objects are clustered into corresponding categories. After 4 rounds of cycles, a knowledge base consisting of 33,412 samples with category relationships is obtained, and the remaining 31,212 samples are no longer considered.

[0075] According to one embodiment of the present invention, in response to receiving processing tasks for other knowledge bases to be processed that have a similar scale to the knowledge base to be processed, the method further includes: obtaining the optimization parameters of the clustering algorithm when the stop condition is met, initializing the parameters of the clustering algorithm using the optimization parameters, and then clustering the clustering objects corresponding to the processing tasks to realize the processing of the other knowledge bases to be processed.

[0076] Specifically, the knowledge base processing method of the embodiment of the present invention can not only be used to screen the question-answer texts in the knowledge base, but also the optimization parameters of the clustering algorithm in the screening process can be used to guide other knowledge bases to be processed with similar data scales in related businesses, so as to achieve the purpose of improving processing efficiency and saving computing resources. When receiving processing tasks of other knowledge bases to be processed with similar data scales as the knowledge base to be processed, the process record table of the knowledge base that has completed the screening process is retrieved, and the optimization parameters of the clustering algorithm in the reference record with a data scale similar to that of the processing task are queried. The optimization parameters are then used to initialize the parameters of the clustering algorithm, and based on the initialized parameters of the clustering algorithm, the clustering objects corresponding to the processing tasks are clustered, so as to achieve the purpose of reducing the number of clustering parameter adjustments, thereby improving the processing efficiency of the knowledge base.

[0077] The embodiment of the present invention performs multiple rounds of cyclic clustering on the question and answer texts corresponding to the knowledge base, and obtains clustering results by tuning the parameters of the clustering algorithm, and then selects the target category from the clustering results, so as to realize the screening of the texts in the knowledge base by selecting the target texts from the target category, thereby realizing an accurate and efficient knowledge base processing method. Based on the tuned parameters, the texts in the knowledge base are classified by the clustering algorithm, and then the target text encoding is selected from the target category, which not only ensures the accuracy and rationality of the knowledge base processing, but also improves the processing efficiency.

[0078] Figure 4 Schematic diagram of the main modules of the knowledge base processing device according to an embodiment of the present invention. Figure 4 As shown, the knowledge base processing device 400 mainly includes a knowledge base acquisition module 401 , a clustering module 402 and a screening module 403 .

[0079] The knowledge base acquisition module 401 is used to acquire the knowledge base to be processed;

[0080] The clustering module 402 is used to take the text in the knowledge base to be processed as a clustering object, and execute this step repeatedly until the loop end condition is met: based on the clustering algorithm, by tuning the parameters of the clustering algorithm, clustering the clustering objects, selecting the target category from the clustering results, and taking the remaining clustering results as the clustering objects for the next clustering;

[0081] The screening module 403 is used to select a specified number of target texts from the target category and construct a processed knowledge base based on the target texts.

[0082] According to one embodiment of the present invention, the knowledge base acquisition module 401 is further used to: obtain question-answering conversation information within a specified time period; input the question-answering conversation information into a preset knowledge base extraction model, and use the knowledge base extraction model to extract question-answering pair text from the question-answering conversation information; and use the question-answering pair text as the knowledge base to be processed.

[0083] According to another embodiment of the present invention, the clustering module 402 is further used to: initialize the parameters of the clustering algorithm; loop the following steps until at least one category in the clustering results meets a preset stop condition; cluster the clustering objects based on the parameters of the clustering algorithm to obtain corresponding clustering results; and adjust and update the parameters of the clustering algorithm based on the clustering results.

[0084] According to another embodiment of the present invention, the clustering algorithm is a density clustering algorithm; the clustering module 402 is further used to: obtain the number of texts included in each category in the clustering result, filter out valid categories whose text quantity exceeds a preset quantity threshold from each category, and take the total number of texts contained in the valid categories as the number of clusters; and use a heuristic algorithm to adjust the maximum distance parameter value of neighbor nodes in the density clustering algorithm according to the number of clusters and the total number of clustering objects.

[0085] According to another embodiment of the present invention, the processing device 400 of the knowledge base also includes a filtering module (not shown in the figure), which is used to: before using the question-answer pair text as the knowledge base to be processed, perform regular matching on the question-answer pair text according to preset filtering conditions, and filter out noise data in the question-answer pair text, so as to use the filtered question-answer pair text as the knowledge base to be processed.

[0086] According to another embodiment of the present invention, the clustering module 402 is further used to: encode the text to obtain a text encoding, and use the text encoding as a clustering object; the screening module 403 is further used to: select a specified number of target text encodings from the target category, decode each of the target text encodings, and obtain a processed knowledge base.

[0087] According to another embodiment of the present invention, the text is encoded as a text vector; the screening module 403 is further used to: calculate the mean vector of the target category based on the various text vectors contained in the target category; for each of the text vectors in the target category, calculate the distance between the text vector and the mean vector; sort and score the various text vectors according to the distance, and select the target text vectors corresponding to a specified number of scores according to the scoring values.

[0088] According to another embodiment of the present invention, in response to receiving processing tasks of other knowledge bases to be processed that have a similar scale to the knowledge base to be processed, the processing device 400 of the knowledge base also includes a parameter sharing module (not shown in the figure), which is used to: obtain the optimization parameters of the clustering algorithm when the stop condition is met, initialize the parameters of the clustering algorithm using the optimization parameters, and then cluster the clustering objects corresponding to the processing tasks to realize the processing of the other knowledge bases to be processed.

[0089] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention can be applied.

[0090] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0091] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications, such as knowledge base processing applications, etc., can be installed on terminal devices 501, 502, 503 (only as an example).

[0092] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0093] Server 505 may be a server that provides various services, such as a background management server that provides support for the knowledge base used by users using terminal devices 501, 502, and 503 (only as an example). The background management server may obtain the knowledge base to be processed; use the text in the knowledge base to be processed as the clustering object, and execute this step repeatedly until the loop end condition is met: based on the clustering algorithm, cluster the clustering objects by tuning the parameters of the clustering algorithm, select the target category from the clustering results, and use the remaining clustering results as the clustering objects for the next clustering; select a specified number of target texts from the target category, construct a processed knowledge base based on the target text, and feed back the final result (such as the processed knowledge base, etc. - only as an example) to the terminal device.

[0094] It should be noted that the knowledge base processing method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the knowledge base processing device is generally set in the server 505 .

[0095] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0096] Reference below Figure 6 , Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0097] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0099] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0100] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes: a knowledge base acquisition module, a clustering module and a screening module.

[0103] The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the knowledge base acquisition module can also be described as a "module for acquiring a knowledge base to be processed."

[0104] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiment; or it may exist independently without being assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes: obtaining a knowledge base to be processed; taking the text in the knowledge base to be processed as a clustering object, and executing this step repeatedly until the loop end condition is met: clustering the clustering objects based on a clustering algorithm by tuning the parameters of the clustering algorithm, selecting a target category from the clustering results, and taking the remaining clustering results as the clustering objects for the next clustering; selecting a specified number of target texts from the target category, and constructing a processed knowledge base based on the target texts.

[0105] The technical solution according to the embodiment of the present invention has the following advantages or beneficial effects: by obtaining a knowledge base to be processed; taking the text in the knowledge base to be processed as a clustering object, and executing this step in a loop until the loop end condition is met: based on a clustering algorithm, clustering objects are clustered by tuning the parameters of the clustering algorithm, and a target category is selected from the clustering results, and the remaining clustering results are used as clustering objects for the next clustering; a specified number of target texts are selected from the target category, and a technical solution for constructing a processed knowledge base based on the target texts realizes an accurate and efficient knowledge base processing method. Based on the tuned parameters, the text in the knowledge base is classified by a clustering algorithm, and then the target text encoding is selected from the target category, which not only ensures the accuracy and rationality of the knowledge base processing, but also improves the processing efficiency.

[0106] The specific implementation methods described herein do not constitute limitations on the scope of protection of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for processing a knowledge base, characterized in that: include: Obtain the knowledge base to be processed; The text in the knowledge base to be processed is used as a clustering object, and this step is executed repeatedly until the loop end condition is met: based on the clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, and the target category is selected from the clustering results, and the remaining clustering results are used as the clustering objects for the next clustering; A specified number of target texts are selected from the target category, and a processed knowledge base is constructed based on the target texts.

2. The method according to claim 1, characterized in that Get the knowledge base to be processed, including: Get the question-and-answer session information within the specified time period; Inputting the question-answering conversation information into a preset knowledge base extraction model, and extracting question-answer pair texts from the question-answering conversation information using the knowledge base extraction model; The question-answer pair text is used as a knowledge base to be processed.

3. The method according to claim 1, characterized in that Based on the clustering algorithm, clustering the clustering objects is performed by tuning the parameters of the clustering algorithm, including: Initialize the parameters of the clustering algorithm; The following steps are executed repeatedly until at least one category in the clustering result meets the preset stop condition; Based on the parameters of the clustering algorithm, the clustering objects are clustered to obtain corresponding clustering results; and according to the clustering results, the parameters of the clustering algorithm are adjusted and updated.

4. The method according to claim 3, characterized in that: The clustering algorithm is a density clustering algorithm; According to the clustering result, adjusting and updating the parameters of the clustering algorithm includes: Obtaining the number of texts included in each category in the clustering result, screening out valid categories whose number of texts exceeds a preset number threshold from each category, and taking the sum of the number of texts included in the valid categories as the number of clusters; According to the number of clusters and the total number of cluster objects, a heuristic algorithm is used to adjust the maximum distance parameter value of neighbor nodes in the density clustering algorithm.

5. The method according to claim 2, characterized in that: Before using the question-answer pair text as a knowledge base to be processed, the method further includes: According to the preset filtering conditions, the question-answer pair text is regularly matched to filter out the noise data in the question-answer pair text, so as to use the filtered question-answer pair text as the knowledge base to be processed.

6. The method according to claim 1, characterized in that The step of taking the text in the knowledge base to be processed as clustering objects includes: Encoding the text to obtain a text encoding, and using the text encoding as a clustering object; The step of selecting a specified number of target texts from the target category and constructing a processed knowledge base based on the target texts includes: A specified number of target text codes are selected from the target category, and each of the target text codes is decoded to obtain a processed knowledge base.

7. The method according to claim 6, characterized in that The text is encoded as a text vector; The step of selecting a specified number of target text codes from the target category includes: Calculate the mean vector of the target category according to each text vector contained in the target category; For each of the text vectors in the target category, calculating the distance between the text vector and the mean vector; According to the distance, each of the text vectors is sorted and scored, and target text vectors corresponding to a specified number of scores are selected according to the scoring values.

8. The method according to claim 3, characterized in that In response to receiving a processing task of another to-be-processed knowledge base having a similar size to the to-be-processed knowledge base, the method further includes: The optimization parameters of the clustering algorithm are obtained when the stop condition is met, and the parameters of the clustering algorithm are initialized using the optimization parameters, and then the clustering objects corresponding to the processing tasks are clustered to realize the processing of the other knowledge bases to be processed.

9. A knowledge base processing device, characterized in that: include: A knowledge base acquisition module is used to acquire the knowledge base to be processed; A clustering module is used to take the text in the knowledge base to be processed as a clustering object, and execute this step repeatedly until the loop end condition is met: based on the clustering algorithm, the clustering objects are clustered by tuning the parameters of the clustering algorithm, and the target category is selected from the clustering results, and the remaining clustering results are used as the clustering objects for the next clustering; The screening module is used to select a specified number of target texts from the target category and construct a processed knowledge base based on the target texts.

10. A mobile electronic device terminal, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images