Knowledge base construction method and device

By acquiring and clustering keywords in dialogue data, abnormal detection and correction are performed, the autonomy and accuracy problems of knowledge base construction and update are solved, real-time update and efficient retrieval of knowledge base are achieved.

CN119046429APending Publication Date: 2024-11-29BEIJING JINGDONG TUOXIAN TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411147471.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing knowledge base construction methods cannot be built and updated by themselves, and there are error messages, resulting in low retrieval efficiency and accuracy, and the real-time update of the knowledge base cannot be achieved.

Method used

By obtaining dialogue data for preset time periods, identifying keywords, clustering and abnormal detection, correcting abnormal keywords, updating the knowledge base, and realizing self-construction and real-time update of the knowledge base.

Benefits of technology

Improves the accuracy and accessibility of the knowledge base, improves retrieval efficiency and accuracy, and ensures real-time availability of information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046429A_ABST
    Figure CN119046429A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge base construction method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining multiple pieces of dialogue data in a preset time period, and identifying keywords corresponding to the dialogue data from each piece of dialogue data; clustering the keywords corresponding to the multiple pieces of dialogue data to obtain a category to which each keyword belongs; for each keyword corresponding to each category, performing anomaly detection on each keyword according to the category, and correcting the detected abnormal keyword, so as to update each keyword corresponding to the category; and constructing a knowledge base according to each category and each keyword corresponding to the category. According to the embodiment, self-construction of the knowledge base is achieved, the abnormal keywords are detected and corrected, the accuracy of the knowledge base is improved, the accessibility and availability of information are improved, and then the retrieval efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a knowledge base construction method and device. Background Art

[0002] With the growth of massive data, building a knowledge base can help individuals, organizations or enterprises better manage and utilize the information resources of the knowledge base, thereby effectively organizing, storing and retrieving large amounts of knowledge and information.

[0003] The knowledge base construction method in the related art is not perfect in processing abnormal data, there is erroneous information in the knowledge base, and the knowledge base cannot be automatically acquired and updated, and the knowledge base cannot be created automatically. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a knowledge base construction method and device, which can realize self-construction of the knowledge base, detect and correct abnormal keywords, improve the accuracy of the knowledge base, improve the accessibility and availability of information, and thus improve retrieval efficiency and accuracy.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a knowledge base construction method is provided, comprising:

[0006] Acquire multiple conversation data in a preset time period, and identify each keyword corresponding to the conversation data from each conversation data;

[0007] Clustering the keywords corresponding to the plurality of conversation data to obtain a category to which each keyword belongs;

[0008] For each keyword corresponding to each category, perform abnormality detection on each keyword according to preset conditions of the category, and correct the detected abnormal keywords, thereby updating each keyword corresponding to the category;

[0009] A knowledge base is constructed based on each category and its corresponding keywords.

[0010] Optionally, performing abnormality detection on each of the keywords according to the preset conditions of the category includes:

[0011] Obtaining a category dictionary corresponding to the category, and matching the keyword with the category dictionary;

[0012] In response to the keyword failing to match the category dictionary, the keyword is determined to be an abnormal keyword.

[0013] Optionally, the detected abnormal keywords are corrected, including:

[0014] In response to the keyword failing to match the category dictionary, generating a plurality of candidate words corresponding to the keyword;

[0015] For each candidate word, the candidate word is matched with the category dictionary, and the keyword is replaced with the successfully matched candidate word.

[0016] Optionally, generating a plurality of candidate words corresponding to the keyword includes:

[0017] A plurality of candidate words corresponding to the keyword are generated according to an edit distance algorithm and a preset distance.

[0018] Optionally, identifying each keyword corresponding to each conversation data from each conversation data includes:

[0019] For each piece of conversation data, parse text data from the conversation data;

[0020] Processing the text data using natural language processing technology to obtain multiple words corresponding to each piece of text data;

[0021] For each word, calculating the word frequency-inverse document frequency of the word in the text data;

[0022] According to the word frequency-inverse document frequency of each word in the text data, each keyword corresponding to the conversation data is screened out from the multiple words.

[0023] Optionally, before clustering the keywords corresponding to the plurality of conversation data, the method further includes:

[0024] For each keyword corresponding to each piece of conversation data, calculating the matching degree between the keyword and the conversation data;

[0025] Delete keywords whose matching degree is lower than the preset matching degree threshold.

[0026] Optionally, clustering each keyword corresponding to the plurality of conversation data to obtain a category to which each keyword belongs includes:

[0027] Initialize multiple cluster centers;

[0028] Perform the following clustering process: for each keyword, calculate the distance between the keyword and each cluster center, add the keyword to the cluster corresponding to the cluster center with the smallest distance, so as to obtain multiple clusters, each cluster corresponding to a category; update each cluster center according to each keyword in each cluster;

[0029] The clustering process is executed cyclically until the iteration ends, and the category to which each keyword belongs is obtained.

[0030] According to another aspect of an embodiment of the present invention, there is provided a knowledge base construction device, comprising:

[0031] An acquisition module, which acquires a plurality of conversation data in a preset time period, and identifies each keyword corresponding to the conversation data from each conversation data;

[0032] A clustering module, clustering each keyword corresponding to the plurality of conversation data to obtain a category to which each keyword belongs;

[0033] A detection module, for each keyword corresponding to each category, performs abnormal detection on the keywords according to preset conditions of the category, and corrects the detected abnormal keywords, thereby updating the keywords corresponding to the category;

[0034] The building module builds a knowledge base based on each category and its corresponding keywords.

[0035] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0036] one or more processors;

[0037] a storage device for storing one or more programs,

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the knowledge base construction method provided by the present invention.

[0039] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the knowledge base construction method provided by the present invention is implemented.

[0040] One embodiment of the above invention has the following advantages or beneficial effects: the knowledge base construction device of the embodiment of the present invention first obtains multiple conversation data of a preset time period, and then identifies each keyword corresponding to each corresponding data; then clusters each keyword corresponding to the multiple conversation data to obtain the category to which each keyword belongs; for each keyword corresponding to each category, uses the preset conditions of the category to perform abnormal detection on each keyword, and corrects the detected abnormal keywords, updates each keyword corresponding to each category, and constructs a knowledge base according to each category and its corresponding keywords. This method can realize the self-construction of the knowledge base and the real-time update of the knowledge base; by performing abnormal detection and correction on keywords, the accuracy of the knowledge base is improved, thereby improving the retrieval efficiency and accuracy, and improving the accessibility and availability of information.

[0041] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0043] Figure 1 is a schematic diagram of the main process of a knowledge base construction method according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the main process of a method for clustering keywords according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of the main process of another knowledge base construction method according to an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of main modules of a knowledge base construction device according to an embodiment of the present invention;

[0047] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0048] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0050] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0051] Figure 1 is a schematic diagram of the main process of a knowledge base construction method according to an embodiment of the present invention, such as Figure 1 As shown, the knowledge base construction method includes the following steps:

[0052] Step S101: Acquire multiple conversation data in a preset time period, and identify each keyword corresponding to the conversation data from each conversation data;

[0053] Step S102: clustering the keywords corresponding to the plurality of conversation data to obtain the category to which each keyword belongs;

[0054] Step S103: for each keyword corresponding to each category, perform anomaly detection on each keyword according to the preset conditions of the category, and correct the detected abnormal keywords, thereby updating each keyword corresponding to the category;

[0055] Step S104: construct a knowledge base according to each category and its corresponding keywords.

[0056] In an embodiment of the present invention, the knowledge base construction method can be applied to the construction of knowledge bases in multiple fields, such as medical knowledge bases, industrial knowledge bases, agricultural knowledge bases, etc., so that personnel in related fields can efficiently retrieve relevant knowledge information and improve retrieval efficiency and accuracy.

[0057] In the embodiment of the present invention, when constructing the knowledge base, firstly, a plurality of conversation data in a preset time period is obtained, and the preset time period can be set by a user, such as nearly one month, nearly one week, etc. The plurality of conversation data in the preset time period can be obtained from a website or platform through a crawler tool. For the construction of the medical knowledge base, the web crawler technology can be used to use the Scrapy framework to obtain a plurality of doctor-patient conversation data in the preset time period from the Internet medical consultation platform.

[0058] In an embodiment of the present invention, multiple conversation data in a preset time period can also be obtained through a scheduled task. Specifically, an asynchronous task processor Celery (a Python distributed task queue used to process asynchronous tasks) is used to periodically execute the acquisition of multiple conversation data, and the scheduling rules of the scheduled task are configured in the Celery configuration file. The scheduling rules of the scheduled task include information such as the execution time interval and execution time point of the scheduled task. After the conversation data is obtained through the crawler technology, the conversation data is generated into a message and sent to the message queue to achieve real-time update, so that multiple conversation data can be obtained from the message queue.

[0059] In the embodiment of the present invention, identifying each keyword corresponding to each conversation data from each conversation data includes:

[0060] For each conversation data, parse the text data from the conversation data;

[0061] Using natural language processing technology to process text data, multiple words corresponding to each piece of text data are obtained;

[0062] For each word, calculate the word frequency in the text data - inverse document frequency;

[0063] According to the word frequency-inverse file frequency of each word in the text data, each keyword corresponding to the conversation data is screened out from multiple words.

[0064] In an embodiment of the present invention, after obtaining multiple conversation data in a preset time period, each of the multiple conversation data is parsed, and corresponding text data, namely, conversation text, is extracted from each conversation data to perform keyword recognition on the text data.

[0065] After parsing the text data from the conversation data, the text data can be preprocessed, specifically including at least one of cleaning, deduplication, and formatting the text data. For example, deduplication of text data includes: detecting whether there is text data that is exactly the same as the text data in multiple text data, and if so, deleting the duplicate text data to update the multiple text data, ensuring the uniqueness of each text data in the multiple text data, thereby improving the efficiency and accuracy of data processing. For example, formatting text data can be converting text data according to a preset format, such as converting different upper and lower cases in text data to uniform upper and lower cases, uniform units, etc. Cleaning text data includes removing invalid characters in text data, such as removing spaces, etc. Preprocessing text data can facilitate subsequent analysis and storage of text data.

[0066] In an embodiment of the present invention, for each piece of text data, the text data is processed using natural language processing technology, such as using the Jieba word segmentation tool to perform word segmentation processing on the text data, and multiple words corresponding to each piece of data are obtained. Then, for multiple words, based on the TF-IDF (Term Frequency-Inverse Document Frequency) model, the frequency of occurrence of the words in the text data is calculated to determine each keyword according to the final TF-IDF value. For example, the calculation formula of the term frequency-inverse document frequency TF-IDF of each word in the text data is: TF-IDF=TF·IDF, where TF is determined according to the term frequency of the word in the text data, the greater the term frequency, the greater the TF value, and IDF is the importance of the text data and the format of the text data containing a certain word. By calculating the term frequency-inverse text frequency, i.e., TF-IDF value, of each word in the text data, and sorting them in order from high to low according to the TF-IDF value, the top N words in the sorting are selected according to the sorting result as the keywords corresponding to the text data, where N is a positive integer, or the words with a TF-IDF value higher than a preset value are used as the keywords corresponding to the text data. For example, the keyword "cold" of the disease can be obtained based on the text data area. For example, in the text data: "I feel a little uncomfortable. It may be a runny nose caused by a cold. I need a doctor to help me see if I have influenza." It can be found that "cold" appears many times in the text data. By calculating the TF-IDF value, it is found that the word is more important. Therefore, the word "cold" is determined to be the keyword of the text data.

[0067] In the embodiment of the present invention, before clustering the keywords corresponding to the plurality of conversation data, the method further includes: for each keyword corresponding to each conversation data, calculating the matching degree between the keyword and the conversation data; and deleting the keywords whose matching degree is lower than a preset matching degree threshold.

[0068] In an embodiment of the present invention, before clustering the keywords corresponding to the multiple conversation data, it is necessary to further screen the keywords, that is, for each keyword in the keywords corresponding to each conversation data, calculate the matching degree between the keyword and the conversation data, such as using the BM25 (Best Matching 25) algorithm to calculate the matching degree or relevance between the keyword and the conversation data. If the matching degree is lower than the preset matching degree threshold, the keyword can be deleted from the keywords, thereby improving the clustering accuracy and efficiency. Among them, the BM25 algorithm is a relevance scoring algorithm used in the field of information retrieval (IR).

[0069] In the embodiment of the present invention, after identifying each keyword corresponding to each corresponding data, the keywords corresponding to the multiple conversation data are clustered to obtain the category to which each keyword belongs. The keywords corresponding to the multiple conversation data can constitute a keyword set. In order to facilitate clustering of the keywords in the keyword set, the keywords in the keyword set are preprocessed before clustering to improve the efficiency and accuracy of clustering.

[0070] In an embodiment of the present invention, before clustering the keywords corresponding to the multiple dialogue data, the method further includes: preprocessing the keywords corresponding to each dialogue data. Specifically, for each keyword corresponding to each dialogue data, i.e., each text data, each keyword corresponding to the dialogue data is vectorized to obtain a word vector corresponding to each keyword, and each word vector is subjected to data cleaning, data conversion, and data dimensionality reduction processing. Specifically, data cleaning includes processing missing values, abnormal values, or erroneous data, such as filling missing values, deleting abnormal values, or performing data error correction to achieve data cleaning. Data conversion includes data standardization or normalization, such as converting the data into a standard normal distribution with a mean of 0 and a standard deviation of 1 by subtracting the mean and then dividing by the standard deviation. Then, the word vector is subjected to data dimensionality reduction processing to reduce the data dimension, reduce the computational complexity, and eliminate redundant information, so as to facilitate clustering.

[0071] In an embodiment of the present invention, before clustering each keyword in a plurality of conversation data, each keyword is imported into a database, so as to perform exploratory analysis on each keyword through a clustering algorithm, discover potential patterns, structures and relationships in the data, and self-classify, i.e., cluster, the keywords. The database may be an ElasticSearch database. The ES database is an open source distributed search and analysis engine that provides efficient full-text search, real-time analysis and data visualization functions. It can be used in scenarios such as full-text search, log analysis, security information and business analysis, and can be used to store, search and analyze large amounts of data. It provides a powerful RESTful API and rich query functions, and can be used to build various types of applications.

[0072] In the embodiment of the present invention, clustering is performed on each keyword corresponding to the plurality of conversation data to obtain the category to which each keyword belongs, including:

[0073] Initialize multiple cluster centers;

[0074] Perform the following clustering process: for each keyword, calculate the distance between the keyword and each cluster center, add the keyword to the cluster corresponding to the cluster center with the smallest distance, so as to obtain multiple clusters, each cluster corresponding to a category; update each cluster center according to each keyword in each cluster;

[0075] The clustering process is executed cyclically until the iteration ends, and the category to which each keyword belongs is obtained.

[0076] In an embodiment of the present invention, a clustering algorithm may be used to cluster the keywords corresponding to the multiple conversation data. The clustering algorithm is an unsupervised learning algorithm that aims to group or cluster objects in a data set into clusters with similar characteristics. Its main goal is to make the data points in the same cluster more similar, while the data points between different clusters are less similar. The clustering algorithm can help explore the hidden structures and patterns in the data without the need to label the training data in advance. The clustering algorithm may be at least one of a K-means clustering algorithm, a K-medoids algorithm, a BRICH algorithm, a hierarchical clustering algorithm, and the like.

[0077] Take the K-means clustering algorithm as an example. Figure 2 As shown, first read the keyword set and initialize multiple cluster centers. The number of multiple cluster centers can be customized according to business needs. For example, when building a medical knowledge base, three cluster centers can be set according to diseases, prescriptions and departments; for each keyword in the keyword set, calculate the distance between each keyword and each cluster center, and then select the cluster center with the smallest distance, and divide the keyword into the cluster corresponding to the cluster center with the smallest distance. After traversing each keyword, multiple clusters and each keyword corresponding to each cluster are obtained; then calculate the cluster center of each cluster according to each keyword corresponding to the cluster, update each cluster center, and judge whether the cluster center converges, that is, judge whether the distance between the updated cluster center of each cluster and the previous cluster center of the cluster is less than or equal to the preset distance threshold. If not, the clustering process is cyclically executed according to each updated cluster center until the cluster center converges or reaches a preset number of iterations, the iteration ends, and each cluster and each keyword corresponding to each cluster are obtained, thereby obtaining the category to which each keyword belongs.

[0078] When calculating the distance between the keyword and the cluster center, a similarity algorithm can be used. Specifically, the first vector corresponding to the keyword and the second vector corresponding to the cluster center are similarly calculated. The higher the similarity, the smaller the distance. The similarity algorithm can be selected from cosine similarity, Euclidean distance, Manhattan distance, etc. The first vector and the second vector can be realized by algorithms such as TF-IDF, supervised DSSM (Deep Structured Semantic Model) and unsupervised sentence2vector.

[0079] When calculating the cluster center of each cluster according to the keywords corresponding to the cluster, for each keyword corresponding to the cluster, the average value of each first vector is calculated according to the first vectors corresponding to the keywords, and the average value is used as the updated cluster center of the cluster.

[0080] In the embodiment of the present invention, Figure 3 As shown, each keyword is detected for anomalies according to the preset conditions of the category, including:

[0081] Step S301: Obtain a category dictionary corresponding to a category, and match the keyword with the category dictionary;

[0082] Step S302: In response to the keyword failing to match the category dictionary, the keyword is determined to be an abnormal keyword.

[0083] In an embodiment of the present invention, after obtaining each cluster and each keyword corresponding to each cluster, that is, after obtaining the category to which each keyword belongs, it is necessary to perform anomaly detection on the keywords of each category to prevent erroneous information from appearing in the knowledge base. Specifically, the preset condition of the category can be a category dictionary corresponding to each category, such as a category is a prescription, and the category dictionary is a prescription dictionary. By matching each keyword in each keyword corresponding to each category with the category dictionary, anomaly detection is performed on each keyword corresponding to each category. If the keyword exists in the category dictionary, that is, there is a word in the category dictionary that is exactly the same as the keyword, the match is successful. If they are not exactly the same, the match fails, and the keyword is an abnormal keyword.

[0084] In an embodiment of the present invention, the detected abnormal keywords are corrected. The edit distance between the abnormal keywords and each word in the category dictionary can be calculated according to the edit distance algorithm, and the target word with the smallest edit distance can be determined. The target word can be used to replace the abnormal keywords to achieve the correction of the abnormal keywords. Further, it can also be determined whether the edit distance between the abnormal keywords and the target word is less than or equal to a preset distance threshold. If so, the abnormal keywords are replaced with the target word, otherwise, the abnormal keywords are deleted.

[0085] In an embodiment of the present invention, correcting the detected abnormal keywords includes:

[0086] In response to a failure of matching the keyword with the category dictionary, generating a plurality of candidate words corresponding to the keyword;

[0087] For each candidate word, the candidate word is matched with the category dictionary, and the keyword is replaced with the successfully matched candidate word.

[0088] In an embodiment of the present invention, when abnormal keywords are detected, the abnormal keywords need to be corrected to prevent erroneous information from appearing in the knowledge base, affecting the accuracy of the knowledge base, and thus reducing the retrieval accuracy. Specifically, if the keyword fails to match the category dictionary, the keyword is an abnormal keyword. Based on the abnormal keyword, a corresponding plurality of candidate words are generated, and the candidate words can be obtained by editing one or more characters in the abnormal keyword. After generating multiple candidate words, each candidate word is matched with the category dictionary, that is, the degree of similarity between the candidate word and each word in the category dictionary is calculated, such as the edit distance between each candidate word and each word in the category dictionary can be calculated according to the edit distance algorithm. If the edit distance is 0, the match is successful; then, the abnormal keyword is replaced with the successfully matched candidate word to correct the abnormal keyword and improve the accuracy of the knowledge base.

[0089] In an embodiment of the present invention, generating multiple candidate words corresponding to keywords includes: generating multiple candidate words corresponding to keywords according to an edit distance algorithm. Among them, the edit distance algorithm is an algorithm for calculating the similarity between two strings, and is usually used to measure the minimum number of steps required to convert one string into another string through operations such as insertion, deletion, and substitution. The edit distance algorithm mainly has two common forms: Levenshtein distance and Damerau-Levenshtein distance. For example, the abnormal keyword is "aspirin", and the Levenshtein distance between it and "aspirin" is 1, indicating that only one editing operation is required to convert one string into another.

[0090] Specifically, multiple candidate words whose edit distance with the keyword is less than or equal to a preset distance threshold are generated according to the edit distance algorithm. For example, the preset distance threshold may be 1. Multiple candidate words can be obtained by replacing any word in the keyword with a word with the same or similar pronunciation.

[0091] After correcting the detected abnormal keywords, the keywords corresponding to each category are updated, and then a knowledge base is built based on each category and its corresponding keywords, so as to achieve efficient retrieval through ES.

[0092] The knowledge base construction device of the embodiment of the present invention first obtains multiple conversation data of a preset time period, and then identifies each keyword corresponding to each corresponding data; then clusters each keyword corresponding to the multiple conversation data to obtain the category to which each keyword belongs; for each keyword corresponding to each category, uses the preset conditions of the category to perform abnormal detection on each keyword, and corrects the detected abnormal keywords, and updates each keyword corresponding to each category, so as to construct a knowledge base according to each category and its corresponding keywords. This method can realize the self-construction of the knowledge base and the real-time update of the knowledge base; by performing abnormal detection and correction on keywords, the accuracy of the knowledge base is improved, and then the retrieval efficiency and accuracy are improved, and the accessibility and availability of information are improved.

[0093] According to another aspect of the embodiment of the present invention, Figure 4 As shown, a knowledge base construction device 400 is provided, comprising:

[0094] An acquisition module 401 acquires a plurality of conversation data in a preset time period, and identifies each keyword corresponding to the conversation data from each conversation data;

[0095] A clustering module 402 clusters the keywords corresponding to the plurality of conversation data to obtain the category to which each keyword belongs;

[0096] The detection module 403 performs abnormal detection on each keyword corresponding to each category according to the preset conditions of the category, and corrects the detected abnormal keywords, thereby updating each keyword corresponding to the category;

[0097] The construction module 404 constructs a knowledge base according to each category and its corresponding keywords.

[0098] In the embodiment of the present invention, the detection module 403 is further used to: obtain a category dictionary corresponding to the category, and match the keyword with the category dictionary; in response to a failure in matching the keyword with the category dictionary, determine that the keyword is an abnormal keyword.

[0099] In the embodiment of the present invention, the detection module 403 is further used to: in response to the failure of matching the keyword with the category dictionary, generate multiple candidate words corresponding to the keyword; for each candidate word, match the candidate word with the category dictionary, and replace the keyword with the successfully matched candidate word.

[0100] In the embodiment of the present invention, the detection module 403 is further configured to generate a plurality of candidate words corresponding to the keyword according to an edit distance algorithm.

[0101] In the embodiment of the present invention, the acquisition module 401 is further used to: for each conversation data, parse text data from the conversation data; use natural language processing technology to process the text data to obtain multiple words corresponding to each text data; for each word, calculate the word frequency-reverse file frequency of the word in the text data; according to the word frequency-reverse file frequency of each word in the text data, filter out each keyword corresponding to the conversation data from the multiple words.

[0102] In the embodiment of the present invention, the clustering module 402 is further used to: before clustering the keywords corresponding to the multiple conversation data, for each keyword corresponding to each conversation data, calculate the matching degree between the keyword and the conversation data; and delete the keywords whose matching degree is lower than a preset matching degree threshold.

[0103] In an embodiment of the present invention, the clustering module 402 is further used to: initialize multiple cluster centers; perform the following clustering process: for each keyword, calculate the distance between the keyword and each cluster center, add the keyword to the cluster corresponding to the cluster center with the smallest distance, thereby obtaining multiple clusters, each cluster corresponding to a category; update each cluster center according to each keyword in each cluster; and execute the clustering process in a loop until the iteration ends to obtain the category to which each keyword belongs.

[0104] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the knowledge base construction method provided by the present invention.

[0105] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the knowledge base construction method provided by the present invention is implemented.

[0106] Figure 5 An exemplary system architecture 500 is shown to which the knowledge base construction method or knowledge base construction apparatus according to the embodiment of the present invention can be applied.

[0107] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0108] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0109] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0110] The server 505 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0111] It should be noted that the knowledge base construction method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the knowledge base construction device is generally set in the server 505 .

[0112] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0113] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0114] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0116] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0117] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, they may be described as: a processor includes an acquisition module, a clustering module, a detection module, and a construction module. The names of these modules do not constitute a limitation on the modules themselves in some cases. For example, the acquisition module may also be described as "a module for acquiring multiple conversation data in a preset time period and identifying each keyword corresponding to the conversation data from each conversation data".

[0120] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining multiple conversation data of a preset time period, identifying each keyword corresponding to the conversation data from each conversation data; clustering each keyword corresponding to the multiple conversation data to obtain the category to which each keyword belongs; for each keyword corresponding to each category, performing anomaly detection on each keyword according to the preset conditions of the category, and correcting the detected abnormal keywords, thereby updating each keyword corresponding to the category; and building a knowledge base according to each category and its corresponding keywords.

[0121] According to the technical solution of the embodiment of the present invention, the knowledge base construction device of the embodiment of the present invention first obtains multiple conversation data of a preset time period, and then identifies each keyword corresponding to each corresponding data; then clusters each keyword corresponding to the multiple conversation data to obtain the category to which each keyword belongs; for each keyword corresponding to each category, uses the preset conditions of the category to perform abnormal detection on each keyword, and corrects the detected abnormal keywords, and updates each keyword corresponding to each category, so as to construct a knowledge base according to each category and its corresponding keywords. This method can realize the self-construction of the knowledge base and the real-time update of the knowledge base; by performing abnormal detection and correction on keywords, the accuracy of the knowledge base is improved, and then the retrieval efficiency and accuracy are improved, and the accessibility and availability of information are improved.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A knowledge base construction method, It is characterized in that include: Acquire multiple conversation data in a preset time period, and identify each keyword corresponding to the conversation data from each conversation data; Clustering the keywords corresponding to the plurality of conversation data to obtain a category to which each keyword belongs; For each keyword corresponding to each category, perform abnormality detection on each keyword according to preset conditions of the category, and correct the detected abnormal keywords, thereby updating each keyword corresponding to the category; A knowledge base is constructed based on each category and its corresponding keywords.

2. The method according to claim 1, It is characterized in that Performing abnormality detection on each of the keywords according to the preset conditions of the category includes: Obtaining a category dictionary corresponding to the category, and matching the keyword with the category dictionary; In response to the keyword failing to match the category dictionary, the keyword is determined to be an abnormal keyword.

3. The method according to claim 2, It is characterized in that Correct the detected abnormal keywords, including: In response to the keyword failing to match the category dictionary, generating a plurality of candidate words corresponding to the keyword; For each candidate word, the candidate word is matched with the category dictionary, and the keyword is replaced with the successfully matched candidate word.

4. The method according to claim 3, It is characterized in that Generate multiple candidate words corresponding to the keyword, including: A plurality of candidate words corresponding to the keyword are generated according to an edit distance algorithm.

5. The method according to claim 1, It is characterized in that Identify each keyword corresponding to each conversation data from each conversation data, including: For each piece of conversation data, parse text data from the conversation data; Processing the text data using natural language processing technology to obtain multiple words corresponding to each piece of text data; For each word, calculating the word frequency-inverse document frequency of the word in the text data; According to the word frequency-inverse document frequency of each word in the text data, each keyword corresponding to the conversation data is screened out from the multiple words.

6. The method according to claim 1, It is characterized in that Before clustering the keywords corresponding to the plurality of conversation data, the method further includes: For each keyword corresponding to each piece of conversation data, calculating the matching degree between the keyword and the conversation data; Delete keywords whose matching degree is lower than the preset matching degree threshold.

7. The method according to claim 1, It is characterized in that Clustering each keyword corresponding to the plurality of conversation data to obtain a category to which each keyword belongs includes: Initialize multiple cluster centers; Perform the following clustering process: for each keyword, calculate the distance between the keyword and each cluster center, add the keyword to the cluster corresponding to the cluster center with the smallest distance, so as to obtain multiple clusters, each cluster corresponding to a category; update each cluster center according to each keyword in each cluster; The clustering process is executed cyclically until the iteration ends, and the category to which each keyword belongs is obtained.

8. A knowledge base construction device, It is characterized in that include: An acquisition module, which acquires a plurality of conversation data in a preset time period, and identifies each keyword corresponding to the conversation data from each conversation data; A clustering module, clustering each keyword corresponding to the plurality of conversation data to obtain a category to which each keyword belongs; A detection module, for each keyword corresponding to each category, performs abnormal detection on the keywords according to preset conditions of the category, and corrects the detected abnormal keywords, thereby updating the keywords corresponding to the category; The building module builds a knowledge base based on each category and its corresponding keywords.

9. An electronic device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Question-answer pair knowledge base construction method, device and equipment and storage medium

    CN112650843A

  • Text error correction method, electronic equipment and storage medium

    CN113361266A

  • Question and answer method and system with traceability based on multi-source knowledge base

    CN117874190A