Data processing method and device, data query method and device

Abstract

The embodiments of this specification provide a data processing method and device, wherein the data processing method is applied to a data management platform, including preprocessing a plurality of acquired text data, and dividing the plurality of text data into at least one type based on the similarity between the plurality of preprocessed text data; determining a target question, a target answer, and an extended question associated with the target question for each type of text data; determining a database corresponding to the target question on a data storage platform, and storing the target question, the target answer, and the extended question in the corresponding database; obtaining the same target question in all databases, and sharing different extended questions associated with the same target question in the database.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data query method, a data processing device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the popularity of intelligent chatbots, more and more project scenarios use chatbots to answer customers' pre-sales and after-sales questions to improve purchase conversion rates and user satisfaction. In chatbot applications, the knowledge base is an important module. The degree of structure and content quality of the knowledge base determine the accuracy of user demand retrieval and the quality of answers. However, with the increase in project scenarios, the knowledge base will become large and complex, making it difficult to maintain in the future. Summary of the invention

[0003] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data query method, a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a data management platform, including:

[0005] Preprocessing the acquired multiple text data, and classifying the multiple text data into at least one type based on similarities between the preprocessed multiple text data;

[0006] Determining a target question, a target answer, and an extended question associated with the target question for each type of text data;

[0007] Determine a database corresponding to the target question in a data storage platform, and store the target question, the target answer, and the extended question in the corresponding database;

[0008] The same target problem in all databases is obtained, and different extended problems associated with the same target problem are shared among the databases.

[0009] According to a second aspect of an embodiment of this specification, a data query method is provided, which is applied to a data query platform, including:

[0010] Receive a data query request containing text questions sent by a client;

[0011] Preprocessing the text question based on the data query request to determine a database corresponding to the text question in a data storage platform;

[0012] Acquire multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers, wherein the data storage platform is the data storage platform in claim 1.

[0013] According to a third aspect of an embodiment of this specification, there is provided a data processing device, including:

[0014] A preprocessing module, configured to preprocess the acquired multiple text data, and classify the multiple text data into at least one type based on the similarity between the preprocessed multiple text data;

[0015] A question determination module configured to determine a target question, a target answer, and an extended question associated with the target question for each type of text data;

[0016] A storage module is configured to determine a database corresponding to the target question in a data storage platform, and store the target question, the target answer and the extended question in the corresponding database;

[0017] The sharing module is configured to obtain the same target question in all databases, and share different extended questions associated with the same target question in the database.

[0018] According to a fourth aspect of the embodiments of this specification, a data query device is provided, including:

[0019] A request receiving module is configured to receive a data query request carrying a text question sent by a client;

[0020] A preprocessing module is configured to preprocess the text question based on the data query request and determine a database corresponding to the text question in a data storage platform;

[0021] The answer determination module is configured to obtain multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers, wherein the data storage platform is the data storage platform in the above-mentioned data processing method.

[0022] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0023] Memory and processor;

[0024] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned data processing method or the above-mentioned data query method are implemented.

[0025] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above-mentioned data processing method or the above-mentioned data query method are implemented.

[0026] An embodiment of the present specification implements a data processing method and device, wherein the data processing method is applied to a data management platform, including preprocessing a plurality of acquired text data, and dividing the plurality of text data into at least one type based on the similarity between the plurality of preprocessed text data; determining a target question, a target answer, and an extended question associated with the target question for each type of text data; determining a database corresponding to the target question in a data storage platform, and storing the target question, the target answer, and the extended question in the corresponding database; obtaining the same target question in all databases, and sharing different extended questions associated with the same target question in the database. Specifically, the data processing method processes a plurality of text data in different fields, stores the processed plurality of text data in different databases according to the target question, and diffusely clones the different extended questions in the same target question in all databases to other databases, thereby ensuring unified data management and reducing subsequent maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of a specific application framework of a data processing method provided by an embodiment of this specification;

[0028] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0029] Figure 3 is a flow chart of a data query method provided by an embodiment of this specification;

[0030] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0031] Figure 5 It is a structural schematic diagram of a data query device provided by an embodiment of this specification;

[0032] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0033] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0034] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0036] First, the terms involved in one or more embodiments of this specification are explained.

[0037] Knowledge base: A collection of questions and answers about project knowledge points, serving the chatbot conversation process.

[0038] Language model: A model used to calculate sentence probabilities. The results calculated using the language model can be used to determine the fluency of a sentence. The smaller the calculated probability, the less likely a sentence is to appear.

[0039] Text generation: automatic writing of text.

[0040] Field: The project scenario is considered as a single field, where the project scenario can be understood as life insurance, pension insurance, etc. in the insurance field.

[0041] Structuring: A form of data standardization that allows information to be represented using data or a uniform structure, such as numbers, symbols, and text elements.

[0042] In this specification, a data processing method is provided. One or more embodiments of this specification also relate to a data query method, a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0043] In the prior art, in the knowledge base serving the intelligent chatbot, similar questions under the same knowledge point need to be repeatedly added due to different project fields; and similar questions about the same knowledge point in different project fields are asked in different ways and there is no unified management; in the later stage, when performing question and answer retrieval based on the knowledge base, only single-threaded full-database recall can be performed, and the efficiency and accuracy are not high.

[0044] See also Figure 1 , Figure 1 A specific application framework schematic diagram of a data processing method provided according to an embodiment of the present specification is shown.

[0045] Figure 1 It includes a knowledge factory 102, a knowledge base 104, a retrieval center 106 and an interactive interface 108; wherein the knowledge factory 102 can be understood as the data management platform in the following embodiments, the knowledge base 104 can be understood as the data storage platform in the following embodiments, the retrieval center 106 can be understood as the data query platform in the following embodiments, and the interactive interface 108 can be understood as the user interactive interface of the client in the following embodiments.

[0046] Specifically, the knowledge factory 102 includes a data center, a quality inspection module, and a management module. The data center includes two parts: an online data source and an offline data source. The online data source includes the conversation content between the user and the chat robot, the text comments posted by the user, and other user interaction text information. The offline data source includes text information obtained from public data sets (such as public articles) and related cooperative content producers. The quality inspection module performs quality inspection on the data in the data center, and controls the data and produces knowledge points through rule filtering, sentence scoring, sentiment analysis, and knowledge clustering. Especially for financial field knowledge, the information transmitted by C-end (C-end corresponds to individuals) users requires extremely high quality control. By controlling the quality of the data through the quality module, manual review can be avoided, labor costs can be reduced, and data security can be guaranteed. The management module performs field identification, knowledge point identification, knowledge point expansion, and multi-library dissemination on the data processed by the quality inspection module.

[0047] Among them, rule filtering in the quality inspection module can be understood as filtering out corpus that does not meet project expectations through pre-defined rules, such as garbled data, too short text, sentences that meet specific syntactic structures or have overly complex syntactic components, etc.; sentence scoring can be understood as scoring sentences through a language model, and filtering out data with incoherent sentences by setting a score threshold; sentiment analysis can be understood as analyzing the sentiment of the text, determining the positive and negative emotions of the text corpus, and detecting sensitive words. When returning data for negative emotions and sensitive words, it is necessary to control and remove some negative data that does not meet project expectations and data containing sensitive words; knowledge clustering can be understood as aggregating texts with similar semantics and summarizing knowledge points. For example, extract question-answer pairs in the text, aggregate semantically similar question-answer pairs, and then extract standard questions and standard answers for the aggregated question-answer pairs.

[0048] Domain identification in the management module can be understood as identifying the domains included in the knowledge points, such as the pension insurance domain, health insurance domain, etc., through text classification, keyword extraction, etc.; knowledge point identification can be understood as judging whether the knowledge points (such as standard questions) summarized in the quality inspection module exist in the corresponding domain through similarity analysis. If they already exist, the knowledge points are linked; if not, new knowledge points are added; knowledge point expansion can be understood as expanding the similar questions of knowledge points through text generation to improve the richness of data, and combining sentiment analysis technology to generate controllable text content and improve the quality of generated text; multi-database propagation can be understood as when managing hierarchical knowledge bases, the same knowledge point will have different similar questions. If the similar questions of the same knowledge point are different in different fields, problems will be encountered in the later recall and fine sorting operations in the retrieval center. The role of multi-database propagation is to diffusely clone different similar questions under the same knowledge point to other domain knowledge bases to ensure unified data management.

[0049] The structure of the knowledge base 104 is stored in a hierarchical form, and the knowledge points are returned from the knowledge factory 102. In order to meet the needs of customers consulting in different project scenarios, the knowledge base manages the knowledge points in each project field in a hierarchical manner. The main structure includes a platform knowledge base, a domain knowledge base, and knowledge points. In a specific application scenario, the domain knowledge base can be further subdivided as needed. For example, the domain knowledge base can be a financial domain knowledge base, a communication domain knowledge base, an insurance domain knowledge base, a banking domain knowledge base in the financial domain knowledge base, and a pension insurance knowledge base, a health insurance knowledge base, etc. in the insurance domain knowledge base.

[0050] The platform knowledge base in the knowledge base 104 can be understood as the platform entrance. There are multiple domain knowledge bases under the platform knowledge base, and each domain knowledge base has multiple knowledge points. Each knowledge point contains multiple question methods related to the knowledge point. The domain knowledge base includes a collection of questions in different project fields. The knowledge points stored in the domain knowledge base do not need to carry project keywords, such as pension, health and other keywords. Different domain knowledge bases may contain the same knowledge points, such as Figure 1 The domain knowledge base 1 and the domain knowledge base 2 in the domain knowledge base 104 both contain the same knowledge point A. In practical applications, the set of similar questions under the same knowledge point in the domain knowledge base remains consistent. The knowledge point is the smallest unit stored in the knowledge base 104. Under the same knowledge point, there will be different similar questions. When searching for user questions later, similar questions in specific domain knowledge points will be recalled according to the search conditions. After fine sorting, the processed knowledge points are output to the user, and the knowledge point storage does not need to carry project keywords.

[0051] The search center 106 mainly interacts with the user interface 108 and the knowledge base 104. The search center 106 receives questions sent by users from the user interface 108, searches the knowledge base 104, and returns the processed results (i.e., answers) to the users. The search center 106 mainly includes the following six submodules: domain identification, multi-database recall, question and answer cleaning, knowledge sorting, weight adjustment, and knowledge disambiguation. Among them, the role of the domain identification submodule is to identify the project domain, and assign the user's consultation questions to the appropriate domain through text classification, keyword matching, etc.; the role of the multi-database recall submodule is to recall similar questions from the corresponding domain knowledge base if the specific domain is accurately identified in the domain identification submodule, and recall similar questions from all knowledge bases if the specific domain is not identified. Because the knowledge base has been layered, the multi-database recall submodule can be operated in parallel to improve the recall efficiency; the question and answer cleaning submodule mainly performs pre-processing on the questions consulted by users, including removing special characters, synonym replacement, punctuation removal, dirty data filtering, etc.; the knowledge sorting submodule performs refined sorting on the answers recalled from the knowledge base, calculates the semantic similarity between the user questions processed by the question and answer cleaning submodule and the recalled answers, and outputs the similarity score; the weight adjustment submodule may have multiple matching answers for the questions consulted by users. However, in actual applications, merchants will strategically choose the words when conducting conversations, that is, they are more inclined to certain answers, or they are less willing to show certain answers to users. The function of the weight adjustment submodule is to perform post-processing adjustments on the results of different precision ranking submodules. The methods used can include keyword matching, rule filtering, etc.; in the knowledge disambiguation submodule, different knowledge bases may contain the same knowledge points, and disambiguation operations need to be performed when returning them to users for display. The main disambiguation method is to bring the library name, such as "library name-knowledge point". For example, if a user asks what insurance is, then health insurance and pension insurance may give standard answers. In order to improve the user experience, the names of their respective databases and standard answers are returned to the user: health insurance field-insurance is *** and pension insurance field-insurance is ***, etc.

[0052] The interactive interface 108 is mainly an entry for users to ask questions, and is mainly divided into a platform entry and an entry for each project field. Users enter from different entrances, and even if they ask the same question, they will get different answers belonging to the current entrance, so as to increase the personalization capability of the embodiment of this specification and improve the user experience.

[0053] In the embodiments of this specification, the "knowledge factory" module is carefully designed, which can automatically perform quality inspection and knowledge reflux on data sources from different data sources. It supports the dissemination of similar questions in multiple libraries to ensure the unity of the knowledge base; in the "knowledge base" module, through the hierarchical design of the knowledge base, it can meet the management of the same knowledge points in different project fields and reduce the cost of later maintenance; in the "retrieval center" module, through the functional points such as field identification, multi-library recall, knowledge disambiguation, etc., it can support parallel indexing from knowledge bases in various fields to improve retrieval integrity and retrieval efficiency; the embodiments of this specification integrate the various technical points of structured questions and answers into an integrated intelligent solution, and also support plug-in calls for each module, so that the solution can be flexibly transplanted to various projects.

[0054] See also Figure 2 , Figure 2 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, wherein the data processing method is applied to a data management platform and specifically includes the following steps.

[0055] Step 202: pre-processing the acquired multiple text data, and classifying the multiple text data into at least one type based on the similarity between the pre-processed multiple text data.

[0056] Among them, the data management platform can be understood as the above-mentioned knowledge factory.

[0057] Specifically, before the preprocessing of the acquired multiple text data, the method further includes:

[0058] A plurality of online text data and a plurality of offline text data are acquired, and a plurality of text data are formed based on the plurality of online text data and the plurality of offline text data.

[0059] Among them, online text data can be understood as the conversation content between users and chatbots, text comments posted by users and other user interaction information. Offline text data can be understood as information obtained from public data sets, relevant cooperative content producers and other channels, such as articles, content introductions, etc. Moreover, online text data and offline text data are not limited to text data in any field, such as insurance, transportation, and banking.

[0060] In the embodiments of this specification, before performing data partitioning, online and offline text data in different fields are obtained to increase the richness of the text data fields and contents, so that subsequent databases can contain rich text data.

[0061] In a specific implementation, the preprocessing of the acquired multiple text data includes:

[0062] Perform rule filtering on the acquired multiple text data according to preset filtering rules;

[0063] Perform sentence filtering on the multiple text data after being filtered by the rule according to the sentence scoring rule;

[0064] Sentiment filtering is performed on multiple text data after sentence filtering according to sentiment analysis rules.

[0065] Among them, the preset filtering rules, sentence scoring rules and sentiment analysis rules can be set according to actual needs, and the embodiments of this specification do not impose any limitations on this.

[0066] Specifically, the preset filtering rules can be understood as garbled data filtering rules, too short text filtering rules, too long text filtering rules, specific syntactic structure filtering rules or syntactic component filtering rules, etc.

[0067] The multiple text data obtained are filtered according to preset filtering rules. It can be understood that text data containing multiple garbled data among the multiple text data obtained are deleted, text data that is too short is deleted, text data that is too long is deleted, text data that conforms to a specific syntactic structure is deleted, and / or text data with too complex syntactic structure is deleted.

[0068] For example, text data containing 30 garbled data among the multiple text data obtained are deleted, text data with less than 3 characters are deleted, text data with more than 100 characters are deleted, text data containing formulas are deleted, and / or text data containing special symbols are deleted, etc.

[0069] In the embodiments of this specification, the quality of text data is controlled by pre-setting filtering rules, sentence scoring rules, and sentiment analysis rules to ensure the quality of text data subsequently stored in the database.

[0070] In addition, the sentence filtering of the plurality of text data after the rule filtering according to the sentence scoring rule includes:

[0071] Inputting the multiple text data after filtering by the rule into the language model to obtain a sentence score of each text data in the multiple text data after filtering by the rule;

[0072] From the multiple text data after rule filtering, delete the text data whose sentence scores are less than or equal to a preset score threshold.

[0073] Among them, sentence filtering is performed on the multiple text data after passing the rule filtering according to the sentence scoring rule, which can be understood as sentence filtering is performed on the multiple text data after passing the rule filtering through a pre-trained language model.

[0074] Specifically, each text data among the multiple text data after filtering by the rule can be input into the language model to obtain the sentence score of each text data among the multiple text data after filtering by the rule, and the text data with a sentence score less than or equal to a preset score threshold can be deleted from the multiple text data after filtering by the rule, wherein the preset score threshold can be set according to the actual application, for example, the preset score threshold can be set to 50, 60, etc.

[0075] In the embodiments of the present specification, each text data in the multiple text data filtered by the rules is scored through the language model to achieve the score of each text data, which can quickly filter out text data with incoherent sentences and ensure the quality of the text data stored in the database later.

[0076] Furthermore, the sentiment analysis rules are used to filter the plurality of text data after the sentence filtering, including:

[0077] Obtain the corpus sentiment and sensitive words of each text data in the multiple text data filtered by the sentence;

[0078] From the multiple text data after sentence filtering, delete the text data whose corpus emotions meet the preset emotion threshold or contain sensitive words.

[0079] Among them, the preset emotion threshold and sensitive words can be set according to actual applications, and this manual does not impose any limitations on this.

[0080] Specifically, after filtering the multiple text data after the rule filtering through the language model, in order to further improve the quality of the text data, the corpus sentiment and sensitive words of each text data in the multiple text data after the sentence filtering are obtained, and the text data whose corpus sentiment meets the preset sentiment threshold or contains sensitive words are deleted from the multiple text data after the sentence filtering. The corpus sentiment can be obtained through the pre-trained expected emotion recognition model, and the sensitive words can be set according to the actual project requirements, for example, some violent and colorful words are set as sensitive words.

[0081] In practical applications, the sentiment of each text data in the multiple text data filtered by the sentence is first analyzed to determine the positive or negative sentiment of each text data, and the text data containing negative sentiment is deleted; at the same time, the sensitive words of each text data in the multiple text data filtered by the sentence are extracted, and the text data containing sensitive words are deleted. Among them, the text data containing negative sentiment can be understood as text data containing negative data that does not meet the expectations of the project. For example, if the project is an insurance project and the text data is "this insurance project is particularly bad", then the text data is text data containing negative sentiment.

[0082] In the embodiments of the present specification, sentiment filtering is performed on multiple text data after sentence filtering using corpus sentiment and sensitive words, and text data containing negative sentiment and sensitive words in the sentences are filtered out to ensure the quality of the text data stored in the database later.

[0083] In a specific implementation, after preprocessing the acquired multiple text data, the multiple text data are divided into at least one type based on the similarity between the preprocessed multiple text data, that is, the preprocessed multiple text data are aggregated according to the semantic similarity, and the knowledge points are summarized.

[0084] For example, text data includes: transfer to manual service, consult manual customer service, help contact manual service, and what is insurance. At this time, after aggregation according to the semantic similarity program, transfer to manual service, consult manual customer service, and help contact manual service can be aggregated together and divided into one type, and the knowledge points are summarized, that is, the knowledge point is manual customer service consultation.

[0085] In practical applications, question-answer pairs can be extracted from text data, and then semantically similar question-answer pairs can be aggregated together. The knowledge points of each category of aggregated text data can be summarized, that is, a standard question can be summarized based on each category of text data, and a standard answer can be summarized based on the standard question and other text questions similar to the standard question.

[0086] Step 204: Determine a target question, a target answer, and an extended question associated with the target question for each type of text data.

[0087] Specifically, each type of text data may be summarized to determine a target question, a target answer, and an extended question associated with the target question (eg, an extended question similar to the target question) for each type of text data.

[0088] Step 206: Determine the database corresponding to the target question in the data storage platform, and store the target question, the target answer and the extended question in the corresponding database.

[0089] The database can be understood as a domain database, that is, each database corresponds to a domain, such as finance, insurance, transportation, etc.; or a domain under a project, for example, if the project is insurance, the domain can be understood as life insurance, children's insurance, and pension insurance, etc. The data storage platform can be understood as the knowledge base 104 in the above embodiment.

[0090] Specifically, the step of determining a database corresponding to the target problem on a data storage platform includes:

[0091] Determining the field corresponding to the target question by processing the text of the target question;

[0092] A database field corresponding to the target problem is determined based on the field corresponding to the target problem, and a database corresponding to the target problem on a data storage platform is determined based on the database field.

[0093] The target problem can be understood as the knowledge point of the above-mentioned embodiment.

[0094] In actual applications, when determining the database corresponding to the target problem on the data storage platform, the target problem for each type of text data is first determined, and then the field corresponding to the target problem is identified through text processing methods such as text classification and keyword extraction, so as to determine the database field corresponding to the target problem on the data storage platform based on the field corresponding to the target problem; finally, based on the field corresponding to the target problem of each type of text data, the field database corresponding to the target problem of each type of text data is determined.

[0095] In the embodiments of this specification, the target question can be understood as the knowledge point summarized based on each type of text data. After determining the domain database corresponding to the target question of each type of text data, the target question of each type of text data can be accurately stored in the corresponding domain database, and then the target answer corresponding to the target question can be quickly obtained from the corresponding domain database based on the user's target question and returned to the user, thereby improving the user experience.

[0096] In a specific implementation, storing the target question, the target answer, and the extended question in a corresponding database includes:

[0097] Determine whether the target question has been stored in the database corresponding to the target question.

[0098] If yes, the extended question associated with the target question is stored in the corresponding database and associated with the target question.

[0099] If not, the target question, the target answer and the extended question are stored in a corresponding database.

[0100] Specifically, in one case, the target problem may already exist in the corresponding domain database; in another case, the target problem may not exist in the corresponding domain database.

[0101] If the target question does not exist in the corresponding domain database, the target question, target answer, and extended questions associated with the target question of each type of text data can be directly stored in the corresponding domain database. If the target question already exists in the corresponding domain database, the extended questions associated with the target question can be stored in the corresponding domain database to enrich the richness of the text questions in the corresponding domain database, ensure that various text questions of users can be answered in subsequent use, and improve the user experience.

[0102] Continuing with the above example, the target question: the knowledge point for manual customer service consultation already exists in the corresponding domain database, but there is no extended question associated with the target question in the domain database: the extended question for help contacting manual service, then the extended question is stored in the domain database corresponding to the target question and associated with the target question.

[0103] In actual applications, a target question may correspond to multiple fields, and therefore multiple field databases. When a field does not have a corresponding database, a corresponding field database can be created based on the field, and then the target question is stored in the database. Similarly, when a target question corresponds to multiple fields and there are multiple field databases, the target question needs to be stored in each field database according to the storage method of the above embodiment, so that the search results can be quickly obtained from any field database during subsequent searches.

[0104] In addition, in order to improve the richness of the target questions stored in the domain database, the target questions can be expanded. The specific implementation method is as follows:

[0105] Before obtaining the same target question in all databases, the method further includes:

[0106] Generate at least one target-related question associated with the target question by using a preset text generation method;

[0107] The at least one target-associated question is stored in a database corresponding to the target question and associated with the target question.

[0108] Among them, the preset text generation method can be set according to actual needs, and this manual does not impose any limitation on this.

[0109] For example, the target question is: What is insurance? The target related questions associated with the target question generated by the preset text generation method may be: What is insurance? What is the role of insurance? And so on.

[0110] In the embodiment of this specification, at least one target-related question associated with the target question is generated by a preset text generation method; the at least one target-related question is stored in a database corresponding to the target question and associated with the target question. In this way, the target question can be expanded, the richness of the data can be improved, and the sentiment analysis technology can be combined to generate controllable target-related questions, which greatly improves the quality of generating target-related questions.

[0111] Step 208: Acquire the same target question in all databases, and share different extended questions associated with the same target question in the database.

[0112] In actual applications, in different fields, similar questions about the same knowledge point are asked in different ways. If the questions about the same knowledge point in a database are all asked in different ways, when searching later, if the user changes the way of asking, the corresponding text answer may not be found in the database. In order to solve the above problem, different text questions about the same knowledge point can be combined and copied to each database with the same knowledge point. The specific implementation method is as follows:

[0113] The step of sharing a database of different extended questions associated with the same target question includes:

[0114] Determine the expansion problem that does not exist in each target problem among the same target problems;

[0115] The extended questions that do not exist in each target question are stored in a database corresponding to each target question and associated with each target question.

[0116] For example, both database 1 and database 2 contain the target question, and the text questions of the target question in database 1 are: text question 1, text question 2, and the text questions in database 2 are: text question 3, text question 4. In this case, you can copy text question 1 and text question 2 to database 2 and associate them with the target file in database 2; at the same time, copy text question 3 and text question 4 to database 1 and associate them with the target file in database 1.

[0117] Specifically, reference may be made to the domain knowledge base 1 and the domain knowledge base 2 in the knowledge base 104 in the above embodiment. For the same knowledge point A, the expansion problem of the knowledge point A will exist in both the domain knowledge base 1 and the domain knowledge base 2.

[0118] In the embodiments of this specification, the data processing method processes multiple text data in different fields, stores the processed multiple text data in different databases according to target texts, and diffusely clones different text problems in the same target text in all databases to other databases, thereby ensuring unified data management and reducing subsequent maintenance costs.

[0119] See also Figure 3 , Figure 3 A flow chart of a data query method provided by an embodiment of the present specification is shown, which is applied to a data query platform and specifically includes the following steps.

[0120] Step 302: Receive a data query request containing a text question sent by a client.

[0121] The client can be understood as the interactive interface 108 in the above embodiment; the text question can be understood as a text question of any field and any type, such as the text question "What is insurance?" in the above embodiment.

[0122] Specifically, receiving a data query request carrying a text question sent by a client may be understood as receiving a data query request carrying a text question sent by a user based on a user interaction interface of the client.

[0123] Step 304: pre-process the text question based on the data query request, and determine the database corresponding to the text question in the data storage platform.

[0124] Specifically, preprocessing the text question can be understood as preprocessing the text question consulted by the user, such as removing special characters in the text question, replacing synonyms in the text question, removing numbers, punctuation marks, etc. in the text question, and filtering sensitive words in the text question.

[0125] In practical applications, if a text question has multiple corresponding databases, the corresponding text answer can be obtained from each of the corresponding multiple databases. If a text question has one corresponding database, the corresponding text answer can be directly obtained from the corresponding database. If the text question does not have a corresponding database, the corresponding text answer can be obtained from each of all databases of the data storage platform.

[0126] Specifically, determining the database corresponding to the text problem in the data storage platform includes:

[0127] Determine the field corresponding to the text question, and based on the field, determine multiple databases corresponding to the text question on a data storage platform.

[0128] In practical applications, after preprocessing the text problem, the field corresponding to the text problem is first determined, and then multiple corresponding databases of the text problem existing in the data storage platform are determined based on the corresponding field.

[0129] In addition, after preprocessing the text question based on the data query request, the method further includes:

[0130] When it is determined that no corresponding database exists in the data storage platform for the text question, multiple text answers corresponding to the text question are obtained from all databases of the data storage platform.

[0131] Specifically, if there is no corresponding database for the text question on the data storage platform, multiple text answers corresponding to the text question can be obtained from all databases of the data storage platform to ensure that the target answer to solve the text question can be returned to the client user, thereby improving the user experience.

[0132] Step 306: Acquire multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers.

[0133] The data storage platform is the data storage platform in the above data processing method, that is, the knowledge base 104 in the above embodiment.

[0134] Specifically, obtaining a plurality of text answers corresponding to the text question from a database corresponding to the text question includes:

[0135] Determine, from a database corresponding to the text question, an extended question corresponding to the text question, and determine a target answer corresponding to each extended question;

[0136] The target answers corresponding to all the extended questions are used as multiple text answers corresponding to the text questions.

[0137] In specific implementation, after determining the database corresponding to the text question, first obtain the extended question corresponding to the text question from each database. At this time, the extended question may include the extended question and the target question of the above-mentioned embodiment. The extended question in the embodiment of this specification can be understood as a question that is similar and related to the text question.

[0138] After obtaining the extended questions corresponding to the text question, the target answer corresponding to each extended question is obtained, and then the target answers corresponding to all extended questions are used as the text answers corresponding to the text question, so that the subsequent target text answer extraction can be performed more accurately.

[0139] Specifically, determining a target text answer corresponding to the text question from the multiple text answers includes:

[0140] Calculating the semantic similarity between each text answer in the multiple text answers and the text question;

[0141] sorting the plurality of text answers based on the semantic similarity;

[0142] The sorted multiple text answers are sequentially adjusted based on a preset weight rule, and a target text answer corresponding to the text question is selected from the adjusted multiple text answers.

[0143] In actual applications, after determining multiple text answers corresponding to multiple text questions, the target text answers corresponding to the text questions can be determined from the multiple text answers. First, the semantic similarity between each text answer in the multiple text answers and the text question is calculated, and the multiple text answers are sorted based on the semantic similarity.

[0144] In order to further ensure the accuracy of the target text answer, the order of the sorted multiple text answers will be adjusted based on the preset weight rules, and then the text answer with the highest weight value will be selected as the target text answer corresponding to the text question.

[0145] In another embodiment of the present specification, after determining the target text answer corresponding to the text question from the multiple text answers, the method further includes:

[0146] The database corresponding to the target text answer is determined, and the target text answer and attribute information of the database corresponding to the target text answer are sent to the client.

[0147] In actual applications, when the retrieval center obtains the target text answer corresponding to a text question from the data storage platform, if the text question is a general question and there are answers to the text question in multiple domain databases of the data storage platform, in order to ensure user experience, more abundant answers can be provided to users for selection. In this case, if the text question corresponds to target text answers returned by multiple domain databases, when the target text answer returned by each domain database is returned to the client, the library name of the corresponding domain database can be added to each target text answer.

[0148] For example, the text question is: What is insurance? This text question has corresponding target text answers in the pension insurance field database, health insurance field database, and children's insurance field database. When the target text answer is returned to the client, the library name of each field database can be included. For example, the target text answer returned to the client is: Pension insurance - insurance is ***, health insurance - insurance is ***; children's insurance - insurance is ***.

[0149] In the embodiments of this specification, the data query method can support the parallel acquisition of target text answers corresponding to text questions from knowledge bases in various fields through functional points such as field identification, multi-base recall, and knowledge disambiguation, which can greatly improve the retrieval completeness and retrieval efficiency.

[0150] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 4 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0151] A preprocessing module 402 is configured to preprocess the acquired multiple text data and classify the multiple text data into at least one type based on the similarity between the preprocessed multiple text data;

[0152] A question determination module 404 is configured to determine a target question, a target answer, and an extended question associated with the target question for each type of text data;

[0153] The storage module 406 is configured to determine a database corresponding to the target question in a data storage platform, and store the target question, the target answer and the extended question in the corresponding database;

[0154] The sharing module 408 is configured to obtain the same target question in all databases, and share different extended questions associated with the same target question in the database.

[0155] According to the data processing method of claim 1, before preprocessing the acquired multiple text data, the method further comprises:

[0156] A plurality of online text data and a plurality of offline text data are acquired, and a plurality of text data are formed based on the plurality of online text data and the plurality of offline text data.

[0157] Optionally, the preprocessing module 402 is further configured to:

[0158] Perform rule filtering on the acquired multiple text data according to preset filtering rules;

[0159] Perform sentence filtering on the multiple text data after being filtered by the rule according to the sentence scoring rule;

[0160] Sentiment filtering is performed on multiple text data after sentence filtering according to sentiment analysis rules.

[0161] Optionally, the preprocessing module 402 is further configured to:

[0162] Inputting the multiple text data after filtering by the rule into the language model to obtain a sentence score of each text data in the multiple text data after filtering by the rule;

[0163] From the multiple text data after rule filtering, delete the text data whose sentence scores are less than or equal to a preset score threshold.

[0164] Optionally, the preprocessing module 402 is further configured to:

[0165] Obtain the corpus sentiment and sensitive words of each text data in the multiple text data filtered by the sentence;

[0166] From the multiple text data after sentence filtering, delete the text data whose corpus emotions meet the preset emotion threshold or contain sensitive words.

[0167] Optionally, the storage module 406 is further configured to:

[0168] Determining the field corresponding to the target question by processing the text of the target question;

[0169] A database field corresponding to the target problem is determined based on the field corresponding to the target problem, and a database corresponding to the target problem on a data storage platform is determined based on the database field.

[0170] Optionally, the storage module 406 is further configured to:

[0171] Determine whether the target question has been stored in the database corresponding to the target question.

[0172] If yes, the extended question associated with the target question is stored in the corresponding database and associated with the target question.

[0173] If not, the target question, the target answer and the extended question are stored in a corresponding database.

[0174] Optionally, the sharing module 408 is further configured to:

[0175] Determine the expansion problem that does not exist in each target problem among the same target problems;

[0176] The extended questions that do not exist in each target question are stored in a database corresponding to each target question and associated with each target question.

[0177] In the embodiments of the present specification, the data processing device processes multiple text data in different fields, stores the processed multiple text data in different databases according to target problems, and diffusely clones different extended problems in the same target problem in all databases to other databases, thereby ensuring unified data management and reducing subsequent maintenance costs.

[0178] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the data processing device can be referred to the description of the technical scheme of the above data processing method.

[0179] Corresponding to the above method embodiment, this specification also provides a data query device embodiment, Figure 5 FIG. 1 shows a schematic diagram of a data query device provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0180] The request receiving module 502 is configured to receive a data query request carrying a text question sent by a client;

[0181] A preprocessing module 504 is configured to preprocess the text question based on the data query request and determine a database corresponding to the text question in a data storage platform;

[0182] The answer determination module 506 is configured to obtain multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers, wherein the data storage platform is the data storage platform in the above-mentioned data processing method.

[0183] Optionally, the answer determination module 506 is further configured to:

[0184] Determine, from a database corresponding to the text question, an extended question corresponding to the text question, and determine a target answer corresponding to each extended question;

[0185] The target answers corresponding to all the extended questions are used as multiple text answers corresponding to the text questions.

[0186] Optionally, the device further comprises:

[0187] The answer acquisition module is configured to acquire multiple text answers corresponding to the text question from all databases of the data storage platform when it is determined that there is no corresponding database for the text question on the data storage platform.

[0188] Optionally, the answer determination module 506 is further configured to:

[0189] Calculating the semantic similarity between each text answer in the multiple text answers and the text question;

[0190] sorting the plurality of text answers based on the semantic similarity;

[0191] The sorted multiple text answers are sequentially adjusted based on a preset weight rule, and a target text answer corresponding to the text question is selected from the adjusted multiple text answers.

[0192] Optionally, the device further comprises:

[0193] The sending module is configured to determine the database corresponding to the target text answer, and send the target text answer and the attribute information of the database corresponding to the target text answer to the client.

[0194] In the embodiments of this specification, the data query device can support the parallel acquisition of target text answers corresponding to text questions from knowledge bases in various fields through functional points such as field identification, multi-base recall, and knowledge disambiguation, which can greatly improve the retrieval completeness and retrieval efficiency.

[0195] The above is a schematic scheme of a data query device of this embodiment. It should be noted that the technical scheme of the data query device and the technical scheme of the above data query method belong to the same concept, and the details not described in detail in the technical scheme of the data query device can be referred to the description of the technical scheme of the above data query method.

[0196] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0197] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0198] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0199] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.

[0200] The processor 620 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the data processing method or the data query method.

[0201] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data processing method or the above-mentioned data query method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned data processing method or the above-mentioned data query method.

[0202] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, wherein the computer-executable instructions implement the steps of the data processing method or the data query method when executed by a processor.

[0203] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data processing method or the above-mentioned data query method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned data processing method or the above-mentioned data query method.

[0204] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0205] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0206] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0207] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0208] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a data management platform, comprising: Preprocessing the acquired multiple text data, and classifying the multiple text data into at least one type based on similarities between the preprocessed multiple text data; Determining a target question, a target answer, and an extended question associated with the target question for each type of text data; Determine a database corresponding to the target question in a data storage platform, and store the target question, the target answer, and the extended question in the corresponding database, wherein each of the databases corresponds to a field, and the field corresponding to each of the databases is the same as the field corresponding to the target question; The same target problem in all databases is obtained, and different extended problems associated with the same target problem are shared in the database, wherein the different extended problems associated with the same target problem are shared in the database, including: determining the extended problems that do not exist in each target problem in the same target problem; storing the extended problems that do not exist in each target problem in the database corresponding to each target problem, and associating them with each target problem.

2. The data processing method according to claim 1, before preprocessing the acquired multiple text data, further comprising: A plurality of online text data and a plurality of offline text data are acquired, and a plurality of text data are formed based on the plurality of online text data and the plurality of offline text data.

3. The data processing method according to claim 1, wherein the preprocessing of the acquired multiple text data comprises: Perform rule filtering on the acquired multiple text data according to preset filtering rules; Perform sentence filtering on the multiple text data after being filtered by the rule according to the sentence scoring rule; Sentiment filtering is performed on multiple text data after sentence filtering according to sentiment analysis rules.

4. The data processing method according to claim 3, wherein the step of performing sentence filtering on the plurality of text data after being filtered by the rule according to the sentence scoring rule comprises: Inputting the multiple text data after filtering by the rule into the language model to obtain a sentence score of each text data in the multiple text data after filtering by the rule; From the multiple text data after rule filtering, delete the text data whose sentence scores are less than or equal to a preset score threshold.

5. The data processing method according to claim 3 or 4, wherein the sentiment filtering of the plurality of text data after sentence filtering according to the sentiment analysis rule comprises: Obtain the corpus sentiment and sensitive words of each text data in the multiple text data filtered by the sentence; From the multiple text data after sentence filtering, delete the text data whose corpus emotions meet the preset emotion threshold or contain sensitive words.

6. The data processing method according to claim 1, wherein determining the database corresponding to the target problem in the data storage platform comprises: Determining the field corresponding to the target question by processing the text of the target question; A database field corresponding to the target problem is determined based on the field corresponding to the target problem, and a database corresponding to the target problem on a data storage platform is determined based on the database field.

7. The data processing method according to claim 1, wherein storing the target question, the target answer and the extended question in a corresponding database comprises: Determine whether the target question has been stored in the database corresponding to the target question. If yes, the extended question associated with the target question is stored in the corresponding database and associated with the target question. If not, the target question, the target answer and the extended question are stored in a corresponding database.

8. A data query method, applied to a data query platform, comprising: Receive a data query request containing text questions sent by a client; Preprocessing the text question based on the data query request, determining a database corresponding to the text question in a data storage platform, wherein the database includes extended questions corresponding to the text question, and the extended questions are obtained by obtaining the same text question from all databases to determine that the extended questions that do not exist in each text question are stored in the database corresponding to each text question; Acquire multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers, wherein the data storage platform is the data storage platform in claim 1.

9. The data query method according to claim 8, wherein obtaining a plurality of text answers corresponding to the text question from a database corresponding to the text question comprises: Determine, from a database corresponding to the text question, an extended question corresponding to the text question, and determine a target answer corresponding to each extended question; The target answers corresponding to all the extended questions are used as multiple text answers corresponding to the text questions.

10. The data query method according to claim 8, after preprocessing the text question based on the data query request, further comprising: When it is determined that no corresponding database exists in the data storage platform for the text question, multiple text answers corresponding to the text question are obtained from all databases of the data storage platform.

11. The data query method according to claim 8, wherein determining a target text answer corresponding to the text question from the plurality of text answers comprises: Calculating the semantic similarity between each text answer in the multiple text answers and the text question; sorting the plurality of text answers based on the semantic similarity; The sorted multiple text answers are sequentially adjusted based on a preset weight rule, and a target text answer corresponding to the text question is selected from the adjusted multiple text answers.

12. The data query method according to claim 8, after determining the target text answer corresponding to the text question from the multiple text answers, further comprising: The database corresponding to the target text answer is determined, and the target text answer and attribute information of the database corresponding to the target text answer are sent to the client.

13. A data processing device, comprising: A preprocessing module, configured to preprocess the acquired multiple text data, and classify the multiple text data into at least one type based on the similarity between the preprocessed multiple text data; A question determination module configured to determine a target question, a target answer, and an extended question associated with the target question for each type of text data; A storage module is configured to determine a database corresponding to the target question in a data storage platform, and store the target question, the target answer, and the extended question in the corresponding database, wherein each of the databases corresponds to a field, and the field corresponding to each of the databases is the same as the field corresponding to the target question; A sharing module is configured to obtain the same target problem in all databases and share different extended problems associated with the same target problem in the database, wherein the sharing module is further configured to: determine the extended problems that do not exist in each target problem in the same target problem; store the extended problems that do not exist in each target problem in the database corresponding to each target problem, and associate them with each target problem.

14. A data query device, comprising: A request receiving module is configured to receive a data query request carrying a text question sent by a client; A preprocessing module is configured to preprocess the text question based on the data query request, and determine a database corresponding to the text question in a data storage platform, wherein the database includes an extended question corresponding to the text question, and the extended question is obtained by obtaining the same text question from all databases to determine that the extended question that does not exist in each text question is stored in the database corresponding to each text question; The answer determination module is configured to obtain multiple text answers corresponding to the text question from a database corresponding to the text question; and determine a target text answer corresponding to the text question from the multiple text answers, wherein the data storage platform is the data storage platform in claim 1.

15. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data processing method described in any one of claims 1 to 7 or the data query method described in any one of claims 8 to 12 are implemented.

16. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 7 or the data query method according to any one of claims 8 to 12.

Images