A data lake based text prediction method

Abstract

The application discloses a kind of text prediction methods based on data lake, belong to artificial intelligence technical field, including: obtaining initial text data generated by application;The initial text data is filtered, and qualified text data is put into text type data pool as metadata;Text disambiguation model is constructed;According to the metadata and the corresponding metadata identifier, generate original data set;According to the metadata, the corresponding original process data and the corresponding metadata identifier, generate original process data set;Original data set and original process data set are respectively input into text disambiguation model, and obtain original data fitting value and original process data fitting value;The corresponding metadata and original process data are combined as keyword;Markov chain keyword prediction model is constructed, and the hot-cold transition probability of keyword that last user asked is used to predict the hot-cold transition probability of keyword that user may initiate next time;Output the preset scheme of question that user may care.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a text prediction method based on a data lake. BACKGROUND

[0002] With the development of science and technology and the Internet, the advent of the big data era is promoted, and a large number of data fragments are generated every day in various industries, and the data measurement unit has developed from Byte, KB, MB, GB, TB to PB, EB, ZB, YB, even BB, NB and DB. In the big data era, data collection is no longer a technical problem, but how can we find the internal law of such a large amount of data. The data lake architecture faces multi-data source information storage, including the Internet of Things. Big data analysis or archiving / processing or delivering data subsets to requesting users through accessing the data lake. With the in-depth study of database theory, in order to overcome the shortcomings of the database model in the expression ability and strengthen the semantic knowledge component, scientists have proposed many concept models at a higher abstraction level than the original basis. On the other hand, artificial intelligence all has a problem of ability to describe and complete intellectual behavior with knowledge, that is, how to establish a knowledge base. Under such a premise, the research of database systems and artificial intelligence, the research of formal language and natural language processing, the research, development and application of knowledge base systems are gathered together.

[0003] In the prior art, the research on the knowledge base system lacks the research on predicting user questions. In the process of providing services for users, users often ask questions, and in the prior art, the user's question needs to be waited until the user gives all the questions, and then the customer service personnel can answer the question according to the content of the question, so that the efficiency of solving the user's question is low. SUMMARY

[0004] The purpose of the embodiment of the application is to provide a text prediction method and device, which can solve the technical problems that the prior art lacks research on predicting user questions, it is difficult to quickly solve the user's question when the user asks a question, and the efficiency of solving the user's question is low.

[0005] In order to solve the above technical problems, the application is implemented as follows:

[0006] The embodiment of the application provides a text prediction method based on a data lake, which comprises the following steps:

[0007] S101: acquiring initial text data generated by an application and storing the initial text data in a created initial data lake;

[0008] S102: screening the initial text data, putting qualified text data as metadata into a text type data pool, and putting unqualified text data into a miscellaneous data pool;

[0009] S103: constructing a text disambiguation model;

[0010] S104: establishing a correlation between the metadata and the corresponding metadata identifier, and generating an original data set;

[0011] S105: combining the metadata and the corresponding original process data content into a meta-process data set, establishing a correlation between the meta-process data and the corresponding metadata identifier, and generating an original process data set;

[0012] S106: inputting the original data set and the original process data set into the text disambiguation model to obtain original data fitting values and original process data fitting values;

[0013] S107: in the case where the difference between the original data fitting value and the original process data fitting value is less than a preset proportion, combining the corresponding metadata and the original process data into a keyword, and putting the keyword into a knowledge base;

[0014] S108: constructing a Markov chain keyword prediction model, and predicting the hot-cold transition probability of the keyword that the user is likely to initiate next time according to the hot-cold transition probability of the keyword that the user asked last time;

[0015] S109: outputting a preset scheme of the problem that the user is likely to care about according to the hot-cold transition probability of the keyword that the user is likely to initiate next time.

[0016] Further, the original process data content includes data source, date, location, and description.

[0017] Further, the S102 specifically includes:

[0018] S1021: in the case where the text data meets one of the following unqualified conditions, putting the unqualified text data into a miscellaneous data pool:

[0019] the length of the data is less than a preset length;

[0020] the data value is repeated;

[0021] the data fluctuation is small;

[0022] the data has garbled code;

[0023] S1022: in the case where the text data meets the following qualified conditions at the same time, putting the qualified text data as metadata into a miscellaneous data pool:

[0024] The length of the data is greater than or equal to a preset length.

[0025] The value of the data is not repeated.

[0026] The fluctuation of the data is large.

[0027] The data does not exist garbled.

[0028] Further, the preset length is 30 kb.

[0029] Further, after the S106, further comprising:

[0030] S110: In the case that the difference between the original data fitting value and the original process data fitting value is greater than or equal to the preset proportion, filtering the corresponding metadata and original process data.

[0031] Further, the preset proportion is 10%.

[0032] Further, the text disambiguation model comprises a ridge regression algorithm, and the ridge regression algorithm is used to prevent overfitting.

[0033] Further, the Markov chain keyword prediction model can be expressed as:

[0034] X(k+1)=X(k)×P Formula 1

[0035] Wherein, X(k) represents the state vector of the trend analysis and prediction object at t=k moment, P represents one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at t=k+1 moment.

[0036] In the embodiment of the application, through the text disambiguation model and the Markov chain keyword prediction model, the keyword hot-cold change probability of the next time that the user may initiate can be predicted according to the keyword hot-cold change probability of the last time that the user asks, and then the preset scheme of the question that the user may care about is output according to the keyword hot-cold change probability of the next time that the user may initiate. When the user asks a question, the preset scheme of the question that the user may care about can be output in advance without waiting for the user to give all his questions, thereby improving the efficiency of solving the user's questions. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of a text prediction method based on a data lake provided by the embodiment of the application;

[0038] Figure 2 is a flowchart of a text disambiguation model provided by the embodiment of the application.

[0039] The objectives, functional features and advantages of the present application will be further illustrated in combination with embodiments and with reference to the drawings. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings accompanying the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] The embodiments of the present application will be described in detail below with reference to the drawings, the specific embodiments and application scenarios.

[0042] Referring to Figure 1 , a flowchart of a text prediction method based on a data lake is shown.

[0043] Referring to Figure 2 , a flowchart of a text disambiguation model is shown.

[0044] The text prediction method based on a data lake provided by the embodiments of the present application comprises:

[0045] S101: Obtain initial text data generated by an application and store the initial text data in a created initial data lake.

[0046] S102: Screen the initial text data, put qualified text data as metadata into a text type data pool, and put unqualified text data into a miscellaneous data pool.

[0047] In a possible implementation, S102 specifically comprises:

[0048] S1021: In a case where the text data meets one of the following unqualified conditions, put the unqualified text data into the miscellaneous data pool:

[0049] The length of the data is less than a preset length;

[0050] The numerical value of the data is repeated;

[0051] The fluctuation of the data is small;

[0052] The data has garbled code;

[0053] S1022: In a case where the text data meets the following qualified conditions at the same time, put the qualified text data as metadata into the miscellaneous data pool:

[0054] The length of the data is greater than or equal to a preset length;

[0055] The value of the data is not repeated;

[0056] The fluctuation of the data is large;

[0057] The data does not exist garbled.

[0058] The preset length is 30kb.

[0059] S103: Construct a text disambiguation model.

[0060] The ridge regression algorithm is used to prevent overfitting.

[0061] The formula of the text disambiguation model can be expressed as:

[0062] ||Xθ-y|| 2 +||Γθ|| 2

[0063] ||Xθ-y|| 2 +||Γθ|| 2 Indicates that the operation process is added to the regularization; X represents the input; y represents the output; ||| represents the regularization operation; θ is the fitting hyperparameter; Γ is the weight constant.

[0064] Define Γ=aI (objective training result = fitting value) to prevent overfitting operation process:

[0065] θ(a)=(X T X+aI) -1 X T y

[0066] Where I is the identity matrix; a is the weight of the identity matrix; θ(a) represents the value of θ under the condition that a is determined.

[0067] Specifically, the least squares method commonly used in regression analysis is an unbiased estimate. For a well-posed problem, X is usually a column full-rank Xθ=y,

[0068] Using the least squares method, define the loss function as the square of the residual, minimize the loss function

[0069] ||Xθ-y|| 2 .

[0070] The above optimization problem can be solved by gradient descent method, or can be directly solved by the following formula

[0071] θ=(X T X) -1 XT y,

[0072] When X is not full rank, or the linear correlation between some columns is large, X T The determinant of X is close to 0, that is, X T X is close to singular, the above problem becomes an ill-posed problem, at this time, the calculation (X T X) -1 The error will be large, and the traditional least squares method lacks stability and reliability.

[0073] In order to solve the above problems, it is necessary to transform the ill-posed problem into a well-posed problem: we add a regularization term to the above loss function, which becomes

[0074] ||Xθ-y|| 2 +||Γθ|| 2

[0075] Where, define Γ=aI, then:

[0076] θ(a)=(X T X+aI) -1 X T y

[0077] In the above formula, I is the unit matrix.

[0078] S104: Establish an association between the metadata and the corresponding metadata identifier, and generate the original data set.

[0079] S105: Form the meta-process data set by combining the metadata and the corresponding original process data content, establish an association between the meta-process data and the corresponding metadata identifier, and generate the original process data set.

[0080] Wherein, the original process data content includes data source, date, place and description.

[0081] In actual application process, each metadata has associated original data, but may not have original process data.

[0082] Wherein, the data merging can unify the semantics, which is convenient for subsequent analysis.

[0083] S106: Input the original data set and the original process data set into the text disambiguation model respectively, and obtain the original data fitting value and the original process data fitting value.

[0084] Optionally, the original data fitting value is the number of times each keyword appears in the original data set, and the number of times from high to low represents the heat of the keyword from high to low.

[0085] Optionally, the original process data fitting value is the number of occurrences of each keyword in the original process data set, and the number of occurrences arranged from high to low represents the use frequency of the keyword from high to low.

[0086] In a possible implementation, after S106, the method further includes:

[0087] S110: In the case that the difference between the original data fitting value and the original process data fitting value is greater than or equal to a preset proportion, filtering the corresponding metadata and original process data.

[0088] The preset proportion is 10%.

[0089] S107: In the case that the difference between the original data fitting value and the original process data fitting value is less than the preset proportion, combining the corresponding metadata and original process data into a keyword, and putting the keyword into the knowledge base.

[0090] S108: Constructing a Markov chain keyword prediction model, and predicting the hot-cold transition probability of the keyword that the user is likely to initiate next time according to the hot-cold transition probability of the keyword that the user asked last time.

[0091] The Markov chain keyword prediction model can be expressed as:

[0092] X(k+1) = X(k) × P Formula 1

[0093] X(k) represents the state vector of the trend analysis and prediction object at t=k, P represents a one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at t=k+1.

[0094] S109: Outputting a preset scheme of the question that the user is likely to care about according to the hot-cold transition probability of the keyword that the user is likely to initiate next time.

[0095] In the embodiment of the application, through the text disambiguation model and the Markov chain keyword prediction model, the hot-cold transition probability of the keyword that the user is likely to initiate next time can be predicted according to the hot-cold transition probability of the keyword that the user asked last time, and then a preset scheme of the question that the user is likely to care about can be output according to the hot-cold transition probability of the keyword that the user is likely to initiate next time. When the user asks a question, the user does not need to wait for the user to give all the questions, and a preset scheme of the question that the user is likely to care about can be output in advance, thereby improving the efficiency of solving the user's question.

[0096] The above merely describes the embodiments of the application and is not intended to limit the application. The application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application shall be included in the scope of the claims of the application.

Claims

1. A text prediction method based on a data lake, characterized in that, include: S101: Obtain the initial text data generated by the application and store it in the created initial data lake; S102: Filter the initial text data, put qualified text data into the text type data pool as metadata, and put unqualified text data into the miscellaneous data pool; S103: Construct a text disambiguation model; S104: Establish an association between the metadata and the corresponding metadata identifier, and generate the original data set; S105: Combine the metadata and the corresponding original process data content into a meta-process data set, establish an association between the meta-process data and the corresponding metadata identifier, and generate the original process data set; S106: Input the original data set and the original process data set into the text disambiguation model respectively to obtain the original data fitting value and the original process data fitting value; S107: If the difference between the original data fitting value and the original process data fitting value is less than a preset ratio, combine the corresponding metadata and the original process data into keywords, and put the keywords into the knowledge base; S108: Construct a Markov chain keyword prediction model to predict the probability of hot-cold transition of the keyword that the user may ask next, based on the hot-cold transition probability of the keyword in the previous user question. S109: Based on the probability of the user's next keyword hot-to-cold transition, output a preset solution for the question the user may be concerned about; Specifically, S102 includes: S1021: If the text data meets one of the following non-compliant conditions, the non-compliant text data shall be placed in the miscellaneous data pool: The data length is less than the preset length; The data contains repeated values; The data fluctuates little; The data contains garbled characters; S1022: If the text data simultaneously meets the following eligibility conditions, the qualified text data will be placed into the text type data pool as metadata: The length of the data is greater than or equal to the preset length; The data values ​​are not repeated; The data fluctuates greatly; The data does not contain garbled characters; The preset length is 30kb; Following S106, the following is also included: S110: If the difference between the original data fitting value and the original process data fitting value is greater than or equal to the preset ratio, filter the corresponding metadata and the original process data. The original process data content includes data source, date, location, and description; The preset ratio is 10%.

2. The text prediction method according to claim 1, characterized in that, The text disambiguation model includes a ridge regression algorithm, which is used to prevent overfitting.

3. The text prediction method according to claim 1, characterized in that, The Markov chain keyword prediction model can be expressed as: X(k+1)=X(k)×P where X(k) represents the state vector of the trend analysis and prediction object at time t=k, P represents the one-step transition probability matrix, and X(k+1) represents the state vector of the trend analysis and prediction object at time t=k+1.

Images