Natural language enhanced data analysis method and system oriented to big data

Abstract

The invention discloses a big-data-oriented natural language enhanced data analysis method and system, and relates to the technical field of natural language conversion management, and the big-data-oriented natural language enhanced data analysis system comprises a historical query data acquisition module, a language text screening module, a query information acquisition module, an information retrieval planning module and a retrieval planning optimization module. Historical retrieval information and historical natural language texts of a user are collected through a historical query data collection module, and the historical natural language texts are classified and preprocessed; screening a preferred natural language text through a language text screening module and generating a preferred database; recognizing the voice of the current user through a query information acquisition module and converting the voice into a natural language text; planning a selection mode of a to-be-converted text through an information retrieval planning module, converting the to-be-converted text into sql, and performing information retrieval; the selection mode of the to-be-converted text is optimized through the retrieval planning optimization module, and the probability that information does not meet user requirements due to improper use of the to-be-converted text is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language conversion management, and in particular to a natural language enhanced data analysis method and system for big data. Background Art

[0002] Data enhancement in natural language processing aims to better meet specific needs by performing various transformations on text data. The purpose of converting natural language into structured query language SQL is to improve the efficiency and convenience of database queries and further help users use natural language to query the required information.

[0003] However, when a user searches for the required information through voice input, the user's input voice must first be converted into a natural language text form through voice recognition, and then converted into SQL. Since the natural language text obtained by voice recognition conversion is diverse, there may be multiple natural language texts that can express the same meaning as the user's input voice. However, for a specific SQL language conversion model, there may be a problem that the information retrieved after some language texts in a text expressing one meaning are converted into SQL does not meet the user's needs. At this time, the user needs to input voice search again, which reduces the efficiency of information retrieval. The existing technology fails to pre-judge the matching situation between natural language text and user needs to replace the natural language to be converted in advance, that is, to perform natural language enhancement processing, but directly converts the recognized natural language text into SQL and performs information retrieval, which cannot effectively reduce the probability that the retrieved information does not meet the user's needs due to improper use of the text to be converted. Summary of the invention

[0004] The purpose of the present invention is to provide a natural language enhanced data analysis method and system for big data to solve the problems raised in the prior art.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a natural language enhanced data analysis system for big data, comprising a historical query data collection module, a language text screening module, a query information collection module, an information retrieval planning module and a retrieval planning optimization module; the historical query data collection module collects the user's historical retrieval information and the historical natural language text generated by converting the voice input by the user who needs to perform information retrieval, and classifies and pre-processes the historical natural language text; the language text screening module analyzes the matching situation between the pre-processed historical natural language text and the user's needs, screens the preferred natural language text and generates a preferred database; the query information collection module collects the voice information of the user who currently needs to perform information retrieval, and converts the voice information into the current natural language text; the information retrieval planning module plans the selection method of the text to be converted, converts the selected text to be converted into SQL language, and uses SQL language to perform information retrieval; the retrieval planning optimization module establishes a text to be converted selection optimization model, and optimizes the selection method of the text to be converted.

[0006] Furthermore, the historical query data collection module includes a natural language text collection unit, a historical text classification processing unit and a retrieval information collection unit; the natural language text collection unit collects historical natural language text generated after the user input voice each time information needs to be retrieved, and recognizes the user input voice through voice recognition technology; the historical text classification processing unit classifies the historical natural language text according to semantic similarity; the retrieval information collection unit collects the number of times the user re-enters the voice and the interval time information of the re-entering voice after converting the historical natural language text into SQL language and transmitting the information retrieved by SQL language to the user terminal.

[0007] Furthermore, the language and text screening module includes a demand matching degree analysis unit, a preferred text screening unit and a preferred database generation unit; the demand matching degree analysis unit analyzes the matching coefficients of historical natural language texts classified into the same category and user needs; the preferred text screening unit compares the matching coefficients of historical natural language texts classified into the same category and user needs, and screens out preferred natural language texts from historical natural language texts; the preferred database generation unit screens out the preferred natural language texts in each category and generates a preferred database, which includes the screened preferred natural language texts in each category; the query information acquisition module includes a speech receiving and recognition unit and a natural language text generation unit, and the speech receiving and recognition unit receives the speech and performs speech recognition on the user's input speech after the user currently inputs the speech; the natural language text generation unit generates the current natural language text after completing the speech recognition.

[0008] Furthermore, the information retrieval planning module includes a text matching unit in the library, a text selection unit to be converted, a SQL conversion unit and an information retrieval unit; the current natural language text is matched with the preferred natural language text in the preferred database through the text matching unit in the library: the semantic similarity between the current natural language text and the preferred natural language text is analyzed, and the semantic similarity and the semantic similarity threshold are compared; if there is no preferred natural language text in the preferred database whose semantic similarity with the current natural language text is higher than the threshold, the current natural language text is selected as the text to be converted through the text selection unit to be converted; if there is a preferred natural language text in the preferred database whose semantic similarity with the current natural language text is higher than the threshold, the preferred natural language text with the highest semantic similarity with the current natural language text is selected as the text to be converted; the text to be converted is converted into SQL language using the T5 model through the SQL conversion unit; information retrieval is performed using SQL language through the information retrieval unit and the retrieved information is sent to the user.

[0009] Furthermore, the retrieval planning optimization module includes a selection model building unit and a selection method optimization unit; after performing several operations of selecting texts to be converted, the selection model building unit screens out the preferred natural language texts selected as the texts to be converted, collects the semantic similarity between the screened texts and the natural language texts generated after the corresponding recognition of the user input voice, calculates the deviation value between the semantic similarity and the semantic similarity threshold, analyzes the matching coefficient of the screened text and the user demand of the corresponding input voice, performs fitting training on the deviation value and the matching coefficient, and establishes a selection optimization model for the texts to be converted; the selection method of the texts to be converted is optimized by the selection method optimization unit: The speech input by the user is recognized to generate a natural language text. If there is no preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than a threshold, the corresponding generated natural language text is selected as the text to be converted; if there is a preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than the threshold, the deviation value between the semantic similarity and the threshold is calculated, and the deviation value is input into the optimization model for selecting the text to be converted, and the matching coefficient between the preferred natural language text whose semantic similarity with the corresponding generated natural language text is higher than the threshold and the corresponding user is output, and the preferred natural language text with the highest matching coefficient is selected as the text to be converted.

[0010] A natural language enhanced data analysis method for big data, characterized by comprising the following steps:

[0011] Z100: Collects historical search information of users and converts the voice input of users who need to search for information into historical natural language text, and performs classification preprocessing on the historical natural language text;

[0012] Z101: Analyze the matching between the pre-processed historical natural language text and user needs, select the optimal natural language text and generate the optimal database;

[0013] Z102: Collect the voice information of the user who currently needs to retrieve information, and convert the voice information into the current natural language text;

[0014] Z103: Plan the method of selecting the text to be converted, convert the selected text to be converted into SQL language, and use SQL language to perform information retrieval;

[0015] Z104: Establish an optimization model for selecting texts to be converted and optimize the selection method of texts to be converted.

[0016] Furthermore, step Z100 includes: collecting historical natural language texts generated after the user input voice is recognized by voice recognition technology each time information needs to be retrieved in the past, analyzing the semantic similarity between the historical natural language texts, setting the semantic similarity threshold to S, randomly extracting a text from the historical natural language texts, analyzing the semantic similarity between the remaining texts and the extracted text one by one, classifying the texts with a semantic similarity higher than S to the extracted text into the same category as the extracted text, and after the first classification is completed, classifying the remaining unclassified texts in the same way until there is no unclassified text, converting the historical natural language texts into SQL language, using SQL language to retrieve information and transmitting the retrieved information to the user, and collecting the number of times the user has re-entered the voice after receiving the information and the interval time of re-entering the voice.

[0017] Further, step Z101 includes: counting that there are n historical natural language texts classified as random, retrieving the number of times the user re-entered the voice after receiving the information retrieved by converting the corresponding n historical natural language texts into SQL language as L = {L1, L2, ..., Ln}, retrieving the number of times the user re-entered the voice after receiving the information retrieved by converting the i-th historical natural language text into SQL language as Li, and obtaining the interval time set of re-entering the voice as T = {T1, T2, ..., T Li}, calculate the matching coefficient Ri between the i-th historical natural language text and user needs according to the following formula:

[0018] Ri=(1 / Li)+[(∑ Li j＝1(T j )) / (Li);

[0019] j represents the j-th time the user re-enters the voice after receiving the information retrieved after converting the i-th historical natural language text into SQL language. The matching coefficients between n historical natural language texts and the user's requirements are calculated in the same way. The n matching coefficients are compared, and the n historical natural language texts are arranged in descending order of the matching coefficients. The first m historical natural language texts after the arrangement are selected as the preferred natural language texts, where 1 < m < n. The preferred natural language texts in each category are selected to generate a preferred database;

[0020] Using big data technology to collect historical information query data, analyze the number of times and time interval information of the user re-entering the voice for retrieval after receiving the information retrieved after converting the input voice into a natural language text and then converting the natural language text into SQL language. The more times of re-retrieval, the more it is judged that the retrieved information does not meet the user's requirements. Combining the time interval information to analyze the matching coefficient aims to consider that the user may have another retrieval requirement when entering the voice after a long interval, and the previous retrieval requirement may have been met. In this case, if only the number of times is used to analyze the matching coefficient, the obtained matching coefficient has an error. Combining the number of times and the interval time to analyze the matching coefficient can effectively avoid this error. According to the matching coefficient, the preferred natural language texts are selected and a preferred database is generated. Considering the problem of insufficient data in the previous preferred database, there is more than one preferred text in each category selected here;

[0021] Further, step Z103 includes: after obtaining the current natural language text in step Z102, analyze the semantic similarity between the current natural language text and the preferred natural language texts, and compare the semantic similarity with S: if there is no preferred natural language text in the preferred database with a semantic similarity higher than S to the current natural language text, select the current natural language text as the text to be converted; if there is a preferred natural language text in the preferred database with a semantic similarity higher than S to the current natural language text, select the preferred natural language text with the highest semantic similarity to the current natural language text as the text to be converted, use the T5 model to convert the text to be converted into SQL language, perform information retrieval using the SQL language and send the retrieved information to the user;

[0022] After generating the preferred database, a preliminary selection plan is made for the text to be converted: the text to be converted is selected by confirming whether there is a preferred natural language text matching the current text in the preferred database. If there is a preferred text matching it in the preferred database, the current text is replaced with the best matching preferred text as the text to be converted. After the replacement, SQL conversion is performed and information is retrieved. Timely replacement when necessary can effectively reduce the probability that the retrieved information does not meet user needs due to improper use of the text to be converted.

[0023] Further, step Z104 includes: after performing k text selection operations to be converted using the selection method planned in step Z103, a total of r preferred natural language texts selected as texts to be converted are screened out, the semantic similarity set between the r texts and the natural language texts generated after the corresponding recognition of the user input voice is collected as {F1, F2, ..., Fr}, the deviation value set between the semantic similarity and S is calculated as {F1-S, F2-S, ..., Fr-S}, and it is counted that after the e-th text selected is converted into SQL language for information retrieval and the retrieved information is transmitted to the user, the number of times the user re-enters the voice is g, and the interval time set of g times of re-entering the voice is {t1, t2, ..., t g}, and obtain the matching coefficient We between the filtered e-th text and the user's needs, We=(1 / g)+[(∑ g u＝1 (t u )) / g], u represents the u-th re-input of speech, and the set of matching coefficients between the selected r texts and user needs is {W1, W2, ...We, ...Wr}, generating training data {(F1-S, W1), (F2-S, W2), ...(Fr-S, Wr)}, and establishing the optimization model for selecting texts to be converted after linear fitting of the training data:

[0024] Y = δ1*X+δ2;

[0025] Among them, δ1 and δ2 represent fitting coefficients, X represents the variable referring to the deviation value in the model, and Y represents the variable referring to the matching coefficient in the model. The optimization selection method is as follows: after performing speech recognition on the speech input by the user, a natural language text is generated. If there is no preferred natural language text with a semantic similarity higher than S with the corresponding generated natural language text in the preferred database, the corresponding generated natural language text is selected as the text to be converted; if there are a total of H preferred natural language texts with a semantic similarity higher than S with the corresponding generated natural language text in the preferred database, the deviation value between the semantic similarity between the a-th text in the H texts and the corresponding generated natural language text and S is calculated to be A, let X=A, output the matching coefficient between the a-th text and the user demand as δ1*A+δ2, input the H deviation values ​​into the model one by one, output H matching coefficients, and select the preferred natural language text with the highest matching coefficient in the H texts as the text to be converted;

[0026] After multiple selections of texts to be converted were made according to the preliminary selection planning method, taking into account the variability of users who need to retrieve information, the information retrieved by selecting texts to be converted according to the preliminary planning method may also not match the changing user needs. Further analysis is conducted on the situation after selecting texts to be converted according to the preliminary planning method to establish an optimization model for selecting texts to be converted. The model is used as reference data for optimizing the selection of texts to be converted, which further optimizes the selection method of texts to be converted and further improves the matching between the retrieved information and users.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention collects historical information query data through big data technology, analyzes the number of times and time interval information of the user re-enters the voice for retrieval after receiving the information retrieved after converting the input voice into natural language text and then converting the natural language text into SQL language, analyzes the matching coefficient of the historical natural language text and the user demand in combination with the number of times and the interval time, screens out the preferred natural language text and generates a preferred database, performs preliminary selection planning on the text to be converted after the preferred database is generated, and replaces the text to be converted in time, effectively reducing the probability that the retrieved information does not meet the user demand due to improper use of the text to be converted, and further optimizes the selection method of the text to be converted by using the optimization model for selecting the text to be converted, thereby further improving the matching of the retrieved information with the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a flow chart of a natural language enhanced data analysis method for big data according to the present invention;

[0030] Figure 2This is a structural schematic diagram of a natural language enhanced data analysis system for big data according to the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Example 1: Figure 1 As shown, this embodiment provides a natural language enhanced data analysis method for big data, including the following steps:

[0033] Z100: Collect historical search information of users and convert the voice input by users who need to search for information into historical natural language texts, and perform classification preprocessing on the historical natural language texts: collect the voice input by users who need to search for information in the past, and use voice recognition technology to recognize the voice input by users to generate historical natural language texts, analyze the semantic similarity between historical natural language texts, set the semantic similarity threshold to S, randomly extract a text from the historical natural language texts, analyze the semantic similarity between the remaining texts and the extracted texts one by one, and classify the texts with semantic similarity higher than S to the extracted texts into the same category as the extracted texts. After the first classification is completed, perform classification preprocessing on the remaining unclassified texts in the same way until there are no unclassified texts, convert the historical natural language texts into SQL language, use SQL language to search for information and transmit the retrieved information to the user, collect the number of times the user has re-entered the voice after receiving the information and the interval time between re-entering the voice, and the historical search information refers to the number and interval time information;

[0034] Z101: Analyze the matching situation between the pre - processed historical natural language text and the user's needs, screen the preferred natural language text and generate a preferred database: It is statistically found that there are n historical natural language texts classified as random. After retrieving the information that the user has retrieved in the past after converting the corresponding n historical natural language texts into SQL language and then re - inputting the voice, the set of the number of times of re - inputting the voice is L = {L1, L2,..., Ln}. When the retrieved information is not the information the user needs, the user will re - input the voice to retrieve information. When the re - retrieved information is still not the information the user needs, the user will re - input the voice to retrieve information until the required information is obtained. Therefore, the number of times of re - inputting the voice may be more than once. The number of times of re - inputting the voice after retrieving the information that the user has retrieved after converting the i - th historical natural language text into SQL language is Li, and the set of the time intervals between re - inputting the voice is T = {T1, T2,..., T Li}, T1 represents the time interval between the time when the user re - inputs the voice for the first time and the time when the information is received, T2 represents the time interval between the time when the user re - inputs the voice for the second time and the time when the user re - inputs the voice for the first time. Calculate the matching coefficient Ri of the i - th historical natural language text and the user's needs according to the following formula:

[0035] Ri=(1 / Li)+[(∑ Li j＝1 (T j )) / (Li)];

[0036] j represents the j - th time the user re - inputs the voice after retrieving the information that the user has retrieved after converting the i - th historical natural language text into SQL language. Calculate the matching coefficients of the n historical natural language texts and the user's needs in the same way, compare the n matching coefficients, arrange the n historical natural language texts in descending order of the matching coefficients, screen out the first m historical natural language texts after arrangement as the preferred natural language text, 1 < m < n, screen out the preferred natural language texts in each category, and generate a preferred database;

[0037] Z102: Collect the voice information of the user who needs to perform information retrieval currently, and convert the voice information into the current natural language text;

[0038] Z103: Plan the selection method of the text to be converted, convert the selected text to be converted into SQL language, and use SQL language for information retrieval: After executing step Z102 to obtain the current natural language text, analyze the semantic similarity between the current natural language text and the preferred natural language text, and compare the semantic similarity and S: If there is no preferred natural language text with a semantic similarity higher than S to the current natural language text in the preferred database, select the current natural language text as the text to be converted; If there is a preferred natural language text with a semantic similarity higher than S to the current natural language text in the preferred database, select the preferred natural language text with the highest semantic similarity to the current natural language text as the text to be converted, use the T5 model to convert the text to be converted into SQL language, use SQL language for information retrieval and send the retrieved information to the user. The T5 model is a pre-trained language model based on the Transformer architecture, which aims to explore the transfer learning of natural language processing tasks through a unified text-to-text conversion framework. The T5 model can realize the conversion of natural language to structured query language SQL;

[0039] Z104: Establish an optimization model for selecting texts to be converted, and optimize the selection method of texts to be converted: after performing k text selection operations to be converted using the selection method planned in step Z103, a total of r preferred natural language texts selected as texts to be converted are screened out, and the semantic similarity set between the r texts and the natural language texts generated after the corresponding recognition of the user input voice is {F1, F2, ..., Fr}. The deviation value set between the semantic similarity and S is calculated to be {F1-S, F2-S, ..., Fr-S}. It is counted that after the e-th text selected is converted into SQL language for information retrieval and the retrieved information is transmitted to the user, the number of times the user re-enters the voice is g, and the interval time set of g times of re-entering the voice is {t1, t2, ..., t g}, t1 represents the interval between the time when the user re-enters the voice for the first time after receiving the information retrieved by using the converted e-th text and the time when the information is received, and the matching coefficient We between the filtered e-th text and the user's needs is obtained, We=(1 / g)+[(∑ g u＝1 (t u )) / g], u represents the u-th re-input of speech, and the set of matching coefficients between the selected r texts and user needs is {W1, W2, ...We, ...Wr}, generating training data {(F1-S, W1), (F2-S, W2), ...(Fr-S, Wr)}, and establishing the optimization model for selecting texts to be converted after linear fitting of the training data:

[0040] Y = δ1*X+δ2;

[0041] Wherein, * represents a multiplication sign, δ1 and δ2 represent fitting coefficients, X represents a variable in the model that refers to a deviation value, and Y represents a variable in the model that refers to a matching coefficient. The optimization selection method is as follows: after performing speech recognition on the user's input speech, a natural language text is generated. If there is no preferred natural language text in the preferred database that has a semantic similarity higher than S with the corresponding generated natural language text, the corresponding generated natural language text is selected as the text to be converted; if there are a total of H preferred natural language texts in the preferred database that have a semantic similarity higher than S with the corresponding generated natural language text, the deviation value between the semantic similarity between the a-th text in the H texts and the corresponding generated natural language text and S is calculated to be A, and X=A is set, and the matching coefficient between the a-th text and the user's demand is output as δ1*A+δ2, and the H deviation values ​​are input into the model one by one, and H matching coefficients are output, and the preferred natural language text with the highest matching coefficient is selected from the H texts as the text to be converted;

[0042] For example: establish an optimization model for selecting texts to be converted: Y=66.7X-5, generate natural language text after performing speech recognition on the user's input voice, count the preferred natural language texts in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than the threshold, calculate the deviation values ​​between the semantic similarity between the three texts and the corresponding generated natural language text and the semantic similarity threshold, and set X to 0.11, 0.26 and 0.22 respectively. Output the matching coefficients of the three texts with user needs as 2.34, 12.34 and 9.67 respectively, with the highest matching coefficient being 12.34, select the second text as the text to be converted, use the T5 model to convert the text to be converted into sql language, use sql language to retrieve information and send the retrieved information to the user.

[0043] Example 2: Figure 2As shown, this embodiment provides a natural language enhanced data analysis system for big data, which is implemented based on the data analysis method in the embodiment, and specifically includes: a historical query data collection module, a language text screening module, a query information collection module, an information retrieval planning module and a retrieval planning optimization module; the historical query data collection module collects the user's historical retrieval information and the historical natural language text generated by converting the voice input by the user who needs to perform information retrieval, and classifies and pre-processes the historical natural language text; the language text screening module analyzes the matching between the pre-processed historical natural language text and the user's needs, selects the preferred natural language text and generates a preferred database; the query information collection module collects the voice information of the user who currently needs to perform information retrieval, and converts the voice information into the current natural language text; the information retrieval planning module plans the selection method of the text to be converted, converts the selected text to be converted into SQL language, and uses SQL language for information retrieval; the retrieval planning optimization module establishes a text selection optimization model for conversion, and optimizes the selection method of the text to be converted.

[0044] The historical query data collection module includes a natural language text collection unit, a historical text classification processing unit and a retrieval information collection unit; the natural language text collection unit collects the historical natural language text generated after the user input voice each time the information needs to be retrieved, and recognizes the user input voice through voice recognition technology; the historical natural language text is classified and processed according to semantic similarity by the historical text classification processing unit; the retrieval information collection unit collects the number of times the user re-enters the voice and the interval time information of the re-entering voice after the historical natural language text is converted into SQL language and the information retrieved by SQL language is transmitted to the user terminal.

[0045] The language and text screening module includes a demand matching degree analysis unit, a preferred text screening unit and a preferred database generation unit; the demand matching degree analysis unit analyzes the matching coefficients of historical natural language texts classified into the same category and user needs; the preferred text screening unit compares the matching coefficients of historical natural language texts classified into the same category and user needs, and screens out preferred natural language texts from the historical natural language texts; the preferred database generation unit screens out the preferred natural language texts in each category and generates a preferred database, which includes the preferred natural language texts in each category that have been screened out; the query information collection module includes a speech receiving and recognition unit and a natural language text generation unit, and the speech receiving and recognition unit receives the speech and performs speech recognition on the speech input by the user after the user currently inputs the speech; the natural language text generation unit generates the current natural language text after completing the speech recognition.

[0046] The information retrieval planning module includes a text matching unit in the library, a text selection unit to be converted, an SQL conversion unit and an information retrieval unit; the current natural language text is matched with the preferred natural language text in the preferred database through the text matching unit in the library: the semantic similarity between the current natural language text and the preferred natural language text is analyzed, and the semantic similarity and the semantic similarity threshold are compared; if there is no preferred natural language text with a semantic similarity higher than the threshold in the preferred database with the current natural language text, the current natural language text is selected as the text to be converted through the text selection unit to be converted; if there is a preferred natural language text with a semantic similarity higher than the threshold in the preferred database with the current natural language text, the preferred natural language text with the highest semantic similarity to the current natural language text is selected as the text to be converted; the text to be converted is converted into SQL language using the T5 model through the SQL conversion unit; information retrieval is performed using SQL language through the information retrieval unit and the retrieved information is sent to the user.

[0047] The retrieval planning optimization module includes a selection model building unit and a selection method optimization unit; after performing several operations of selecting texts to be converted, the selection model building unit selects the preferred natural language texts selected as the texts to be converted, collects the semantic similarity between the selected texts and the natural language texts generated after the corresponding recognition of the user input speech, calculates the deviation value between the semantic similarity and the semantic similarity threshold, analyzes the matching coefficient of the selected texts and the user requirements of the corresponding input speech, performs fitting training on the deviation value and the matching coefficient, and establishes a selection optimization model for the texts to be converted; the selection method of the texts to be converted is optimized by the selection method optimization unit: the language input by the user is After speech recognition is performed on the voice, a natural language text is generated. If there is no preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than a threshold, the corresponding generated natural language text is selected as the text to be converted; if there is a preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than the threshold, the deviation value between the semantic similarity and the threshold is calculated, the deviation value is input into the optimization model for selecting the text to be converted, and the matching coefficient between the preferred natural language text whose semantic similarity with the corresponding generated natural language text is higher than the threshold and the corresponding user is output, and the preferred natural language text with the highest matching coefficient is selected as the text to be converted.

[0048] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A natural language enhanced data analysis system for big data, characterized by: It includes historical query data collection module, language text screening module, query information collection module, information retrieval planning module and retrieval planning optimization module; The historical query data collection module collects the historical search information of the user and the historical natural language text generated by converting the voice input by the user who needs to perform information retrieval, and classifies and pre-processes the historical natural language text; The language text screening module analyzes the matching between the pre-processed historical natural language text and the user's needs, screens the preferred natural language text and generates a preferred database; The query information collection module collects the voice information of the user who currently needs to perform information retrieval, and converts the voice information into the current natural language text; The information retrieval planning module is used to plan the selection method of the text to be converted, the selected text to be converted is converted into SQL language, and the information retrieval is performed using SQL language; A text selection optimization model for to-be-converted is established through the retrieval planning optimization module to optimize the selection method of the text to-be-converted.

2. The natural language enhanced data analysis system for big data according to claim 1, characterized in that: The historical query data collection module includes a natural language text collection unit, a historical text classification processing unit and a search information collection unit; The natural language text collection unit collects the historical natural language text generated by recognizing the user input voice by the voice recognition technology after the user input voice is recognized each time the user needs to retrieve information in the past; The historical text classification processing unit classifies the historical natural language texts according to semantic similarity; The retrieval information collection unit collects the number of times the user re-enters the voice and the interval time information of the re-entering voice after the historical natural language text is converted into the SQL language and the information retrieved by the SQL language is transmitted to the user terminal.

3. The natural language enhanced data analysis system for big data according to claim 2, characterized in that: The language text screening module includes a demand matching degree analysis unit, a preferred text screening unit and a preferred database generation unit; The matching coefficients of historical natural language texts classified into the same category and user requirements are analyzed by the demand matching degree analysis unit; the matching coefficients of historical natural language texts classified into the same category and user requirements are compared by the preferred text screening unit, and preferred natural language texts are screened out from the historical natural language texts; the preferred natural language texts in each category are screened out by the preferred database generation unit, and a preferred database is generated, wherein the preferred database includes the screened preferred natural language texts in each category; The query information collection module includes a speech receiving and recognition unit and a natural language text generation unit. After the user currently inputs speech, the speech receiving and recognition unit receives the speech and performs speech recognition on the user's input speech; and the natural language text generation unit generates the current natural language text after completing the speech recognition.

4. The natural language enhanced data analysis system for big data according to claim 3, characterized in that: The information retrieval planning module includes a text matching unit in the library, a text selection unit to be converted, a SQL conversion unit and an information retrieval unit; Matching the current natural language text with the preferred natural language text in the preferred database by the text matching unit in the library: analyzing the semantic similarity between the current natural language text and the preferred natural language text, and comparing the semantic similarity with the semantic similarity threshold; If there is no preferred natural language text in the preferred database whose semantic similarity with the current natural language text is higher than the threshold value, the current natural language text is selected as the text to be converted by the text selection unit to be converted; if there is a preferred natural language text in the preferred database whose semantic similarity with the current natural language text is higher than the threshold value, the preferred natural language text with the highest semantic similarity with the current natural language text is selected as the text to be converted; The sql conversion unit converts the text to be converted into sql language using the T5 model; The information retrieval unit uses the SQL language to perform information retrieval and sends the retrieved information to the user.

5. The natural language enhanced data analysis system for big data according to claim 4, characterized in that: The retrieval planning optimization module includes a selection model building unit and a selection mode optimization unit; After performing several operations of selecting texts to be converted, the selection model building unit screens out preferred natural language texts selected as texts to be converted, collects semantic similarity between the screened texts and natural language texts generated after the corresponding recognition of user input speech, calculates the deviation value between the semantic similarity and the semantic similarity threshold, analyzes the matching coefficient between the screened texts and the user requirements of the corresponding input speech, performs fitting training on the deviation value and the matching coefficient, and establishes a text selection optimization model to be converted; The selection method of the text to be converted is optimized by the selection method optimization unit: natural language text is generated after voice recognition is performed on the voice input by the user; if there is no preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than a threshold, the corresponding generated natural language text is selected as the text to be converted; if there is a preferred natural language text in the preferred database whose semantic similarity with the corresponding generated natural language text is higher than the threshold, the deviation value between the semantic similarity and the threshold is calculated, the deviation value is input into the text to be converted selection optimization model, the matching coefficient between the preferred natural language text whose semantic similarity with the corresponding generated natural language text is higher than the threshold and the corresponding user is output, and the preferred natural language text with the highest matching coefficient is selected as the text to be converted.

6. A natural language enhanced data analysis method for big data, characterized by: The following steps are involved: Z100: collects historical search information of users and converts the voice input of users who need to search for information into historical natural language text, and performs classification preprocessing on the historical natural language text; Z101: Analyze the matching between the pre-processed historical natural language text and user needs, select the optimal natural language text and generate the optimal database; Z102: Collect the voice information of the user who currently needs to retrieve information, and convert the voice information into the current natural language text; Z103: Plan the method of selecting the text to be converted, convert the selected text to be converted into SQL language, and use SQL language to perform information retrieval; Z104: Establish an optimization model for the selection of text to be transformed to optimize the selection method of the text to be transformed.

7. The natural language enhanced data analysis method for big data according to claim 6, characterized in that: Step Z100 includes: collecting the historical natural language texts generated after recognizing the user's input speech through speech recognition technology each time information needs to be retrieved in the past, analyzing the semantic similarity between the historical natural language texts, setting the semantic similarity threshold as S, randomly extracting a text from the historical natural language texts, analyzing the semantic similarity between the remaining texts and the extracted text one by one, classifying the texts with a semantic similarity higher than S with the extracted text into the same category, and after the first classification, pre-classifying the remaining unclassified texts in the same way until there are no unclassified texts, transforming the historical natural language texts into sql language, using sql language to retrieve information and transmitting the retrieved information to the user, and collecting the number of times the user re-entered the speech and the interval time between re-entering the speech after receiving the information in the past.

8. The natural language enhanced data analysis method for big data according to claim 7, characterized in that: Step Z101 includes: counting that there are n historical natural language texts classified as random, retrieving the number of times the user re-entered the voice after receiving the information retrieved by converting the corresponding n historical natural language texts into SQL language as L = {L1, L2, ..., Ln}, retrieving the number of times the user re-entered the voice after receiving the information retrieved by converting the i-th historical natural language text into SQL language as Li, and obtaining the interval time set of re-entering the voice as T = {T1, T2, ..., T Li }, T1 represents the interval between the time when the user re-enters the voice for the first time and the time when the information is received. The matching coefficient Ri between the i-th historical natural language text and the user's needs is calculated according to the following formula: Ri=(1 / Li)+[(∑ Li j＝1 (T j )) / (Li)]; j represents the jth time the user re-enters the speech after receiving the information retrieved by transforming the ith historical natural language text into sql language. Calculate the matching coefficients between n historical natural language texts and the user's needs in the same way, compare the n matching coefficients, arrange the n historical natural language texts in descending order of the matching coefficients, screen out the first m historical natural language texts after the arrangement as the preferred natural language texts, where 1 < m < n, screen out the preferred natural language texts in each category, and generate a preferred database.

9. The natural language enhanced data analysis method for big data according to claim 8, characterized in that: Step Z103 includes: after obtaining the current natural language text in step Z102, analyzing the semantic similarity between the current natural language text and the preferred natural language texts, and comparing the semantic similarity with S: if there is no preferred natural language text in the preferred database with a semantic similarity higher than S with the current natural language text, select the current natural language text as the text to be transformed; if there is a preferred natural language text in the preferred database with a semantic similarity higher than S with the current natural language text, select the preferred natural language text with the highest semantic similarity with the current natural language text as the text to be transformed, use the T5 model to transform the text to be transformed into sql language, use sql language to retrieve information and send the retrieved information to the user.

10. The natural language enhanced data analysis method for big data according to claim 9, characterized in that: Step Z104 includes: after performing k times of text selection operations to be converted using the selection method planned in step Z103, a total of r preferred natural language texts selected as texts to be converted are screened out, the semantic similarity set between the r texts and the natural language texts generated after the corresponding recognition of the user input voice is collected as {F1, F2, ..., Fr}, the deviation value set between the semantic similarity and S is calculated as {F1-S, F2-S, ..., Fr-S}, and it is counted that after the e-th text selected is converted into SQL language for information retrieval and the retrieved information is transmitted to the user, the number of times the user re-enters the voice is g, and the interval time set of the g times of re-entering the voice is {t1, t2, ..., t g }, and obtain the matching coefficient We between the filtered e-th text and the user's needs, We=(1 / g)+[(∑ g u＝1 (t u )) / g], u represents the u-th re-input of speech, and the set of matching coefficients between the selected r texts and user needs is {W1, W2, ...We, ...Wr}, generating training data {(F1-S, W1), (F2-S, W2), ...(Fr-S, Wr)}, and establishing the optimization model for selecting texts to be converted after linear fitting of the training data: Y = δ1 * X + δ2; Among them, δ1 and δ2 represent fitting coefficients, X represents the variable referring to the deviation value in the model, and Y represents the variable referring to the matching coefficient in the model. The optimization selection method is: after performing speech recognition on the user's input voice, a natural language text is generated. If there is no preferred natural language text with a semantic similarity higher than S with the corresponding generated natural language text in the preferred database, the corresponding generated natural language text is selected as the text to be converted; if there are H preferred natural language texts with a semantic similarity higher than S with the corresponding generated natural language text in the preferred database, the deviation value between the semantic similarity between the a-th text in the H texts and the corresponding generated natural language text and S is calculated to be A, let X=A, and the matching coefficient between the a-th text and the user's demand is output as δ1*A+δ2, the H deviation values ​​are input into the model one by one, and H matching coefficients are output. The preferred natural language text with the highest matching coefficient is selected from the H texts as the text to be converted.

Images