Text retrieval error correction big data processing system based on Internet platform

Abstract

The invention discloses a text retrieval error correction big data processing system based on an internet platform, which belongs to the technical field of text retrieval error correction and comprises a data source module, a text retrieval module and an error correction module. The data source module is used for determining a text data source and establishing a data source optimization library based on the error correction record; the text retrieval module is used for performing text retrieval, acquiring a text retrieval demand of a user and performing feature analysis on the text retrieval demand to obtain retrieval feature data; performing text retrieval based on each piece of retrieval feature data and a preset feature evaluation model to obtain text data of which the feature evaluation value of each piece of retrieval feature data is 1, and marking the text data as a retrieval text; sorting the retrieval texts to obtain a retrieval list; displaying the retrieval list to the user; the error correction module is used for performing error correction on the retrieval text and establishing an error correction model based on the text data source and the data source optimization library; and performing error correction analysis on the retrieved text through the error correction model.

Description

Technical Field

[0001] The present invention belongs to the technical field of text retrieval and error correction, and specifically relates to a big data processing system for text retrieval and error correction based on an Internet platform. Background Art

[0002] With the rapid development of Internet technology, the amount of text data has increased explosively, posing great challenges to traditional text retrieval and error correction methods. Existing text retrieval systems usually adopt a keyword-based retrieval method, which can quickly locate texts containing specific keywords. However, when dealing with a large amount of text data, the retrieval efficiency and accuracy are often not satisfactory. Therefore, in order to solve the above problems, the present invention provides a big data processing system for text retrieval and error correction based on an Internet platform. Summary of the Invention

[0003] In order to solve the problems existing in the above solutions, the present invention provides a big data processing system for text retrieval and error correction based on an Internet platform.

[0004] The object of the present invention can be achieved by the following technical solutions: A big data processing system for text retrieval and error correction based on an Internet platform includes a data source module, a text retrieval module, and an error correction module; The data source module is used to determine a text data source and establish a data source optimization library based on error correction records.

[0005] Further, the method for determining the text data source includes: Determine each candidate data source, set a corresponding simulated error correction text set and corresponding standard error correction results according to the user's text error correction requirements; Perform error correction simulation on the simulated error correction text set through each candidate data source to obtain error correction simulation data corresponding to each candidate data source, compare the error correction simulation data with the standard error correction results, and determine uncorrected data and error correction error data; Set corresponding uncorrected values and error values according to the uncorrected data and error correction error data; According to the formula Calculate the candidate values of each candidate data source; In the formula: PY is the candidate value; b1 and b2 are both proportionality coefficients, and the value range is 0 < b1 ≤ 1, 0 < b2 ≤ 1; WA is the uncorrected value; WB is the error value; AZ is the comprehensive value; e is the natural constant; Sort each candidate data source in descending order to obtain a recommended list, and the user determines the applied text data source according to the recommended list.

[0006] Further, the method for determining error correction error data includes: Establish an error correction judgment model, the expression of the error correction judgment model is: ; Where: s is the input data, the input data is the error correction simulation data and the standard error correction result; the output data is the error correction judgment value CW(s); The error correction simulation data and the standard error correction results are analyzed by the error correction judgment model to obtain various error correction judgment values; The error correction data is determined according to the error correction judgment value.

[0007] Furthermore, the method for establishing the data source optimization library includes: Obtain error correction records in real time, identify each error correction correction data in the error correction records and the text features corresponding to each error correction correction data; count the number of corrections of each error correction correction data; Integrate the error correction data with the same error correction correction data and text features into unit correction data, and mark the sum of the correction times of the error correction correction data corresponding to the unit correction data as the unit correction value; Identify the same correction data of each unit, and merge and filter the same correction data of each unit; Sort the remaining unit correction data in descending order of unit correction values ​​to obtain a first sequence; Calculate the optimized value of each unit correction data in sequence according to the order of each unit correction data in the first sequence until the optimized value is lower than the threshold value X2; A database is established, and the correction data of each unit whose optimization value is greater than the threshold value X2 is stored in the database, and the current database is marked as a data source optimization library.

[0008] Furthermore, the method for merging and screening the same correction data of each unit includes: Identify the text features of each unit correction data, calculate the similarity between the text features, merge the unit correction data with similarity greater than the threshold value X1, calculate the new unit correction value, and integrate the text features of the new unit correction data; And so on, until there is no unit revision data that meets the merging requirements; the merging screening is completed.

[0009] Furthermore, the calculation method of the optimization value includes: Identify the unit correction value of the unit correction data, and mark the obtained unit correction value as XZ; According to the formula Calculate the corresponding optimization value; Where: YH is the optimized value.

[0010] The text retrieval module is used to perform text retrieval, obtain the user's text retrieval requirements, perform feature analysis on the text retrieval requirements, and obtain various retrieval feature data; perform text retrieval based on various retrieval feature data and a preset feature evaluation model, obtain text data whose feature evaluation values ​​of various retrieval feature data are all 1, and mark them as retrieval text; Sort each search text to obtain a search list; and display the search list to the user.

[0011] Furthermore, the expression of the feature evaluation model is: ; Where: q is the input data, the input data is the retrieval feature data; the output data is the feature evaluation value TP(q) corresponding to the corresponding retrieval feature data.

[0012] Furthermore, the retrieval feature data includes a text body, and a method for determining the text body of the text data includes: Step SA1: Establish a text analysis model and mark the text data of the determined text body as reference data; Step SA2: Collect text data that needs to be determined as a text subject, conduct equivalence evaluation on the text data and reference data, and obtain corresponding equivalence evaluation results, where the equivalence evaluation results include deemed equivalent and not deemed equivalent; Step SA3: When the equivalence evaluation result is deemed to be equal, no text body analysis is performed, and the text body corresponding to the corresponding reference data is marked; When the equivalence assessment result is not considered equivalent, the text data is analyzed through the text analysis model to obtain the corresponding text body; The corresponding text data is marked as reference data, and the process returns to step SA2.

[0013] The error correction module is used to correct the search text, establish an error correction model based on the text data source and the data source optimization library; and perform error correction analysis on the search text through the error correction model.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Through the cooperation among the data source module, text retrieval module and error correction module, intelligent retrieval of text data is realized; at the same time, the text body of the text data is determined by pre-iteration to improve the text retrieval efficiency, and retrieval based on multiple retrieval feature data is convenient for users to improve retrieval accuracy, greatly narrow the retrieval scope and improve retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0017] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, the text retrieval and error correction big data processing system based on the Internet platform includes a data source module, a text retrieval module and an error correction module; The data source module is used to determine the text data source and establish a data source optimization library based on the error correction record. Text data sources such as dictionaries and corpora are used for text error correction; the data source optimization library is the data determined by the user for error correction adjustment based on the error correction record, marked as error correction correction data, which is equivalent to the data that the user is dissatisfied with the error correction based on the text data source and the data adjusted by the user, and is used to replace the corresponding corpus in the text data source later.

[0019] The specific process is as follows: Determine each text data source that has, mark it as a candidate data source, or the user can select the candidate data source by himself; the user sets the corresponding simulated error correction text set and the corresponding standard error correction result according to the possible text error correction requirements, and the standard error correction result is set according to the user's needs; perform error correction simulation on the simulated error correction text set through each candidate data source, and obtain the error correction simulation data corresponding to each candidate data source, the error correction simulation data is the result data of the error correction of the simulated error correction text set based on the candidate data source, such as which data has been corrected, what the error correction result is, etc.; Compare the obtained error correction simulation data with the standard error correction results to determine the uncorrected data and the error correction error data. The uncorrected data is the data that needs to be error-corrected according to the standard error correction results but has not been error-corrected, which can reflect the comprehensiveness of the corpus of the candidate data source. The error correction error data is the data whose error correction results do not meet the requirements compared with the standard error correction results. For different results, if the deviation exceeds the requirements, it is regarded as not meeting the error correction requirements, that is, a certain result deviation can be allowed according to the user's needs. Specifically, the error correction requirements are determined according to the user's needs, and the deviation may not be available. In one embodiment, an error correction judgment model can be established to determine the error correction error data. The expression of the error correction judgment model is , where: s is the input data, and the input data is the error correction simulation data and the standard error correction results; the output data is the error correction judgment value CW(s); analyze the error correction simulation data and the standard error correction results through the error correction judgment model to determine the error correction error data.

[0020] Set the corresponding uncorrected value and error value according to the uncorrected data and the error correction error data. The uncorrected value and the error value are the quantities corresponding to the respective uncorrected data and error correction error data; according to the formula Calculate the candidate value of each candidate data source; where: PY is the candidate value; b1 and b2 are both proportionality coefficients, and the value range is 0 < b1 ≤ 1, 0 < b2 ≤ 1; WA is the uncorrected value; WB is the error value; AZ is the comprehensive value, that is, the quantity that needs to be error-corrected in the standard error correction results; e is the natural constant; Sort each candidate data source in descending order to obtain a recommended list, and send the recommended process to the user. The user determines the text data source to be applied according to the recommended list.

[0021] The method for establishing the data source optimization library includes: Obtain error correction records in real time, identify each error correction data and the text features corresponding to each error correction data in the error correction records. The text features refer to the semantics of the context before and after the error correction data, and use existing semantic recognition technologies for recognition; count the number of corrections of each error correction data. Integrate the error correction data with the same error correction data and text features into unit correction data, that is, include one error correction data as the unit correction data, and mark the sum of the number of corrections of the respective error correction data corresponding to the unit correction data as the unit correction value; An independent error correction data is also regarded as a unit correction data. Identify the same unit correction data and perform combined screening on the same unit correction data; Sort the remaining unit correction data in descending order according to the unit correction value to obtain the first sequence; Calculate the optimized value of each unit correction data in sequence according to the order of each unit correction data in the first sequence until the optimized value is lower than the threshold value X2; A database is established, and the correction data of each unit whose optimization value is greater than the threshold value X2 is stored in the database, and the current database is marked as a data source optimization library.

[0022] The method for merging and screening the same correction data of each unit includes: Identify the text features of each unit correction data, calculate the similarity between each text feature, merge the unit correction data with a similarity greater than a threshold value X1, that is, delete one unit correction data, obtain new unit correction data, and calculate a new unit correction value, set the text features of the new unit correction data, that is, the set of text features corresponding to the merged unit correction data, including each text feature merged before, that is, the subsequent merge judgment needs to all meet the similarity requirements; And so on, until there is no unit revision data that meets the merging requirements; the merging screening is completed.

[0023] The optimization value calculation method includes: Identify the unit correction value of the unit correction data, and mark the obtained unit correction value as XZ; According to the formula Calculate the corresponding optimization value; Where: YH is the optimized value.

[0024] The text retrieval module is used to perform text retrieval and obtain the user's text retrieval needs. The text requirements can have a variety of retrieval feature data, such as keywords, time intervals, text size, text source security requirements, text body, etc., where the text body is an overview of what direction the text is mainly used to describe. The text retrieval requirement template is set according to the platform's needs. The retrieval feature data that can be included in the preset text retrieval requirements are integrated into the retrieval requirement template; and a corresponding feature evaluation model is established based on each retrieval feature. The feature evaluation model is used to evaluate whether the text meets the retrieval feature data. It can be combined with neural network, data recognition and judgment and other technologies to evaluate whether it meets the retrieval feature data; the expression of the feature evaluation model is ; In the formula: q is the input data, the input data is text data; the output data is the feature evaluation value TP(q) corresponding to the corresponding retrieval feature data; Perform feature analysis on the user's text search needs according to each search feature to obtain each search feature data; Perform text retrieval based on each retrieval feature data and feature evaluation model, obtain text data whose feature evaluation value of each retrieval feature data is 1, and mark it as retrieval text; Sort each search text according to the keyword match to obtain a search list; and display the search list to the user.

[0025] In one embodiment, because it is necessary to search from a large amount of data, other search features can be evaluated synchronously during the search. However, in order to evaluate whether the text body meets the search features, it is necessary to first determine the text body of each text, which will affect the search efficiency during the search process. Therefore, in this embodiment, the following method is used to solve this problem: Step SA1: Establish a text analysis model based on a neural network such as a CNN network or a DNN network, where the text analysis model is used to analyze the text and determine its text body; establish a corresponding training set for training manually, where the training set includes input data and output data, where the input data is text data and the output data is the text body; Analyze the text data in the Internet platform through the successfully trained text analysis model to obtain the corresponding text body, and mark the text data of the determined text body as reference data; Step SA2: collect text data that needs to be determined as a text subject, and conduct an equivalence assessment between the identified text data and the reference data, that is, assess whether the identified text data is considered equivalent to the reference data, such as the same data from different sources, partial data in the reference data, etc., and use existing text recognition technology to quickly identify them; obtain corresponding equivalence assessment results, which include considered equivalent and not considered equivalent; Step SA3: When the equivalence evaluation result is deemed to be equal, no text body analysis is performed, and the text body corresponding to the corresponding reference data is marked; When the result of the equivalence assessment is that the two are not considered equivalent, the text analysis model is used to analyze the two to obtain the corresponding text body, and the corresponding text data is marked as reference data; Return to step SA2.

[0026] The error correction module is used to correct the search text, establish an error correction model based on the text data source and the data source optimization library, the error correction model is established based on the neural network, and error correction evaluation is performed based on the text data source and the corpus, words, etc. in the data source optimization library, wherein the data source optimization library is used as a benchmark. If the data source optimization library does not exist, the text data source is used as a benchmark; specifically, the error correction model is established using existing technology; and the search text is analyzed for error correction through the error correction model.

[0027] The above formulas are all calculated by removing dimensions and taking numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0028] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A text retrieval and error correction big data processing system based on the Internet platform, characterized in that: It includes a data source module, a text retrieval module, and an error correction module; The data source module is used to determine the text data source and establish a data source optimization library based on the error correction records; The text retrieval module is used to perform text retrieval, obtain the user's text retrieval requirements, analyze the characteristics of the text retrieval requirements to obtain various retrieval feature data; perform text retrieval based on the various retrieval feature data and a preset feature evaluation model, and obtain the text data with the feature evaluation value of 1 for each retrieval feature data, which is marked as the retrieved text; Sort the retrieved texts to obtain a retrieval list; display the retrieval list to the user; The error correction module is used to correct the retrieved text, establish an error correction model based on the text data source and the data source optimization library; perform error correction analysis on the retrieved text through the error correction model.

2. The text retrieval and error correction big data processing system based on the Internet platform according to claim 1 is characterized in that: The method for determining the text data source includes: Determine each candidate data source, set a corresponding simulated error correction text set and the corresponding standard error correction results according to the user's text error correction requirements; Perform error correction simulation on the simulated error correction text set through each candidate data source to obtain the error correction simulation data corresponding to each candidate data source, compare the error correction simulation data with the standard error correction results, and determine the uncorrected data and the error correction error data; Set the corresponding uncorrected value and error value according to the uncorrected data and the error correction error data; According to the formula Calculate the candidate value of each candidate data source; In the formula: PY is the candidate value; b1 and b2 are both proportionality coefficients, and the value range is 0 < b1 ≤ 1, 0 < b2 ≤ 1; WA is the uncorrected value; WB is the error value; AZ is the comprehensive value; e is the natural constant; Sort each candidate data source in descending order to obtain a recommended list, and the user determines the applied text data source according to the recommended list.

3. The text retrieval and error correction big data processing system based on the Internet platform according to claim 2 is characterized in that: The method for determining the error correction error data includes: Establish an error correction judgment model, and the expression of the error correction judgment model is: ； In the formula: s is the input data, and the input data is the error correction simulation data and the standard error correction result; the output data is the error correction judgment value CW(s); Analyze the error correction simulation data and the standard error correction result through the error correction judgment model to obtain each error correction judgment value; Determine the error correction error data according to the error correction judgment value.

4. The text retrieval and error correction big data processing system based on the Internet platform according to claim 1 is characterized in that: The method for establishing the data source optimization library includes: Obtain the error correction records in real time, identify each error correction and correction data and the corresponding text features in the error correction records; count the number of corrections for each error correction and correction data; Integrate the error correction and correction data with the same error correction and correction data and text features into unit correction data, and mark the sum of the number of corrections of each error correction and correction data corresponding to the unit correction data as the unit correction value; Identify the same unit correction data, and perform merging and screening on the same unit correction data; Sort the remaining unit correction data in descending order according to the unit correction value to obtain a first sequence; Calculate the optimization value of each unit correction data in turn according to the sorting of each unit correction data in the first sequence until the optimization value is lower than the threshold X2; Establish a database, store the unit correction data with the optimization value greater than the threshold X2 in the database, and mark the current database as the data source optimization library.

5. The text retrieval and error correction big data processing system based on the Internet platform according to claim 4 is characterized in that: The method for merging and screening the same unit correction data includes: Identify the text features of each unit correction data, calculate the similarity between the text features, merge the unit correction data with similarity greater than the threshold value X1, calculate the new unit correction value, and integrate the text features of the new unit correction data; And so on, until there is no unit revision data that meets the merging requirements; the merging screening is completed.

6. The text retrieval and error correction big data processing system based on the Internet platform according to claim 4 is characterized in that: The optimization value calculation method includes: Identify the unit correction value of the unit correction data, and mark the obtained unit correction value as XZ; According to the formula Calculate the corresponding optimization value; Where: YH is the optimized value.

7. The text retrieval and error correction big data processing system based on the Internet platform according to claim 1 is characterized in that: The expression for evaluating the model is: ； Where: q is the input data, the input data is the retrieval feature data; the output data is the feature evaluation value TP(q) corresponding to the corresponding retrieval feature data.

8. The text retrieval and error correction big data processing system based on the Internet platform according to claim 1 is characterized in that: The retrieval feature data includes a text body, and a method for determining the text body of the text data includes: Step SA1: Establish a text analysis model and mark the text data of the determined text body as reference data; Step SA2: Collect text data that needs to be determined as a text subject, conduct equivalence evaluation on the text data and reference data, and obtain corresponding equivalence evaluation results, where the equivalence evaluation results include deemed equivalent and not deemed equivalent; Step SA3: When the equivalence evaluation result is deemed to be equal, no text body analysis is performed, and the text body corresponding to the corresponding reference data is marked; When the equivalence assessment result is not considered equivalent, the text data is analyzed through the text analysis model to obtain the corresponding text body; The corresponding text data is marked as reference data, and the process returns to step SA2.

Images