A query system and method for text big data
Through semantic acquisition, word frequency statistics and type analysis of the text big data query system, combined with user feedback control, the problem of inefficient text big data query in the existing technology is solved, and an efficient query process is realized.
Patent Information
- Application Number
- CN202410692052.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing text big data query is inefficient and cannot meet market demand.
A text big data query system was designed, including semantic acquisition module, statistical analysis module, type confirmation module, comparison query module and feedback control module. Through real-time acquisition and semantic acquisition, word frequency statistics, type analysis and user feedback control, query is improved. efficiency.
It effectively improves the efficiency of text big data query and can better meet market demand.
Smart Images

Figure CN118467679B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text big data, and in particular to a query system and method for text big data. Background Art
[0002] Text big data is an important branch in the field of big data. It involves the collection, processing, analysis and application of text data. Text big data analysis can provide an in-depth understanding of massive text data, thus playing an important role in multiple fields.
[0003] The Chinese patent with publication number CN114461817A discloses a graph analysis method for text big data, including the following steps: collecting, screening and publishing text big data, and annotating the screened text big data; selecting a graph analysis object from the screened text big data, and extracting knowledge triples from the selected graph analysis object through a model; searching the extracted knowledge triples to obtain all numbered triples, taking the smallest number, and performing knowledge reasoning on the smallest numbered triple to obtain a knowledge graph. By analyzing the text big data information and constructing a knowledge graph, not only the complexity of text big data analysis is reduced, the interpretation of text big data is also more convenient, the accuracy of text big data analysis is improved, and the matching degree between enterprise data and text big data is further improved; however, the patent has the following defects:
[0004] When querying existing text big data, all text big data needs to be queried and analyzed, which leads to low efficiency of text big data query and cannot meet market demand well. Summary of the invention
[0005] The purpose of the present invention is to provide a query system and method for text big data, which can effectively improve the query efficiency of text big data, better meet market demand, and solve the problems raised in the above-mentioned background technology.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] A query system for text big data, comprising:
[0008] The semantic acquisition module is used to collect the text big data to be queried in real time, and to obtain the semantics of the text big data collected in real time to determine the semantics of the text big data to be queried;
[0009] A statistical analysis module is used to perform retrieval statistics on the semantics of the text big data to be queried, determine the number of text words based on the text words, and determine the keywords of the text big data to be queried based on the text words with the largest number of text words;
[0010] A type confirmation module is used to perform type analysis on the stored text big data type and the text big data keywords to be queried based on the text big data query requirements, and determine the text big data type to be queried;
[0011] A comparison query module is used to compare the text big data type to be queried, determine the text big data query type, and query the text big data query type to determine the text big data query data;
[0012] The feedback control module is used to provide user feedback on the text big data query data, determine the user feedback situation based on the text big data query data, perform query control on the text big data query data based on the user feedback situation, and intelligently control the text big data query data.
[0013] Preferably, the semantic acquisition module includes:
[0014] A data collection unit, used for collecting text big data to be queried in real time;
[0015] Based on the text big data query demand, the text big data to be queried is collected in real time, and the text big data collected in real time is determined;
[0016] A semantic acquisition unit, used to acquire semantics from text big data collected in real time;
[0017] Obtaining text big data collected in real time;
[0018] Semantic acquisition of text big data collected in real time;
[0019] Construct the semantic paradigm model of query objects involved in the query analysis process of text big data;
[0020] The semantic description level of each object is characterized by semantic description specifications of different levels of paradigms;
[0021] Among them, the semantic paradigm model includes the text content semantic paradigm model CSNF, the query instruction semantic paradigm model ISNF, the query method semantic paradigm model MSNF and the query workflow semantic paradigm model FSNF;
[0022] Different levels of paradigm transformation criteria are designed for different query objects to obtain a more accurate semantic description and determine the semantics of the text big data to be queried.
[0023] Preferably, the statistical analysis module includes:
[0024] A retrieval statistics unit is used to perform retrieval statistics on the semantics of the text big data to be queried;
[0025] Based on the query requirements of text big data, the semantics of the text big data to be queried is retrieved and counted;
[0026] In the process of retrieving the semantics of the text big data to be queried, the number of occurrences of each text word in the semantics of the text big data to be queried is counted to determine the number of text words based on the text words;
[0027] A keyword analysis unit for analyzing the number of times a text word is used based on the text word;
[0028] Get the number of text words based on the text words;
[0029] Analyze the number of times text words are used based on the text words;
[0030] Find the text word with the most occurrences;
[0031] Based on the text words that appear the most times, the text big data keywords to be queried are determined.
[0032] Preferably, the type confirmation module includes:
[0033] Type storage unit, used to store text large data type;
[0034] Each text big data type contains text big data that matches the text big data type;
[0035] An index extraction unit, used to index and extract the stored text big data type;
[0036] Based on the text big data query requirements, the stored text big data types are indexed and the indexed text big data types are extracted;
[0037] An analysis and confirmation unit, used to perform type analysis on the text big data keywords to be queried and determine the type of the text big data;
[0038] Get keywords of text big data;
[0039] Based on the text big data query requirements, the stored text big data types are indexed and extracted, and the indexed and extracted text big data types are analyzed with the text big data keywords to be queried to determine the text big data type to be queried.
[0040] Preferably, the indexed and extracted text big data types and text big data keywords are subjected to type analysis, and the following operations are performed:
[0041] When the indexed and extracted text big data type matches the text big data keyword to be queried, the text big data type is the indexed and extracted text big data type;
[0042] When the indexed and extracted text big data type does not match the text big data keyword to be queried, the text big data type is not the indexed and extracted text big data type;
[0043] When the text big data type is not the indexed and extracted text big data type, the next text big data type is indexed and extracted in sequence, and the next text big data type indexed and extracted in sequence is subjected to type analysis with the text big data keyword to be queried until the next text big data type indexed and extracted in sequence matches the text big data keyword to be queried, and the text big data type is the next text big data type indexed and extracted in sequence.
[0044] Preferably, the comparison query module includes:
[0045] A type comparison unit is used to compare the type of text big data to be queried and determine the text big data query type;
[0046] Get the text big data type to be queried;
[0047] Based on the text big data query requirements and the stored text big data types, the text big data types to be queried are compared to determine the text big data query type;
[0048] A data query unit, used to query text big data query types;
[0049] Obtaining a text big data query type and querying the text big data query type;
[0050] Determine text big data query data.
[0051] Preferably, the feedback control module includes:
[0052] A data feedback unit, used for providing user feedback on text big data query data;
[0053] Get text big data query data;
[0054] Based on the text big data query data, obtain the user's feedback on the text big data query data;
[0055] Determine user feedback based on text big data query data;
[0056] A query control unit, used to query and control text big data query data;
[0057] Obtain user feedback based on text big data query data;
[0058] Based on user feedback, query control is performed on text big data query data, and text big data query data is intelligently controlled.
[0059] Preferably, determining whether the indexed and extracted text big data types match the text big data keywords to be queried includes:
[0060] Extract the first keyword of the index and the extracted text big data type;
[0061] The keyword of the text big data to be queried is used as the second keyword;
[0062] Constructing a target word library based on the sample keywords, and storing the target word library in a storage module according to the semantic hierarchy; determining a first storage position of the first keyword in the storage module and a second storage position of the second keyword in the storage module;
[0063] According to the first storage position and the second storage position, the position difference value between the first keyword and the second keyword is calculated:
[0064]
[0065] in, is the position difference value between the first keyword and the second keyword; is the first keyword; is the second keyword; is the first storage location; is a second storage location; is the number of locations in the storage module where the keywords are stored;
[0066] According to the position difference value of the first keyword and the second keyword, the matching degree of the first keyword and the second keyword is calculated:
[0067]
[0068] in, is the matching degree between the first keyword and the second keyword; The first keyword The value of the substring; The second keyword The value of the substring; is the number of substrings of the first keyword, and is also the number of substrings of the second keyword, and both are equal; is a natural constant;
[0069] The matching degree is compared with the preset matching degree threshold. When it is determined that the matching degree is greater than or equal to the preset matching degree threshold, it indicates that the indexed and extracted text big data type matches the text big data keyword to be queried; otherwise, it indicates that the indexed and extracted text big data type does not match the text big data keyword to be queried.
[0070] Preferably, it also includes: a data preprocessing module, which is used to:
[0071] Before the semantic acquisition module performs semantic acquisition on the text big data collected in real time, the local preservation projection LPP algorithm is used to perform dimensionality reduction processing on the collected text big data to obtain dimensionality reduction data;
[0072] Regularize the dimension-reduced data based on the adapted regular expression to obtain regularized data;
[0073] Obtain the characteristic string of the regularized data, match it through the matching automaton, filter out the unmatched characteristic strings, and remove them to obtain the corrected regularized data;
[0074] The regularized data is corrected by The clustering method performs clustering processing to obtain several cluster sets;
[0075] Calculate the standard deviation of the corresponding cluster set according to the data in each cluster set, compare it with the preset standard deviation, and select the cluster set with a standard deviation greater than the preset standard deviation as the cluster set to be processed;
[0076] Obtain type information of the cluster set to be processed, query the preset data cleaning rule database according to the type information, obtain the corresponding data cleaning rule, and perform data cleaning according to the corresponding data cleaning rule;
[0077] In the data cleaning process, after the first data in the cluster set to be processed is cleaned based on the data cleaning rule, first difference information of the first data before and after the data cleaning is determined;
[0078] Determine a first cleaning indicator to be adjusted for the data cleaning rule according to the first difference information;
[0079] Obtain second difference information between a feature vector of second data in the cluster set to be processed and a feature vector of the first data;
[0080] Determine a second cleaning indicator to be adjusted for the data cleaning rule according to the second difference information;
[0081] The data cleaning rule is adjusted according to the first cleaning index to be adjusted and the second cleaning index to be adjusted to obtain an adjusted data cleaning rule, and data cleaning is performed on the second data in the to-be-processed cluster set based on the adjusted data cleaning rule;
[0082] Repeat the data cleaning process of the previous data to adjust the data cleaning rules of the next data to obtain the preprocessed text big data.
[0083] According to another aspect of the present invention, a text big data query method is provided, which is implemented based on a text big data query system as described above and includes the following steps:
[0084] S1: Based on the text big data query demand, the text big data to be queried is collected in real time, and the semantics of the text big data collected in real time are obtained to determine the semantics of the text big data to be queried;
[0085] S2: Based on the text big data query demand, in the process of retrieving the text big data semantics to be queried, the number of occurrences of each text word in the text big data semantics to be queried is counted, the number of text words based on the text words is determined, and the text big data keywords to be queried are determined based on the text words with the largest number of occurrences;
[0086] S3: Based on the text big data query requirements, perform type analysis on the stored text big data types and the text big data keywords to be queried to determine the text big data type to be queried;
[0087] S4: comparing the text big data type to be queried to determine the text big data query type, and querying the text big data query type to determine the text big data query data;
[0088] S5: Based on the text big data query data, obtain user feedback on the text big data query data, and based on the user feedback, perform query control on the text big data query data to intelligently control the text big data query data.
[0089] Compared with the prior art, the present invention has the following beneficial effects:
[0090] The present invention collects text big data to be queried in real time, and obtains semantics of the text big data collected in real time to determine the semantics of the text big data to be queried. In the process of retrieving the semantics of the text big data to be queried, the number of occurrences of each text word in the text big data semantics to be queried is counted to determine the number of text words based on the text words, and the text big data keywords to be queried are determined based on the text words with the largest number of occurrences of the text words. Based on the text big data query demand, the stored text big data type and the text big data keywords to be queried are analyzed to determine the text big data type to be queried, the text big data type to be queried is compared to determine the text big data query type, and the text big data query type is queried to determine the text big data query data, and the text big data query type is queried to determine the text big data query data, and the user feedback on the text big data query data is obtained based on the text big data query data, and the text big data query data is query controlled based on the user feedback, and the text big data query data is intelligently controlled, which can effectively improve the text big data query efficiency and better meet the market demand. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] Figure 1 It is a module principle diagram of the text big data query system of the present invention;
[0092] Figure 2 The present invention is a flowchart of the text big data query method. DETAILED DESCRIPTION
[0093] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0094] In order to solve the problem that the existing text big data needs to be queried and analyzed when querying, resulting in low efficiency of text big data query and failure to meet market demand, please refer to Figure 1-Figure 2 , this embodiment provides the following technical solutions:
[0095] A text big data query system includes: a semantic acquisition module, a statistical analysis module, a type confirmation module, a comparison query module and a feedback control module.
[0096] It should be noted that through the mutual communication between the semantic acquisition module, the statistical analysis module, the type confirmation module, the comparison query module and the feedback control module, the efficiency of text big data query can be effectively improved and the market demand can be better met.
[0097] The semantic acquisition module is used to collect the text big data to be queried in real time, and to obtain the semantics of the text big data collected in real time, so as to determine the semantics of the text big data to be queried;
[0098] In this embodiment, the semantic acquisition module includes:
[0099] A data collection unit, used for collecting text big data to be queried in real time;
[0100] Based on the text big data query demand, the text big data to be queried is collected in real time, and the text big data collected in real time is determined;
[0101] A semantic acquisition unit, used to acquire semantics from text big data collected in real time;
[0102] Obtaining text big data collected in real time;
[0103] Semantic acquisition of text big data collected in real time;
[0104] Construct the semantic paradigm model of query objects involved in the query analysis process of text big data;
[0105] The semantic description level of each object is characterized by semantic description specifications of different levels of paradigms;
[0106] Among them, the semantic paradigm model includes the text content semantic paradigm model CSNF, the query instruction semantic paradigm model ISNF, the query method semantic paradigm model MSNF and the query workflow semantic paradigm model FSNF;
[0107] Different levels of paradigm transformation criteria are designed for different query objects to obtain a more accurate semantic description and determine the semantics of the text big data to be queried.
[0108] The statistical analysis module is used to perform retrieval statistics on the semantics of the text big data to be queried, determine the number of text words based on the text words, and determine the keywords of the text big data to be queried based on the text words with the largest number of text words;
[0109] In this embodiment, the statistical analysis module includes:
[0110] A retrieval statistics unit is used to perform retrieval statistics on the semantics of the text big data to be queried;
[0111] Based on the query requirements of text big data, the semantics of the text big data to be queried is retrieved and counted;
[0112] In the process of retrieving the semantics of the text big data to be queried, the number of occurrences of each text word in the semantics of the text big data to be queried is counted to determine the number of text words based on the text words;
[0113] A keyword analysis unit for analyzing the number of times a text word is used based on the text word;
[0114] Get the number of text words based on the text words;
[0115] Analyze the number of times text words are used based on the text words;
[0116] Find the text word with the most occurrences;
[0117] Based on the text words that appear the most times, the text big data keywords to be queried are determined.
[0118] Among them, the type confirmation module is used to perform type analysis on the stored text big data type and the text big data keywords to be queried based on the text big data query requirements, and determine the text big data type to be queried;
[0119] In this embodiment, the type confirmation module includes:
[0120] Type storage unit, used to store text large data type;
[0121] Each text big data type contains text big data that matches the text big data type;
[0122] An index extraction unit, used to index and extract the stored text big data type;
[0123] Based on the text big data query requirements, the stored text big data types are indexed and the indexed text big data types are extracted;
[0124] An analysis and confirmation unit, used to perform type analysis on the text big data keywords to be queried and determine the type of the text big data;
[0125] Get keywords of text big data;
[0126] Based on the text big data query requirements, the stored text big data types are indexed and extracted, and the indexed and extracted text big data types are analyzed with the text big data keywords to be queried to determine the text big data type to be queried.
[0127] In this embodiment, the indexed and extracted text big data types and text big data keywords are subjected to type analysis, and the following operations are performed:
[0128] When the indexed and extracted text big data type matches the text big data keyword to be queried, the text big data type is the indexed and extracted text big data type;
[0129] When the indexed and extracted text big data type does not match the text big data keyword to be queried, the text big data type is not the indexed and extracted text big data type;
[0130] When the text big data type is not the indexed and extracted text big data type, the next text big data type is indexed and extracted in sequence, and the next text big data type indexed and extracted in sequence is subjected to type analysis with the text big data keyword to be queried until the next text big data type indexed and extracted in sequence matches the text big data keyword to be queried, and the text big data type is the next text big data type indexed and extracted in sequence.
[0131] Among them, the comparison query module is used to compare the text big data type to be queried, determine the text big data query type, and query the text big data query type to determine the text big data query data;
[0132] In this embodiment, the comparison query module includes:
[0133] A type comparison unit is used to compare the type of text big data to be queried and determine the text big data query type;
[0134] Get the text big data type to be queried;
[0135] Based on the text big data query requirements and the stored text big data types, the text big data types to be queried are compared to determine the text big data query type;
[0136] A data query unit, used to query text big data query types;
[0137] Obtaining a text big data query type and querying the text big data query type;
[0138] Determine text big data query data.
[0139] Among them, the feedback control module is used to provide user feedback on the text big data query data, determine the user feedback situation based on the text big data query data, perform query control on the text big data query data based on the user feedback situation, and intelligently control the text big data query data.
[0140] In this embodiment, the feedback control module includes:
[0141] A data feedback unit, used for providing user feedback on text big data query data;
[0142] Get text big data query data;
[0143] Based on the text big data query data, obtain the user's feedback on the text big data query data;
[0144] Determine user feedback based on text big data query data;
[0145] A query control unit, used to query and control text big data query data;
[0146] Obtain user feedback based on text big data query data;
[0147] Based on user feedback, query control is performed on text big data query data, and text big data query data is intelligently controlled.
[0148] Preferably, determining whether the indexed and extracted text big data types match the text big data keywords to be queried includes:
[0149] Extract the first keyword of the index and the extracted text big data type;
[0150] The keyword of the text big data to be queried is used as the second keyword;
[0151] Constructing a target word library based on the sample keywords, and storing the target word library in a storage module according to the semantic hierarchy; determining a first storage position of the first keyword in the storage module and a second storage position of the second keyword in the storage module;
[0152] According to the first storage position and the second storage position, the position difference value between the first keyword and the second keyword is calculated:
[0153]
[0154] in, is the position difference value between the first keyword and the second keyword; is the first keyword; is the second keyword; is the first storage location; is a second storage location; is the number of locations in the storage module where the keywords are stored;
[0155] According to the position difference value of the first keyword and the second keyword, the matching degree of the first keyword and the second keyword is calculated:
[0156]
[0157] in, is the matching degree between the first keyword and the second keyword; The first keyword The value of the substring; The second keyword The value of the substring; is the number of substrings of the first keyword, and is also the number of substrings of the second keyword, and both are equal; is a natural constant;
[0158] The matching degree is compared with the preset matching degree threshold. When it is determined that the matching degree is greater than or equal to the preset matching degree threshold, it indicates that the indexed and extracted text big data type matches the text big data keyword to be queried; otherwise, it indicates that the indexed and extracted text big data type does not match the text big data keyword to be queried.
[0159] Working principle of the above technical solution: In this embodiment, extracting the first keyword of the index and the extracted text big data type involves natural language processing (NLP) technology, and dividing the text into separate words or phrases. This is done by using word segmentation tools (such as jieba word segmenter, Stanford NLP, etc.). Extract keywords from the results after word segmentation. Including methods based on TF-IDF (term frequency-inverse document frequency), methods based on TextRank, methods based on topic models (such as LDA), etc.
[0160] In this embodiment, a target word library is constructed based on sample keywords. For example, the sample keywords can be keywords of various text big data types, and the target word library is a dictionary constructed from multiple sample keywords. The target word library is stored in a storage module according to the semantic level, and a storage sequence is generated according to the feature vectors of different words and stored in the storage module. The storage locations of different keywords are also different.
[0161] In this embodiment, the position difference value between the first keyword and the second keyword is calculated based on the first storage position and the second storage position. The position difference value can preliminarily indicate the semantic distance between the first keyword and the second keyword, which serves as an indicator for determining whether there is a match, but it is not accurate enough.
[0162] In this embodiment, the matching degree between the first keyword and the second keyword is calculated based on the position difference value between the first keyword and the second keyword, and the numerically processed values of the substrings included in the first keyword and the second keyword are further considered, and precise calculations are performed to accurately calculate the matching degree between the first keyword and the second keyword.
[0163] The beneficial effects of the above technical solution are: extracting the first keyword of the index and the extracted text big data type; taking the text big data keyword to be queried as the second keyword; calculating the position difference value between the first keyword and the second keyword, and then calculating the matching degree between the first keyword and the second keyword, thereby improving the accuracy of judging the matching degree and the preset matching degree threshold value, thereby improving the judgment of whether the index and the extracted text big data type match the text big data keyword to be queried.
[0164] Preferably, it also includes: a data preprocessing module, which is used to:
[0165] Before the semantic acquisition module performs semantic acquisition on the text big data collected in real time, the local preservation projection LPP algorithm is used to perform dimensionality reduction processing on the collected text big data to obtain dimensionality reduction data;
[0166] Regularize the dimension-reduced data based on the adapted regular expression to obtain regularized data;
[0167] Obtain the characteristic string of the regularized data, match it through the matching automaton, filter out the unmatched characteristic strings, and remove them to obtain the corrected regularized data;
[0168] The regularized data is corrected by The clustering method performs clustering processing to obtain several cluster sets;
[0169] Calculate the standard deviation of the corresponding cluster set according to the data in each cluster set, compare it with the preset standard deviation, and select the cluster set with a standard deviation greater than the preset standard deviation as the cluster set to be processed;
[0170] Obtain type information of the cluster set to be processed, query the preset data cleaning rule database according to the type information, obtain the corresponding data cleaning rule, and perform data cleaning according to the corresponding data cleaning rule;
[0171] In the data cleaning process, after the first data in the cluster set to be processed is cleaned based on the data cleaning rule, first difference information of the first data before and after the data cleaning is determined;
[0172] Determine a first cleaning indicator to be adjusted for the data cleaning rule according to the first difference information;
[0173] Obtain second difference information between a feature vector of second data in the cluster set to be processed and a feature vector of the first data;
[0174] Determine a second cleaning indicator to be adjusted for the data cleaning rule according to the second difference information;
[0175] The data cleaning rule is adjusted according to the first cleaning index to be adjusted and the second cleaning index to be adjusted to obtain an adjusted data cleaning rule, and data cleaning is performed on the second data in the to-be-processed cluster set based on the adjusted data cleaning rule;
[0176] Repeat the data cleaning process of the previous data to adjust the data cleaning rules of the next data to obtain the preprocessed text big data.
[0177] Working principle of the above technical solution: In this embodiment, the locality preserving projection LPP algorithm is used to reduce the dimension of the collected text big data to obtain the reduced dimension data, including: Locality Preserving Projections (LPP) is a linear approximation of a graph-based nonlinear dimensionality reduction method, similar to the linear version of principal component analysis (PCA) and linear discriminant analysis (LDA), but it focuses more on maintaining the local structure of the data. Data preprocessing: Perform necessary preprocessing work on the text data, such as word segmentation, stop word removal, stem extraction, etc., to convert the text data into a vector representation. Construct a similarity matrix: Construct a similarity matrix of the data samples based on the similarity measure between the data samples. In text data processing, methods such as bag of words model and TF-IDF can be used to calculate the similarity between samples. Construct a weight matrix: Construct a weight matrix based on the similarity matrix to maintain the characteristics of the local structure. Usually, the Gaussian kernel function or K nearest neighbor method can be used to determine the value of the weight matrix. Construct a Laplace matrix: Construct a Laplace matrix based on the weight matrix, including the difference between the degree matrix and the adjacency matrix. Solve the generalized eigenvalue problem: By solving the generalized eigenvalue problem, we can obtain the eigenvectors corresponding to the smallest K eigenvalues, which constitute a new data space. Dimensionality reduction: Map the original data to a new low-dimensional space spanned by the selected K eigenvectors, that is, obtain the reduced-dimensional data.
[0178] In this embodiment, the dimension reduction data is regularized based on the adapted regular expression to obtain regularized data, and one or more regular expressions are written according to the processing requirements. For example, you want to delete all non-alphabetic characters, or keep only specific patterns. Use a regular expression library (such as the re module) in a programming language (such as Python) to match and replace patterns in text features.
[0179] In this embodiment, the feature string of the regularized data is obtained, matched by a matching automaton, unmatched feature strings are screened out, and eliminated to obtain corrected regularized data, including: defining feature strings: it is necessary to clarify which fields or data items can be considered as feature strings. Constructing a matching automaton: constructing a matching automaton according to certain rules or patterns. These rules come from regular expressions, known pattern sets, or other forms of string matching logic. Matching feature strings: using the constructed matching automaton to match feature strings in the regularized data. Filtering unmatched items: based on the matching results, those unmatched feature strings are screened out. Elimination: removing unmatched feature strings from the data set (or marking them as invalid, ignored, etc.). Obtaining corrected regularized data: Finally, the corrected regularized data is obtained, which only contains feature strings that match the matching automaton. Data noise reduction is achieved.
[0180] In this embodiment, the regularized data is corrected by The clustering method performs clustering processing to obtain several cluster sets, including: Selecting the k value: Determine how many clusters the data should be divided into (i.e. the k value). Initialize the cluster center: The algorithm needs to initialize k cluster centers. In scikit-learn, specify the method for selecting the initial center points (such as random selection or initialization based on K-Means++). Execute Clustering: performed using the KMeans class from the scikit-learn library Clustering. Pass the data to the fit method and specify the value of k and other optional parameters (such as initialization method, maximum number of iterations, etc.). Get clustering results: After clustering is completed, you can use the labels_ attribute to get the cluster label of each data point, or use the cluster_centers_ attribute to get the center point of each cluster. Evaluate clustering effect (optional): Use some clustering evaluation indicators (such as silhouette coefficient, Calinski-Harabasz index, etc.) to evaluate the effect of clustering. Get several accurate cluster sets.
[0181] In this embodiment, the first cleaning indicator to be adjusted for the data cleaning rule is determined according to the first difference information, and the adjustment is made according to the cleaning effect of the first data, such as whether the cleaning rule needs to be simplified to improve the cleaning rate. In this embodiment, the second cleaning indicator to be adjusted for the data cleaning rule is determined according to the second difference information, and the data cleaning rule is fine-tuned according to the difference between the two data, such as whether segmented cleaning is needed for longer data based on the different data lengths.
[0182] The beneficial effects of the above technical solution are: Before the semantic acquisition module performs semantic acquisition on the text big data collected in real time, the text big data is processed by dimensionality reduction, the data is processed by dimensionality reduction, the calculation complexity is reduced, and the calculation rate is improved. Based on regularization processing and matching, abnormal data is proposed to achieve data noise reduction, and the regularized data is corrected by The clustering method is used to perform clustering processing, realize data sorting, calculate the standard deviation of the cluster set, and compare it with the preset standard deviation. According to the comparison result, the cluster set to be processed is determined as the type of data that needs further data processing. For the cluster set to be processed, the corresponding data cleaning rule is selected for data cleaning. During the data cleaning process, the corresponding data cleaning rule is continuously updated and adjusted according to the cleaning results to improve the accuracy of data cleaning and the effect of data cleaning, thereby obtaining accurate text big data.
[0183] In order to better present the query process of text big data, this embodiment now provides a query method for text big data, which is implemented based on a text big data query system as described above and includes the following steps:
[0184] S1: Based on the text big data query demand, the text big data to be queried is collected in real time, and the semantics of the text big data collected in real time are obtained to determine the semantics of the text big data to be queried;
[0185] S2: Based on the text big data query demand, in the process of retrieving the text big data semantics to be queried, the number of occurrences of each text word in the text big data semantics to be queried is counted, the number of text words based on the text words is determined, and the text big data keywords to be queried are determined based on the text words with the largest number of occurrences;
[0186] S3: Based on the text big data query requirements, perform type analysis on the stored text big data types and the text big data keywords to be queried to determine the text big data type to be queried;
[0187] S4: comparing the text big data type to be queried to determine the text big data query type, and querying the text big data query type to determine the text big data query data;
[0188] S5: Based on the text big data query data, obtain user feedback on the text big data query data, and based on the user feedback, perform query control on the text big data query data to intelligently control the text big data query data.
[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0190] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A text big data query system, characterized in that: include: The semantic acquisition module is used to collect the text big data to be queried in real time, and to obtain the semantics of the text big data collected in real time to determine the semantics of the text big data to be queried; A statistical analysis module is used to perform retrieval statistics on the semantics of the text big data to be queried, determine the number of text words based on the text words, and determine the keywords of the text big data to be queried based on the text words with the largest number of text words; A type confirmation module is used to perform type analysis on the stored text big data type and the text big data keywords to be queried based on the text big data query requirements, and determine the text big data type to be queried; A comparison query module is used to compare the text big data type to be queried, determine the text big data query type, and query the text big data query type to determine the text big data query data; A feedback control module is used to provide user feedback on the text big data query data, determine the user feedback situation based on the text big data query data, perform query control on the text big data query data based on the user feedback situation, and intelligently control the text big data query data; The type confirmation module comprises: Type storage unit, used to store text large data type; Each text big data type contains text big data that matches the text big data type; An index extraction unit, used to index and extract the stored text big data type; Based on the text big data query requirements, the stored text big data types are indexed and the indexed text big data types are extracted; An analysis and confirmation unit, used to perform type analysis on the text big data keywords to be queried and determine the type of the text big data; Get keywords of text big data; Based on the text big data query requirements, the stored text big data types are indexed and extracted, and the indexed and extracted text big data types are analyzed with the text big data keywords to be queried to determine the text big data type to be queried; Perform type analysis on the indexed and extracted text big data types and text big data keywords, and perform the following operations: When the indexed and extracted text big data type matches the text big data keyword to be queried, the text big data type is the indexed and extracted text big data type; When the indexed and extracted text big data type does not match the text big data keyword to be queried, the text big data type is not the indexed and extracted text big data type; When the text big data type is not the indexed and extracted text big data type, the next text big data type is indexed and extracted in sequence, and the next text big data type indexed and extracted in sequence is subjected to type analysis with the text big data keyword to be queried, until the next text big data type indexed and extracted in sequence matches the text big data keyword to be queried, then the text big data type is the next text big data type indexed and extracted in sequence; Determine whether the indexed and extracted text big data types match the text big data keywords to be queried, including: Extract the first keyword of the index and the extracted text big data type; The keyword of the text big data to be queried is used as the second keyword; Constructing a target word library based on the sample keywords, and storing the target word library in a storage module according to the semantic hierarchy; determining a first storage position of the first keyword in the storage module and a second storage position of the second keyword in the storage module; According to the first storage position and the second storage position, the position difference value between the first keyword and the second keyword is calculated: Wherein, D(t1, t2) is the position difference value between the first keyword and the second keyword; t1 is the first keyword; t2 is the second keyword; L1 is the first storage position; L2 is the second storage position; h is the number of positions storing keywords in the storage module; According to the position difference value of the first keyword and the second keyword, the matching degree of the first keyword and the second keyword is calculated: Where P is the matching degree between the first keyword and the second keyword; t 1i is the value of the i-th substring in the first keyword; t 2i is the value of the ith substring in the second keyword; k is the number of substrings of the first keyword, which is also the number of substrings of the second keyword, and the two are equal; e is a natural constant; The matching degree is compared with the preset matching degree threshold. When it is determined that the matching degree is greater than or equal to the preset matching degree threshold, it indicates that the indexed and extracted text big data type matches the text big data keyword to be queried; otherwise, it indicates that the indexed and extracted text big data type does not match the text big data keyword to be queried.
2. A text big data query system according to claim 1, characterized in that: The semantic acquisition module includes: A data collection unit, used for collecting text big data to be queried in real time; Based on the text big data query demand, the text big data to be queried is collected in real time, and the text big data collected in real time is determined; A semantic acquisition unit, used to acquire semantics from text big data collected in real time; Obtaining text big data collected in real time; Semantic acquisition of text big data collected in real time; Construct the semantic paradigm model of query objects involved in the query analysis process of text big data; The semantic description level of each object is characterized by semantic description specifications of different levels of paradigms; Among them, the semantic paradigm model includes the text content semantic paradigm model CSNF, the query instruction semantic paradigm model ISNF, the query method semantic paradigm model MSNF and the query workflow semantic paradigm model FSNF; Different levels of paradigm transformation criteria are designed for different query objects to obtain a more accurate semantic description and determine the semantics of the text big data to be queried.
3. A text big data query system according to claim 2, characterized in that: The statistical analysis module includes: A retrieval statistics unit is used to perform retrieval statistics on the semantics of the text big data to be queried; Based on the query requirements of text big data, the semantics of the text big data to be queried is retrieved and counted; In the process of retrieving the semantics of the text big data to be queried, the number of occurrences of each text word in the semantics of the text big data to be queried is counted to determine the number of text words based on the text words; A keyword analysis unit for analyzing the number of times a text word is used based on the text word; Get the number of text words based on the text words; Analyze the number of times text words are used based on the text words; Find the text word with the most occurrences; Based on the text words that appear the most times, the text big data keywords to be queried are determined.
4. A text big data query system according to claim 3, characterized in that: The comparison query module includes: A type comparison unit is used to compare the type of text big data to be queried and determine the text big data query type; Get the text big data type to be queried; Based on the text big data query requirements and the stored text big data types, the text big data types to be queried are compared to determine the text big data query type; A data query unit, used to query text big data query types; Obtaining a text big data query type and querying the text big data query type; Determine text big data query data.
5. A text big data query system according to claim 4, characterized in that: The feedback control module comprises: A data feedback unit, used for providing user feedback on text big data query data; Get text big data query data; Based on the text big data query data, obtain the user's feedback on the text big data query data; Determine user feedback based on text big data query data; A query control unit, used to query and control text big data query data; Obtain user feedback based on text big data query data; Based on user feedback, query control is performed on text big data query data, and text big data query data is intelligently controlled.
6. A text big data query system according to claim 5, characterized in that: Also includes: Data preprocessing module, used to: Before the semantic acquisition module performs semantic acquisition on the text big data collected in real time, the local preservation projection LPP algorithm is used to perform dimensionality reduction processing on the collected text big data to obtain dimensionality reduction data; Regularize the dimension-reduced data based on the adapted regular expression to obtain regularized data; Obtain the characteristic string of the regularized data, match it through the matching automaton, filter out the unmatched characteristic strings, and remove them to obtain the corrected regularized data; The corrected regularized data is clustered using the k-means clustering method to obtain several cluster sets; Calculate the standard deviation of the corresponding cluster set according to the data in each cluster set, compare it with the preset standard deviation, and select the cluster set with a standard deviation greater than the preset standard deviation as the cluster set to be processed; Obtain type information of the cluster set to be processed, query the preset data cleaning rule database according to the type information, obtain the corresponding data cleaning rule, and perform data cleaning according to the corresponding data cleaning rule; In the data cleaning process, after the first data in the cluster set to be processed is cleaned based on the data cleaning rule, first difference information of the first data before and after the data cleaning is determined; Determine a first cleaning indicator to be adjusted for the data cleaning rule according to the first difference information; Obtain second difference information between a feature vector of second data in the cluster set to be processed and a feature vector of the first data; Determine a second cleaning indicator to be adjusted for the data cleaning rule according to the second difference information; The data cleaning rule is adjusted according to the first cleaning index to be adjusted and the second cleaning index to be adjusted to obtain an adjusted data cleaning rule, and data cleaning is performed on the second data in the to-be-processed cluster set based on the adjusted data cleaning rule; Repeat the data cleaning process of the previous data to adjust the data cleaning rules of the next data to obtain the preprocessed text big data.
7. A text big data query method, implemented based on the text big data query system according to claim 6, characterized in that: The steps include: S1: Based on the text big data query demand, the text big data to be queried is collected in real time, and the semantics of the text big data collected in real time are obtained to determine the semantics of the text big data to be queried; S2: Based on the text big data query demand, in the process of retrieving the text big data semantics to be queried, the number of occurrences of each text word in the text big data semantics to be queried is counted, the number of text words based on the text words is determined, and the text big data keywords to be queried are determined based on the text words with the largest number of text words; S3: Based on the text big data query requirements, perform type analysis on the stored text big data types and the text big data keywords to be queried to determine the text big data type to be queried; S4: comparing the text big data type to be queried to determine the text big data query type, and querying the text big data query type to determine the text big data query data; S5: Based on the text big data query data, obtain user feedback on the text big data query data, and based on the user feedback, perform query control on the text big data query data to intelligently control the text big data query data.
