Data set construction method and device, computer equipment and storage medium
By conducting correctness check, diversity analysis and similarity score on the data set of Text-To-SQL model, the problem of data set quality evaluation and improvement in the existing technology is solved, and the model performance is significantly improved.
Patent Information
- Application Number
- CN202411979880.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-29
AI Technical Summary
The prior art is difficult to accurately evaluate and improve the dataset quality of Text-To-SQL models, especially in the field of Spark SQL, resulting in limited model performance improvement.
By conducting correctness checks, diversity analysis and similarity analysis on the original data set, supplement the data set if there is a lack of diversity, and clustering and similarity scores are performed, and finally high-quality data sets are obtained through filtering conditions.
Accurate evaluation and improvement of data set quality is achieved, and the performance of Text-To-SQL model is significantly improved.
Smart Images
Figure CN119903343A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data set technology, and in particular to a data set construction method, apparatus, computer equipment and storage medium. Background Art
[0002] The Text-To-SQL model is used to check the accuracy of SQL statements output based on text input. The Text-To-SQL model is widely used in the industry, and most of the focus is on standard SQL. Spark SQL is different from standard SQL, and Spark SQL is more commonly used in the big data field, but the industry lacks data set training for Spark SQL. In the Text-To-SQL model, fine-tuning training with high-quality SQL data sets can improve the performance of the Text-To-SQL model. Model fine-tuning requires a large number of training data sets, and the data is too complex to accurately evaluate the quality of the data set, and it is impossible to filter out data that meets the standards. Various studies have shown that high-diversity data sets can significantly improve the quality of the data set after a certain degree of deduplication, further improving the capabilities of the model. However, when the deduplication of high-diversity data sets is inaccurate, it is impossible to accurately evaluate the quality of the data set, and it is difficult to effectively improve the performance of the model. Summary of the invention
[0003] Based on this, it is necessary to provide a data set construction method, apparatus, computer equipment and storage medium that can accurately evaluate and judge the quality of a data set in response to the above technical problems.
[0004] In one aspect, a method for constructing a data set is provided, the method comprising:
[0005] Get the current original data set;
[0006] Performing a correctness check on the original data set;
[0007] Performing diversity analysis on the original data set to determine whether there is diversity loss;
[0008] If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency;
[0009] If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0010] According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
[0011] In one embodiment, the step of checking the correctness of the original data set includes:
[0012] Performing a grammatical correctness check on the original data set;
[0013] A field correctness check is performed on the original data set.
[0014] In one embodiment, the step of checking the grammatical correctness of the original data set includes:
[0015] Determine whether the Spark SQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if conforming, perform syntax correctness check;
[0016] Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
[0017] In one embodiment, the step of performing diversity analysis on the original data set includes:
[0018] Analyze the overall distribution and diversity of the original data set;
[0019] Obtain diversity analysis results and analyze diversity loss;
[0020] The original dataset is supplemented according to the diversity gaps.
[0021] In one embodiment, the step of analyzing the overall distribution and diversity of the original data set includes:
[0022] Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query;
[0023] Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query, insert query, update query, delete query, join query, and aggregate query;
[0024] Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used;
[0025] Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query;
[0026] Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
[0027] In one embodiment, the step of supplementing the original data set according to the diversity deficiency includes:
[0028] Obtain diversity analysis results;
[0029] Compare online data and analyze the lack of diversity;
[0030] Construct suggestive language based on said lack of diversity;
[0031] The deep learning model is called to supplement the original data set according to the suggestive language.
[0032] In one embodiment, the step of performing similarity analysis on each data set in the original data set after clustering to obtain a final similarity comprises:
[0033] Get multiple indicators corresponding to multiple dimensions;
[0034] Determine the weight of each dimension;
[0035] Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index;
[0036] Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function;
[0037] Output the comprehensive similarity score.
[0038] In another aspect, a data set construction device is provided, the device comprising:
[0039] The data set acquisition module is used to acquire the current original data set;
[0040] A correctness checking module, used for checking the correctness of the original data set;
[0041] A diversity analysis module, used to perform diversity analysis on the original data set to determine whether there is a lack of diversity;
[0042] A supplementary dataset module, used for supplementing the original dataset according to the diversity deficiency if there is diversity deficiency;
[0043] A similarity analysis module, for performing similarity analysis on each data set in the original data set after clustering if there is no diversity loss, to obtain a final similarity;
[0044] The data set management module is used to filter and remove duplicates from the original data set according to the final similarity and filtering conditions to obtain a high-quality data set.
[0045] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:
[0046] Get the current original data set;
[0047] Performing a correctness check on the original data set;
[0048] Performing diversity analysis on the original data set to determine whether there is diversity loss;
[0049] If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency;
[0050] If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0051] According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
[0052] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0053] Get the current original data set;
[0054] Performing a correctness check on the original data set;
[0055] Performing diversity analysis on the original data set to determine whether there is diversity loss;
[0056] If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency;
[0057] If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0058] According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
[0059] The above-mentioned dataset construction method, device, computer equipment and storage medium can accurately evaluate and judge the quality of the dataset by performing correctness check and diversity analysis on the current original dataset, supplementing the dataset according to diversity missing, and filtering the dataset to obtain a high-quality dataset. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0061] Figure 1 This is an application environment diagram of a data set construction method in one embodiment of the present application;
[0062] Figure 2 This is a flowchart of a method for constructing a data set in one embodiment of the present application;
[0063] Figure 3 This is a flow chart of a step of checking the correctness of the original data set in one embodiment of the present application;
[0064] Figure 4 This is a flow chart of the step of checking the grammatical correctness of the original data set in one embodiment of the present application;
[0065] Figure 5 This is a flow chart of the steps of performing diversity analysis on the original data set in one embodiment of the present application;
[0066] Figure 6 This is a flow chart of the steps of supplementing the original data set according to diversity deficiencies in one embodiment of the present application;
[0067] Figure 7 This is a structural block diagram of a data set construction device in one embodiment of the present application;
[0068] Figure 8 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0070] The dataset construction method provided in this application can be applied to Figure 1 In the application environment shown.
[0071] In one embodiment, Figure 2 As shown, a method for constructing a dataset is provided, which can be applied to Figure 1 Take this as an example to illustrate, including the following steps:
[0072] Step S1, obtaining the current original data set;
[0073] Step S2, checking the correctness of the original data set;
[0074] Step S3, performing diversity analysis on the original data set to determine whether there is diversity loss;
[0075] Step S4, if there is a diversity deficiency, supplementing the original data set with a data set according to the diversity deficiency;
[0076] Step S5, if there is no diversity loss, clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0077] Step S6: filtering the original data set according to the final similarity and filtering conditions to obtain a high-quality data set.
[0078] like Figure 3 As shown, in this embodiment, the step of checking the correctness of the original data set includes:
[0079] Step S21, checking the syntax correctness of the original data set;
[0080] Step S22: Check the correctness of the fields in the original data set.
[0081] like Figure 4 As shown, in this embodiment, the step of checking the grammatical correctness of the original data set includes:
[0082] Determine whether the SparkSQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if compliant, perform syntax correctness check;
[0083] Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
[0084] SparkSQL is a module of Apache Spark that is specifically designed to process structured data. Through SparkSQL, users can use SQL (Structured Query Language) or Apache Spark's own DataFrame API to perform data query and data operation tasks. Spark provides a wealth of operators and functions so that users can easily process and analyze data.
[0085] This application focuses on whether SQL can pass the predefined rule check. The field correctness check is to observe that there are some fabricated table field information in the data set produced by GPT. Because the predefined rule check does not involve the table structure, it can be further judged by the execution of STS. You can also use STS to execute directly to perform a complete syntax check.
[0086] like Figure 5 As shown, in this embodiment, the step of performing diversity analysis on the original data set includes:
[0087] Step S31, analyzing the overall distribution and diversity of the original data set;
[0088] Step S32, obtaining diversity analysis results, and analyzing diversity loss;
[0089] Step S33, supplementing the original data set according to the diversity gaps.
[0090] In this embodiment, the step of analyzing the overall distribution and diversity of the original data set includes:
[0091] Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query;
[0092] Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query (SELECT), insert query (INSERT), update query (UPDATE), delete query (DELETE), join query (JOIN), and aggregate query (AGGREGATE);
[0093] Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used;
[0094] Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query;
[0095] Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
[0096] This embodiment focuses on the diverse distribution of various grammatical keywords and various grammatical structures in the entire data set, such as:
[0097] Lexical Diversity: Use lexical diversity metrics to measure the diversity of different words or keywords in the generated SQL queries. A common metric is the number of unique words, which is the number of different words used in the query. A higher number of unique words indicates more diversity.
[0098] Query type diversity: Query type diversity focuses on the types or categories of queries. It evaluates how many different types of queries there are in the generated query set, such as select queries (SELECT), insert queries (INSERT), update queries (UPDATE), delete queries (DELETE), join queries (JOIN), aggregate queries (AGGREGATE), etc. Query type diversity tells you how many different types of query operations there are in the query set, reflecting the diversity of the purpose and objectives of the queries.
[0099] Query structure diversity: Query structure diversity focuses on the organization and structure of query statements. It evaluates how diverse the query structures are in the generated query set, including the number of tables, the number of conditions, the number of joins, the use of subqueries, the diversity of operators, the number and type of logical operators used, etc. Query structure diversity tells you how many queries with different structures there are in the query set, reflecting the complexity of the queries and the diversity of their organization.
[0100] Condition Diversity: Condition diversity focuses on the diversity of conditions used in query statements. Conditions are an important part of filtering data, so condition diversity evaluates the type, location, and complexity of different conditions. It can include the following aspects:
[0101] Conditions on different columns: Which conditions on different columns are used in the query.
[0102] Operator diversity: What different types of conditional operators are used in the query (e.g., equal to, greater than, less than, etc.).
[0103] Logical operator diversity: Which different logical operators (e.g., AND, OR, NOT) are used in the query.
[0104] Condition complexity: The complexity of the condition, such as nested conditions or queries containing multiple condition clauses.
[0105] Column Selection Diversity: Column selection diversity focuses on the diversity of columns selected in query statements. It evaluates the types and number of columns selected in different queries. Column selection diversity can include the following aspects:
[0106] Columns from different tables: Which columns from different tables are selected in the query.
[0107] Column data type diversity: The data type diversity of the selected column, such as number, text, date, etc.
[0108] Number of columns: How many distinct columns are selected in the query.
[0109] In summary, we analyze the overall distribution and diversity of the dataset from the following multiple dimensions:
[0110] a. Query type dimension, such as the frequency of query types (query, insert, update, delete);
[0111] b. Table name + field dimension, such as querying the proportion of table name and field in the entire database table;
[0112] c. Keyword dimension, such as the frequency of various grammatical keywords used in the data set;
[0113] d. Frequency distribution of the number of nesting levels.
[0114] This part of data can provide guidance on the angles from which the data set can be supplemented.
[0115] like Figure 6 As shown, in this embodiment, the step of supplementing the original data set according to the diversity deficiency includes:
[0116] Obtain diversity analysis results;
[0117] Compare online data and analyze the lack of diversity;
[0118] Construct suggestive language based on said lack of diversity;
[0119] The deep learning model is called to supplement the original data set according to the suggestive language.
[0120] In this embodiment, the step of performing similarity analysis on each data set in the original data set after clustering to obtain a final similarity includes:
[0121] Get multiple indicators corresponding to multiple dimensions;
[0122] Determine the weight of each dimension;
[0123] Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index;
[0124] Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function;
[0125] Output the comprehensive similarity score.
[0126] This embodiment focuses on the similarity between each SQL in the data set: the similarity of the statements after removing the table and field names; the similarity of the table + field names used by the statements; and the similarity of various operators in the syntax tree of the statements.
[0127] This embodiment first calculates the similarity of each dimension, and then uses a multi-dimensional target decision algorithm to comprehensively obtain the final similarity. The similarity can assist in filtering and deduplicating data sets according to certain rules.
[0128] For example:
[0129]
[0130]
[0131] 1. After removing the table name:
[0132]
[0133]
[0134] 2. Table name + fields used:
[0135] The table and field used in the statement can be obtained through the field lineage code.
[0136] datahub_hive.ods_cart_operation_data_notify_5min_ds.message.
[0137] 3. Key Operators
[0138]
[0139]
[0140] The code of the reusable field lineage is used to analyze the syntax tree to obtain the operator:
[0141]
[0142]
[0143] An example of calculating similarity:
[0144] import pandas as pd
[0145] import nltk
[0146] from sklearn.feature_extraction.text import TfidfVectorizer
[0147] from sklearn.metrics.pairwise import cosine_similarity
[0148] #data is the information corresponding to each extracted keyword
[0149] data = KeyWorldInfo
[0150] #Use TF-IDF to extract keyword features
[0151] nltk.download('stopwords')
[0152] stop_words=set(stopwords.words('english'))
[0153] tfidf_vectorizer=TfidfVectorizer(stop_words=list(stop_words))
[0154] tfidf_matrix=tfidf_vectorizer.fit_transform(data)
[0155] #diversity metrics
[0156] #Use cosine similarity to measure the similarity between queries
[0157] cosine_sim=cosine_similarity(tfidf_matrix,tfidf_matrix)
[0158] similarity_matrix=pd.DataFrame(cosine_sim)
[0159] print(similarity_matrix).
[0160] like Figure 6 As shown, it is understandable that before the similarity analysis is performed, semantic proofreading is also included.
[0161] Semantic proofreading:
[0162] Method 1: Manual inspection, suitable for small data volumes;
[0163] Method 2: Use GPT for verification, such as passing SQL and fields to GPT through prompts and letting GPT generate questions.
[0164] The effect verification method of this embodiment is:
[0165] 1. Based on the differences in keyword distribution of ETL / Ad-Hoc training sets, the training sets are supplemented with relevant keywords;
[0166] 2. Classify the test set according to different keyword categories to analyze the model's ability in this category;
[0167] 3. Filter out the data with grammatical errors in the training set.
[0168] Selecting an appropriate vectorization method can fully compare the similarity of SQL statements and comprehensively consider features such as word frequency, keyword order, and UDF.
[0169]
[0170] Multi-dimensional indicator fusion scoring: Combining multiple indicators into one algorithm to uniformly evaluate the degree of diversity is a multi-criteria decision-making problem. Multi-criteria decision-making methods such as weighted average, multi-objective optimization or aggregation functions can be used to combine different indicators to obtain a comprehensive diversity score.
[0171] In the above dataset construction method, by checking the correctness of the current original dataset, performing diversity analysis, supplementing the dataset according to diversity deficiencies, and filtering the dataset, the quality of the dataset can be accurately evaluated and judged to obtain a high-quality dataset.
[0172] It should be understood that although Figure 1-Figure 6 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 6 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0173] In one embodiment, Figure 7As shown, a data set construction device 10 is provided, comprising: a data set acquisition module 1, a correctness check module 2, a diversity analysis module 3, a data set supplement module 4, a similarity analysis module 5 and a data set management module 6.
[0174] The data set acquisition module 1 is used to acquire the current original data set.
[0175] The correctness checking module 2 is used to perform a correctness check on the original data set.
[0176] The diversity analysis module 3 is used to perform diversity analysis on the original data set to determine whether there is diversity loss.
[0177] The data set supplementing module 4 is used for supplementing the original data set according to the diversity deficiency if there is diversity deficiency.
[0178] The similarity analysis module 5 is used to perform similarity analysis on each data set in the original data set after clustering if there is no diversity loss, so as to obtain a final similarity.
[0179] The data set management module 6 is used to filter and remove duplicates from the original data set according to the final similarity and filtering conditions to obtain a high-quality data set.
[0180] In this embodiment, the step of checking the correctness of the original data set includes:
[0181] Performing a grammatical correctness check on the original data set;
[0182] A field correctness check is performed on the original data set.
[0183] In this embodiment, the step of checking the grammatical correctness of the original data set includes:
[0184] Determine whether the Spark SQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if conforming, perform syntax correctness check;
[0185] Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
[0186] In this embodiment, the step of performing diversity analysis on the original data set includes:
[0187] Analyze the overall distribution and diversity of the original data set;
[0188] Obtain diversity analysis results and analyze diversity loss;
[0189] The original dataset is supplemented according to the diversity gaps.
[0190] In this embodiment, the step of analyzing the overall distribution and diversity of the original data set includes:
[0191] Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query;
[0192] Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query (SELECT), insert query (INSERT), update query (UPDATE), delete query (DELETE), join query (JOIN), and aggregate query (AGGREGATE);
[0193] Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used;
[0194] Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query;
[0195] Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
[0196] In this embodiment, the step of supplementing the original data set according to the diversity deficiency includes:
[0197] Obtain diversity analysis results;
[0198] Compare online data and analyze the lack of diversity;
[0199] Construct suggestive language based on said lack of diversity;
[0200] The deep learning model is called to supplement the original data set according to the suggestive language.
[0201] In this embodiment, the step of performing similarity analysis on each data set in the original data set after clustering to obtain a final similarity includes:
[0202] Get multiple indicators corresponding to multiple dimensions;
[0203] Determine the weight of each dimension;
[0204] Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index;
[0205] Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function;
[0206] Output the comprehensive similarity score.
[0207] In the above-mentioned data set construction device, by checking the correctness of the current original data set, performing diversity analysis, supplementing the data set according to diversity deficiencies, and filtering the data set, the quality of the data set can be accurately evaluated and judged to obtain a high-quality data set.
[0208] For the specific definition of the data set construction device, please refer to the definition of the data set construction method above, which will not be repeated here. Each module in the above-mentioned data set construction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0209] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data set construction data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data set construction method is implemented.
[0210] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0211] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0212] Get the current original data set;
[0213] Performing a correctness check on the original data set;
[0214] Performing diversity analysis on the original data set to determine whether there is diversity loss;
[0215] If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency;
[0216] If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0217] According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
[0218] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0219] The step of checking the correctness of the original data set comprises:
[0220] Performing a grammatical correctness check on the original data set;
[0221] A field correctness check is performed on the original data set.
[0222] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0223] The step of checking the grammatical correctness of the original data set comprises:
[0224] Determine whether the Spark SQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if conforming, perform syntax correctness check;
[0225] Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
[0226] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0227] The step of performing diversity analysis on the original data set comprises:
[0228] Analyze the overall distribution and diversity of the original data set;
[0229] Obtain diversity analysis results and analyze diversity loss;
[0230] The original dataset is supplemented according to the diversity gaps.
[0231] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0232] The step of analyzing the overall distribution and diversity of the original data set includes:
[0233] Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query;
[0234] Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query (SELECT), insert query (INSERT), update query (UPDATE), delete query (DELETE), join query (JOIN), and aggregate query (AGGREGATE);
[0235] Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used;
[0236] Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query;
[0237] Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
[0238] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0239] The step of supplementing the original data set according to the diversity deficiency comprises:
[0240] Obtain diversity analysis results;
[0241] Compare online data and analyze the lack of diversity;
[0242] Construct suggestive language based on said lack of diversity;
[0243] The deep learning model is called to supplement the original data set according to the suggestive language.
[0244] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0245] The step of performing similarity analysis after clustering each data set in the original data set to obtain a final similarity comprises:
[0246] Get multiple indicators corresponding to multiple dimensions;
[0247] Determine the weight of each dimension;
[0248] Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index;
[0249] Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function;
[0250] Output the comprehensive similarity score.
[0251] For the specific limitations on the steps implemented when the processor executes the computer program, please refer to the limitations on the method for constructing the data set above, which will not be repeated here.
[0252] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0253] Get the current original data set;
[0254] Performing a correctness check on the original data set;
[0255] Performing diversity analysis on the original data set to determine whether there is diversity loss;
[0256] If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency;
[0257] If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity;
[0258] According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
[0259] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0260] The step of checking the correctness of the original data set comprises:
[0261] Performing a grammatical correctness check on the original data set;
[0262] A field correctness check is performed on the original data set.
[0263] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0264] The step of checking the grammatical correctness of the original data set comprises:
[0265] Determine whether the Spark SQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if conforming, perform syntax correctness check;
[0266] Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
[0267] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0268] The step of performing diversity analysis on the original data set comprises:
[0269] Analyze the overall distribution and diversity of the original data set;
[0270] Obtain diversity analysis results and analyze diversity loss;
[0271] The original dataset is supplemented according to the diversity gaps.
[0272] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0273] The step of analyzing the overall distribution and diversity of the original data set includes:
[0274] Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query;
[0275] Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query (SELECT), insert query (INSERT), update query (UPDATE), delete query (DELETE), join query (JOIN), and aggregate query (AGGREGATE);
[0276] Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used;
[0277] Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query;
[0278] Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
[0279] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0280] The step of supplementing the original data set according to the diversity deficiency comprises:
[0281] Obtain diversity analysis results;
[0282] Compare online data and analyze the lack of diversity;
[0283] Construct suggestive language based on said lack of diversity;
[0284] The deep learning model is called to supplement the original data set according to the suggestive language.
[0285] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0286] The step of performing similarity analysis after clustering each data set in the original data set to obtain a final similarity comprises:
[0287] Get multiple indicators corresponding to multiple dimensions;
[0288] Determine the weight of each dimension;
[0289] Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index;
[0290] Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function;
[0291] Output the comprehensive similarity score.
[0292] For specific limitations on the steps implemented when the computer program is executed by the processor, please refer to the limitations on the method for constructing a data set above, which will not be repeated here.
[0293] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0294] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0295] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for constructing a data set, characterized in that: include: Get the current original data set; Performing a correctness check on the original data set; Performing diversity analysis on the original data set to determine whether there is diversity loss; If there is a diversity deficiency, the original data set is supplemented with a data set according to the diversity deficiency; If there is no lack of diversity, then clustering each data set in the original data set and performing similarity analysis to obtain a final similarity; According to the final similarity, the original data set is filtered and deduplicated according to the filtering conditions to obtain a high-quality data set.
2. The method for constructing a data set according to claim 1, characterized in that: The step of checking the correctness of the original data set comprises: Performing a grammatical correctness check on the original data set; A field correctness check is performed on the original data set.
3. The method for constructing a data set according to claim 2, characterized in that: The step of checking the grammatical correctness of the original data set comprises: Determine whether the SparkSQL data in the original data set conforms to the Spark syntax verification rules, if not, filter the original data set, if compliant, perform syntax correctness check; Determine whether the syntax correctness check passes, if not, filter the original data set, if yes, output the original data set.
4. The method for constructing a data set according to claim 1, characterized in that: The step of performing diversity analysis on the original data set comprises: Analyze the overall distribution and diversity of the original data set; Obtain diversity analysis results and analyze diversity loss; The original dataset is supplemented according to the diversity gaps.
5. The method for constructing a data set according to claim 4, characterized in that: The step of analyzing the overall distribution and diversity of the original data set includes: Performing a lexical diversity analysis on the original data set, and using a lexical diversity index to measure the diversity of different words or keywords in the generated data set query; Generate a query set, perform query type diversity analysis, and evaluate the frequency of query types in the generated query set; the query types include select query, insert query, update query, delete query, join query, and aggregate query; Performing query structure diversity analysis on query structures in the generated query set to obtain the number of query structures in the query set, wherein the query structure includes the number of tables, the number of conditions, the number of connections, the use of subqueries, the diversity types and number of operators, and the number and type of logical operators used; Perform condition diversity analysis to obtain the diversity of conditions used in the query statement, and obtain the filter column conditions, condition operators, logical operators, nested conditions or condition clauses used in the query; Perform column selection diversity analysis to obtain the types and number of columns selected in the query.
6. The method for constructing a data set according to claim 4, characterized in that: The step of supplementing the original data set according to the diversity deficiency comprises: Obtain diversity analysis results; Compare online data and analyze the lack of diversity; Construct suggestive language based on said lack of diversity; The deep learning model is called to supplement the original data set according to the suggestive language.
7. The method for constructing a data set according to claim 1, characterized in that: The step of performing similarity analysis after clustering each data set in the original data set to obtain a final similarity comprises: Get multiple indicators corresponding to multiple dimensions; Determine the weight of each dimension; Normalize the diversity index values, assign weights, and calculate the similarity scores of the data sets corresponding to each index; Calculate the similarity comprehensive score by weighted average, multi-objective optimization or aggregation function; Output the comprehensive similarity score.
8. A data set construction device, characterized in that: The device comprises: The data set acquisition module is used to acquire the current original data set; A correctness checking module, used for checking the correctness of the original data set; A diversity analysis module, used to perform diversity analysis on the original data set to determine whether there is a lack of diversity; A supplementary dataset module, used for supplementing the original dataset according to the diversity deficiency if there is diversity deficiency; A similarity analysis module, for performing similarity analysis on each data set in the original data set after clustering if there is no diversity loss, to obtain a final similarity; The data set management module is used to filter and remove duplicates from the original data set according to the final similarity and filtering conditions to obtain a high-quality data set.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.