A method and system for hudi asynchronous data clustering based on hot spot prediction

Abstract

The application relates to a method and system for Hudi asynchronous data clustering based on hotspot prediction; the method comprises the following steps: SQL query statement analysis, including collecting SQL statements, word segmentation processing, establishing a vocabulary table, and establishing an embedding layer; based on the obtained and parsed SQL statements, a trained LSTM model based on an online learning algorithm is used to predict query hotspot fields and query hotspot tables; the trained LSTM model based on the online learning algorithm is used to predict the query hotspot fields and the query hotspot tables, and asynchronous data clustering analysis is automatically performed multiple times. The application can avoid the data inconsistency problem caused by asynchronous clustering to a certain extent, so that the data files in the partition have high freshness; in addition, by optimizing the hotspot data layout, the query efficiency when Hudi is used as a Presto engine data source is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data computing, and particularly relates to a method and system for Hudi asynchronous data clustering based on hot spot prediction. BACKGROUND

[0002] Hudi (Hadoop Upsert and Incremental) is an open source data management framework for managing large-scale data lakes. It provides functions such as writing, updating, deleting, merging, etc., while supporting incremental data synchronization and time travel queries. In Hudi, there are two storage methods for data tables. The first is the COW (Copy On Write) table, which supports efficient data updating and changing operations while maintaining data consistency and queryability. The other is the MOR (Merge On Read) table, which has the ability to store column-based data in cloud storage or distributed file systems and use Hudi for data merging and incremental updating.

[0003] However, both COW tables and MOR tables have their own shortcomings in terms of query performance. COW tables use an append-write approach, where each write operation creates a new data file, which leads to continuous growth of data files, affecting write performance, and increasing the management, storage and maintenance costs of the file system, and reducing query performance. In the MOR mode, data merging is based on the "read after merging" mode, and newly written data may need to wait for the completion of the merging operation before it is visible to external queries, which will undoubtedly cause some queries to be delayed in the case of frequent data writing.

[0004] In addition, most query operations are performed by a distributed SQL query engine to query data in the data lake. When data is imported into the lake, it is stored sequentially in Hudi, but when queried, it is random, which affects the query efficiency of both COW tables and MOR tables to varying degrees.

[0005] To solve the above problems, the Hudi native community proposes a solution - synchronous / asynchronous clustering. Hudi Clustering allows data to be divided into different partitions based on certain column values of the data, which helps to divide the data into smaller data blocks; secondly, Clustering clusters data by column value, so that similar data values are usually clustered together, and data compression is more effective; finally, because Clustering only loads the data partitions of interest, it can query and analyze data more quickly, reducing the amount of data that needs to be processed when searching and accessing data in the data lake. Therefore, the clustering mechanism can optimize the data layout in the data lake, thereby reducing storage costs, making data access more efficient, and greatly improving query performance. Although Clustering facilitates the execution of data operations and queries, synchronous clustering requires immediate data rearrangement when data is written, which can result in significant performance overhead, and because synchronous clustering requires immediate data manipulation, it can increase the latency of write operations, so it is not suitable for applications with high requirements for low latency; and asynchronous clustering may have data inconsistency risks because it does not rearrange data immediately after writing, which means that queries may not immediately reflect the latest data, and asynchronous clustering is difficult to manually set without a reasonable strategy.

[0006] Therefore, how to achieve reasonable data layout optimization through clustering mechanism to a certain extent to improve query efficiency is a challenge faced by Hudi data lake scenario. SUMMARY

[0007] To solve the above problems, the Hudi native community proposes a solution - synchronous / asynchronous clustering. Hudi Clustering allows data to be divided into different partitions based on certain column values of the data, which helps to divide the data into smaller data blocks; secondly, Clustering clusters data by column value, so that similar data values are usually clustered together, and data compression is more effective; finally, because Clustering only loads the data partitions of interest, it can query and analyze data more quickly, reducing the amount of data that needs to be processed when searching and accessing data in the data lake. Therefore, the clustering mechanism can optimize the data layout in the data lake, thereby reducing storage costs, making data access more efficient, and greatly improving query performance. Although Clustering facilitates the execution of data operations and queries, synchronous clustering requires immediate data rearrangement when data is written, which can result in significant performance overhead, and because synchronous clustering requires immediate data manipulation, it can increase the latency of write operations, so it is not suitable for applications with high requirements for low latency; and asynchronous clustering may have data inconsistency risks because it does not rearrange data immediately after writing, which means that queries may not immediately reflect the latest data, and asynchronous clustering is difficult to manually set without a reasonable strategy.

[0008] By using the present application, the data inconsistency problem caused by asynchronous clustering can be avoided to a certain extent, so that the data files in the partition have higher freshness; in addition, by optimizing the hot data layout, the query efficiency when using Hudi as the data source of the Presto engine is greatly improved.

[0009] The application also provides a system for Hudi asynchronous data clustering based on hotspot prediction.

[0010] Terminology explanation:

[0011] Hudi: Hadoop Upserts Deletes and Incrementals, an open-source data lake management framework for managing and processing batch and streaming data in a big data environment;

[0012] COW table: "Copy-On-Write" table, a storage mode of Hudi framework, which does not modify existing data, but creates new data files to represent each update or delete operation;

[0013] MOR table: "Merge On Read" table, a table type of Hudi, used to support data update, insertion and deletion operations in large-scale data lake.

[0014] Hudi Clustering: a core function of Hudi framework, used for data management and query optimization in data lake;

[0015] Presto: a distributed SQL query engine for querying large-scale data, aiming to perform fast and interactive query analysis in large-scale data warehouse, supporting multiple data sources and formats;

[0016] Online learning algorithm: "Online Learning Algorithm", a class of machine learning algorithms for processing sequential data. Unlike traditional batch learning, online learning algorithm learns step by step and continuously updates the model to adapt to the continuous data stream;

[0017] LSTM model: Long Short-term memory (LSTM) is a deep learning model mainly used for processing sequential data, especially sequential data with time dependence, such as natural language text and speech.

[0018] The technical solution of the application is:

[0019] A method for Hudi asynchronous data clustering based on hotspot prediction, comprising:

[0020] SQL query statement analysis, including collecting SQL statements, word segmentation processing, establishing a vocabulary table, and establishing an embedding layer;

[0021] Based on the obtained and parsed SQL statement, a LSTM model based on an online learning algorithm is used to predict the query hot field and the query hot table;

[0022] The query hot field and the query hot table predicted by the LSTM model based on the online learning algorithm are obtained, and the asynchronous data clustering analysis is automatically performed multiple times.

[0023] According to the application, the SQL query statement analysis preferably comprises:

[0024] Collecting SQL statements: real-time acquisition of historical query records from Presto log files, including query statements and query times;

[0025] Tokenization processing: pre-processing the SQL query statement to remove unnecessary characters, punctuation marks or white spaces; decomposing the SQL query statement into words or terms, and the decomposition result is a word list;

[0026] Building a vocabulary table: classifying the decomposed words; extracting the field and the table name according to the classification result, and building a vocabulary table.

[0027] Further preferably, the embedding layer is established, which refers to the construction, training and application of the Word2Vec model, including:

[0028] Creating context-target pairs: the context is the surrounding words of the target word, and the target is the target word itself;

[0029] Training the Word2Vec model: using the context-target pair data to train the Word2Vec model;

[0030] Learning word vectors: the Word2Vec model itself learns word vectors through an iterative training process, updates the word vectors according to the loss gradient of the current context-target pair, and minimizes the loss function;

[0031] Obtaining word vectors: after training, the word vector representation of each word is extracted from the Word2Vec model;

[0032] Applying word vectors: the trained word vectors are used as the input of the LSTM model based on the online learning algorithm.

[0033] According to the application, based on the obtained and parsed SQL statement, a LSTM model based on an online learning algorithm is used to predict the query hot field and the query hot table, including:

[0034] The LSTM model based on the online learning algorithm comprises a first layer LSTM network, a second layer LSTM network, a third layer LSTM network, a full connection layer, and different numbers of neurons are set in the three-layer LSTM network.

[0035] The first layer LSTM network: the input data is first passed to the first layer LSTM network, which is used for input sequence processing, information extraction, and long-term dependency modeling;

[0036] The second layer LSTM network: the output of the first layer LSTM network is passed to the second layer LSTM network, which is used for feature extraction and abstraction, and representation learning;

[0037] The third layer LSTM network: the output of the second layer LSTM network is passed to the third layer LSTM network, which is used for further feature extraction and enhances the model's ability;

[0038] Further feature extraction: the third layer can further extract features of the input sequence, making the model better understand the complexity of the input data. Enhance the model's ability: by increasing the depth, the model can handle more complex sequences.

[0039] Fully connected layer: the fully connected layer is used for output prediction, decision-making, and analysis, and also includes a Relu activation function;

[0040] In summary, the three-layer LSTM neural network and one-layer fully connected layer are combined together, allowing the model to effectively process sequence data, from extracting low-level features to capturing high-level abstractions, and finally outputting prediction results.

[0041] Further preferably, the trained LSTM model based on the online learning algorithm is used to predict the query hotspot field and the query hotspot table, including:

[0042] First, according to the query time in the historical query record, divide it by the same time interval, and summarize the number of word vectors obtained by the Word2Vec model appearing in each time interval. Save the time interval and its corresponding word vector and the number of word vector appearances as CSV format as the input data set of the LSTM model based on the online learning algorithm; and preprocess the input data set, that is, normalize the input data set, and perform a sliding window operation on the normalized data;

[0043] Second, use the LSTM model as a prediction model, input the word vector and its number of appearances in the corresponding time interval into the trained LSTM model, and output the word vector and its number of appearances that will appear in the future period of time;

[0044] Sum the output results of the prediction model, summarize all the word vectors and their number of appearances that will appear in the future period of time, and sort them in descending order according to the number of appearances, and define the corresponding field or table with query times greater than the threshold as query hotspot data;

[0045] Finally, the output query hot spot data of the LSTM model is used to find the corresponding word text in the vocabulary by using word embedding inverse solution, and the word vector in the output is finally saved as word text.

[0046] In order to provide real-time feedback query hot spot for Clustering to rewrite data partition after continuously receiving new data, an online learning cycle is added to the LSTM model, which gradually updates the model to adapt to new data, in other words, the weight of the LSTM model needs to be updated using new data.

[0047] Further preferably, the online learning cycle is added to the LSTM model, including:

[0048] Firstly, new data points are obtained from the real-time obtained historical query records, and the new data points are transmitted to the LSTM model for forward propagation, and the loss of forward propagation is calculated, which is usually measured by a loss function.

[0049] Secondly, the gradient of the loss function is propagated to each weight parameter using the back propagation algorithm, and the gradient is the partial derivative of the loss function with respect to the weight; and the weight of the LSTM model is updated using stochastic gradient descent (SGD);

[0050] Finally, the learning rate scheduling method is used to dynamically adjust the learning speed.

[0051] According to the application, the query hot spot field and the query hot spot table predicted by the LSTM model trained based on the online learning algorithm are obtained, and the asynchronous data clustering analysis is automatically performed multiple times, including:

[0052] The related configuration is modified to meet the related requirements of asynchronous clustering analysis;

[0053] Further preferably, the query hot spot data output by the LSTM model based on the online learning algorithm is obtained, and the query hot spot data is used as the basis for data writing, and the clustering analysis is performed multiple times, including:

[0054] The local python connects the Linux virtual machine;

[0055] The local python connects the MySQL library in the Linux virtual machine;

[0056] The query hot spot field and the table involved in the final output of the LSTM model based on the online learning algorithm are respectively summarized into a new file, and named as hot spot field data table and hot spot table data table;

[0057] The hot spot field data table and the hot spot table data table are saved to the MySQL database, and are used as new data files through Flink into the lake to generate a plurality of small data files.

[0058] And write them in the new partition file group, when the commit number of data writing reaches several times, automatically trigger python to send Linux command to Hudi, generate clustering analysis execution plan.

[0059] A computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method for Hudi asynchronous data clustering based on hot spot prediction when executing the computer program.

[0060] A computer readable storage medium having a computer program stored thereon, the computer program implements the steps of the method for Hudi asynchronous data clustering based on hot spot prediction when executed by a processor.

[0061] Generate a clustering analysis execution plan, read the plan, and execute asynchronous data clustering.

[0062] A system for Hudi asynchronous data clustering based on hot spot prediction, comprising:

[0063] The SQL query statement parsing module is configured to obtain and parse the SQL query statement;

[0064] The model construction module is configured to construct and train the LSTM model, and add an online learning loop in the LSTM model;

[0065] The prediction module is configured to predict the hot spot query field and the hot spot query table by using the trained LSTM model based on the online learning algorithm;

[0066] The data clustering module is configured to predict the hot spot query field and the hot spot query table by using the trained LSTM model based on the online learning algorithm, and automatically execute asynchronous data clustering analysis multiple times.

[0067] The beneficial effects of the present application are:

[0068] The present application proposes a prediction method for hot spot query fields and hot spot query tables, and writes the hot spot query fields and the hot spot query tables into new small data files in the Hudi data lake, triggers automatic asynchronous data clustering through the commit number of writing, combines the small files into large files with the same or similar size, reduces the cost of file management; when the Presto cluster requests data shards, first search the data in the large file, if there are query related fields, directly send them to the Presto cluster, if not, search globally, to a certain extent, avoid massive data retrieval, improve data query efficiency; in addition, one step completes asynchronous clustering scheduling and execution, avoids inconsistency problems during data writing. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A flowchart of a method for Hudi asynchronous data clustering based on hotspot prediction;

[0070] Figure 2 A flowchart of SQL query statement parsing;

[0071] Figure 3 A network architecture diagram of an LSTM model based on an online learning algorithm;

[0072] Figure 4 An architecture diagram of a system for Hudi asynchronous data clustering based on hotspot prediction. DETAILED DESCRIPTION

[0073] The present application will be further limited by the specification, drawings and examples below, but not limited thereto.

[0074] Example 1

[0075] A method for Hudi asynchronous data clustering based on hotspot prediction, as shown in Figure 1 , comprising:

[0076] SQL query statement parsing, including collecting SQL statements, word segmentation processing, establishing a vocabulary table, and establishing an embedding layer;

[0077] Based on the obtained and parsed SQL statement, a trained LSTM model based on an online learning algorithm is used to predict the query hotspot field and the query hotspot table;

[0078] Obtain the query hotspot field and the query hotspot table predicted by the trained LSTM model based on the online learning algorithm, and automatically perform asynchronous data clustering analysis multiple times.

[0079] Example 2

[0080] A method for Hudi asynchronous data clustering based on hotspot prediction according to example 1, as shown in Figure 2 , in the method, the SQL query statement parsing comprises:

[0081] Collecting SQL statements: real-time acquisition of historical query records from Presto log files, including query statements and query times;

[0082] Word segmentation processing: pre-processing the SQL query statement to remove unnecessary characters, punctuation marks or white spaces; using functions or methods in the word segmentation library to decompose the SQL query statement into words or terms, and the result of the decomposition is a word list; for Chinese text, the entire query statement is decomposed into single words, including field names, table names, operators, keywords, etc.

[0083] Vocabulary building: Based on the judgment of SQL syntax rules, the decomposed words are classified; for example, field names usually appear in the SELECT clause, while table names usually appear in the FROM clause. According to the classification results, the field and table names are extracted to build the vocabulary.

[0084] In this method, the establishment of the embedding layer refers to the Word2Vec model construction, training and application, including:

[0085] Since the input of the LSTM model is a vector representation, it is necessary to convert the word tokens of the SQL statement (i.e. the decomposed words) into dense vector representations. Word2Vec (Word to Vector) is a natural language processing (NLP) technique used to convert words in text into vector representations. This technology can capture the semantic relationship between words, making words with similar meanings closer in vector space. Among them, the Word2Vec model has two main architectures, the Skip-gram model tries to predict the context words through the target word. This makes the Skip-gram model usually perform better in large corpora, because it can handle more data and capture more semantic information.

[0086] Creating context-target pairs: Word2Vec model learns word vectors by predicting the context around the words in the vocabulary. Therefore, it is necessary to create context-target pairs first, where the context is the surrounding words of the target word, and the target is the target word itself;

[0087] Training the Word2Vec model: Use the context-target pair data to train the Word2Vec model; in the Skip-gram model, the model will try to predict the target word given its context;

[0088] Learning word vectors: Word2Vec model itself learns word vectors through an iterative training process, updating word vectors according to the loss gradient of the current context-target pair, and minimizing the loss function; in order to better capture the distributed representation of language;

[0089] Get word vectors: After training, extract the word vector representation of each word from the Word2Vec model;

[0090] Apply word vectors: Use the trained word vectors as input to the LSTM model based on online learning algorithm.

[0091] Based on the obtained and parsed SQL statement, the trained LSTM model based on online learning algorithm is used to predict the query hot field and query hot table, including:

[0092] For example, Figure 3As shown, the LSTM model based on online learning algorithm includes a first layer LSTM network, a second layer LSTM network, a third layer LSTM network, a fully connected layer, and different numbers of neurons are set in the three layers of LSTM network;

[0093] The first layer LSTM network: the input data is first passed to the first layer LSTM network, so this layer is also the "input layer" of the model. The first layer LSTM network is used for processing of input sequences, information extraction, and long-term dependency modeling;

[0094] Processing of input sequences: it receives input sequences and learns patterns and relationships in the sequences. Each time step has an LSTM unit that processes the current element of the input sequence. Information extraction and long-term dependency modeling: the LSTM unit contains a memory cell that can capture long-term dependencies, which helps the model understand the context information in the sequence.

[0095] The second layer LSTM network: the output of the first layer LSTM network is usually passed to the second layer LSTM network, so the second layer LSTM network is the "middle layer" of the model. The second layer LSTM network is used for feature extraction and abstraction, and representation learning;

[0096] Feature extraction and abstraction: the second layer LSTM can further extract abstract features in the sequence, which helps the model understand higher-level patterns.

[0097] Representation learning: through the combination of multiple LSTM units, the second layer helps to learn more complex representations that can capture more details of the input sequence.

[0098] The third layer LSTM network: the output of the second layer LSTM network can be passed to the third layer LSTM network for further information processing, so this layer is a "deep LSTM". The third layer LSTM network is used for further feature extraction and enhances the model's ability;

[0099] Further feature extraction: the third layer can further extract features of the input sequence, making the model better understand the complexity of the input data. Enhance the ability of the model: by increasing the depth, the model can handle more complex sequences.

[0100] Fully connected layer: the output layer of the LSTM model. The fully connected layer is used for output prediction, decision-making, and analysis, and the fully connected layer also includes a Relu activation function;

[0101] Output prediction: the fully connected layer maps the output of the final LSTM layer to the desired output space, such as classification labels or regression values.

[0102] Decision and classification: this layer integrates the information from the previous layers into a final output for decision-making or classification.

[0103] In summary, the three-layer LSTM neural network and one fully connected layer are combined together, allowing the model to effectively process sequence data, from extracting low-level features to capturing high-level abstraction, and finally outputting the prediction result.

[0104] In the method, the trained LSTM model based on online learning algorithm is used to predict the query hot field and the query hot table, comprising:

[0105] Firstly, according to the query time in the historical query record, each 24 hours is divided into the same time interval (30 minutes), and the occurrence number of the word vector in each time interval obtained by the Word2Vec model is summarized, that is, the query number of each field and table in 30 minutes, and the time interval, the corresponding word vector (field and table) and the occurrence number (query number) of the word vector are saved as CSV format as the input data set of the LSTM model based on online learning algorithm; and the input data set is preprocessed, that is, the input data set is normalized, and the data after normalization is subjected to sliding window operation;

[0106] Secondly, the LSTM model is used as a prediction model, and the word vector and its occurrence number in the corresponding time interval are input into the trained LSTM model, and the word vector and its occurrence number that will appear in the future period of 100 time intervals (24 hours) are output;

[0107] The output result of the prediction model is summed, all the word vectors and their occurrence numbers that will appear in the future period (24 hours) are summarized, and are sequentially sorted in descending order of occurrence number, and the corresponding field or table with query number greater than the threshold (H(x)) is defined as query hot data;

[0108] Wherein, the calculation formula of H(x) is shown as formula (I):

[0109]

[0110] In formula (I), X i is the query number corresponding to the i th time point, X min and X max are the maximum query number and the minimum query number respectively;

[0111] Finally, the output query hot data of the LSTM model is subjected to word embedding reverse solution, the word vector corresponding to the word text in the vocabulary table is found, and the word vector in the output is finally saved in the form of word text. For Clustering partition rewriting, the specific steps of word embedding reverse solution are the reverse operation of the word embedding technology, which is an existing mature implementation technology, and the present application will not be repeated.

[0112] In order to provide real-time feedback on query hotspots for data partition rewriting by Clustering, an online learning loop is added to the LSTM model, which is the core of gradually updating the model to adapt to new data, in other words, the weights of the LSTM model need to be updated using new data.

[0113] The online learning loop is added to the LSTM model, including:

[0114] First, new data points are obtained from the real-time historical query records, and the new data points are passed to the LSTM model for forward propagation, and the loss of forward propagation (the difference between the predicted value and the actual value) is calculated, which is usually measured by a loss function.

[0115] Among them, in the LSTM model based on online learning algorithm, MSE is used as the loss function, as shown in formula (II):

[0116]

[0117] In formula (II), y represents the true observed value, represents the predicted value, and m represents the number of samples.

[0118] Secondly, using the backpropagation algorithm, the gradient of the loss function is propagated back to each weight parameter, and the gradient is the partial derivative of the loss function with respect to the weight; and using stochastic gradient descent (SGD) to update the weights of the LSTM model;

[0119] Finally, in order to control the learning rate to ensure that the model adapts to new data while not forgetting old data too quickly, a learning rate scheduling method is used to dynamically adjust the learning speed. When the performance improves slowly, the learning rate is reduced to avoid oscillation or overfitting; when the performance improves quickly, the learning rate is increased to approach the optimal solution faster.

[0120] Obtain the query hotspot field and query hotspot table predicted by the trained LSTM model based on online learning algorithm, and automatically perform asynchronous data clustering analysis multiple times, including:

[0121] Modify the related configuration to meet the related requirements of asynchronous clustering analysis, including:

[0122] Set the hoodie.clustering.async.enabled configuration parameter to true to enable asynchronous clustering analysis;

[0123] Set the hoodie.clustering.async.max.commits configuration parameter to 16 to control the asynchronous clustering analysis frequency to 16 commits;

[0124] Set the hoodie.clustering.plan.strategy.small.file.limit configuration parameter to 209715200 (200MB) to set the small file upper limit to 200MB;

[0125] Set the hoodie.clustering.plan.strategy.target.file.max.bytes configuration parameter to 524,288,000 (500MB) to set the large file memory upper limit to 500MB;

[0126] Set the hoodie.clustering.plan.strategy.sort.columns configuration parameter to review_date to specify the column for sorting data during clustering analysis;

[0127] Set the hoodie.parquet.small.file.limit configuration parameter to 0 to turn off the Parquet small file check, and any data write will be directly written to a new file;

[0128] Set the hoodie.copyonwrite.record.size.estimate configuration parameter to 175 to specify the estimated size value of 175MB when the inserted data size is not explicitly specified;

[0129] Obtain query hotspot data output by the LSTM model based on the online learning algorithm, and use the query hotspot data as the basis for data writing to perform clustering analysis multiple times, including:

[0130] Locally connect the Linux virtual machine through python;

[0131] Locally connect the MySQL library in the Linux virtual machine through python;

[0132] Collect the query hotspot fields and tables involved in the final output of the LSTM model based on the online learning algorithm into a new file respectively, and name them as hotspot field data table and hotspot table data table;

[0133] Save the hotspot field data table and the hotspot table data table to the MySQL database, and use them as new data files to generate a number of small data files with memory less than 200M through Flink lake;

[0134] And write them into a new partition file group, when the number of data writing submission reaches several times, for example, 16 times, that is, the writing of hot field data table and hot table data table is 8 times respectively, automatically trigger python to send Linux command to Hudi to generate clustering analysis execution plan. Explain which tables or fields will be clustered, that is, hot field data table and hot table data table in the present application.

[0135] At the same time of generating clustering analysis execution plan, read plan, execute asynchronous data clustering. Generate large files with the same or similar size but less than 500MB of memory;

[0136] Because the data writing will not be updated during the generation of asynchronous execution plan to the completion of asynchronous clustering analysis, it may cause the problem of data inconsistency. Therefore, asynchronous scheduling and asynchronous execution are adopted to complete in one step through a command, avoiding the consistency problem caused by the delay of data update during clustering analysis.

[0137] When there is a query request, retrieve from the above-mentioned large files, reduce the request waiting time.

[0138] Embodiment 3

[0139] A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method for Hudi asynchronous data clustering based on hot spot prediction in embodiment 1 or 2 when executing the computer program.

[0140] Embodiment 4

[0141] A computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method for Hudi asynchronous data clustering based on hot spot prediction in embodiment 1 or 2 when executed by a processor.

[0142] Embodiment 5

[0143] A system for Hudi asynchronous data clustering based on hot spot prediction, as shown in Figure 4 , comprising:

[0144] The SQL query statement parsing module is configured to obtain and parse the SQL query statement;

[0145] The model construction module is configured to construct and train the LSTM model, and add an online learning loop in the LSTM model;

[0146] The prediction module is configured to predict the hot query field and the hot query table by using the trained LSTM model based on the online learning algorithm;

[0147] The data clustering module is configured to: based on the trained LSTM model based on the online learning algorithm to predict the hot query field and the hot query table, automatically execute the asynchronous data clustering analysis multiple times.

[0148] Obviously, the examples listed in the specific embodiments are only part of the examples of the present application, not all examples. Based on the examples of the present application, all other examples obtained by the skilled person without creative labor shall belong to the protection scope of the present application.

Claims

1. A method for Hudi asynchronous data clustering based on hotspot prediction, characterized in that, Comprise: SQL query statement analysis, including collecting SQL statements, word segmentation processing, building vocabulary, building embedding layer; Based on the SQL statement obtained and parsed, the LSTM model based on online learning algorithm is used to predict the query hot field and query hot table; Get the query hot field and query hot table predicted by the LSTM model based on online learning algorithm, and automatically perform asynchronous data clustering analysis multiple times; including: Modify the relevant configuration to meet the relevant requirements of asynchronous clustering analysis; Get the query hot data output by the LSTM model based on online learning algorithm, and use the query hot data as the basis for data writing to perform clustering analysis multiple times, including: Local python connects Linux virtual machine; Local python connects MySQL library in Linux virtual machine; The query hot field and table involved in the final output of the LSTM model based on online learning algorithm are respectively summarized into a new file, named hot field data table and hot table data table; Save the hot field data table and hot table data table to the MySQL database, and use it as a new data file to import it into Hudi through Flink to generate a number of small data files; And write them into a new partition file group, when the number of data writing submission reaches several times, automatically trigger python to send Linux command to Hudi, and generate clustering analysis execution plan; Generate clustering analysis execution plan at the same time, read the plan, and execute asynchronous data clustering.

2. The method for Hudi asynchronous data clustering based on hot spot prediction according to claim 1, wherein, SQL query statement analysis, including: Collect SQL statements: real-time acquisition of historical query records from Presto log files, including query statements and query time; Word segmentation processing: pre-processing of SQL query statements, removing unnecessary characters, punctuation marks or white spaces; decompose the SQL query statement into words or terms, and the result of decomposition is a word list; Build vocabulary: classify the decomposed words; according to the classification result, extract the field and table name, and build the vocabulary.

3. The method for Hudi asynchronous data clustering based on hot spot prediction according to claim 1, wherein, Establishing embedding layer refers to Word2Vec model construction, training and application, including: Create context-target pairs: context is the surrounding words of the target word, and target is the target word itself; Train Word2Vec model: use the context-target pair data to train Word2Vec model; Learn word vectors: Word2Vec model itself learns word vectors through iterative training process, updates word vectors according to the loss gradient of current context-target pair, and minimizes loss function; Get word vectors: after training, extract the word vector representation of each word from the Word2Vec model; Apply word vectors: use the trained word vectors as the input of the LSTM model based on online learning algorithm.

4. The method for Hudi asynchronous data clustering based on hot spot prediction according to claim 1, characterized in that, Based on the SQL statement obtained and parsed, the LSTM model based on online learning algorithm is used to predict the query hot field and query hot table, including: The LSTM model based on the online learning algorithm comprises a first layer LSTM network, a second layer LSTM network, a third layer LSTM network, a full connection layer, and different numbers of neurons are arranged in the three-layer LSTM network; The first layer LSTM network: the input data is first transmitted to the first layer LSTM network, and the first layer LSTM network is used for processing of the input sequence, information extraction, and long-term dependency modeling; The second layer LSTM network: the output of the first layer LSTM network is transmitted to the second layer LSTM network, and the second layer LSTM network is used for feature extraction and abstraction and representation learning; The third layer LSTM network: the output of the second layer LSTM network is transmitted to the third layer LSTM network, and the third layer LSTM network is used for further feature extraction and enhancement of the model capability; The full connection layer: the full connection layer is used for output prediction, decision-making, and analysis, and the full connection layer further comprises a Relu activation function.

5. The method for Hudi asynchronous data clustering based on hot spot prediction according to claim 1, characterized in that, The trained LSTM model based on the online learning algorithm is used to predict query hot fields and query hot tables, comprising: First, according to the query time in the historical query record, the same time interval is divided, the occurrence number of the word vector in each time interval obtained through the Word2Vec model is summarized, the time interval and the corresponding word vector and the occurrence number of the word vector are saved as a CSV format, and the input data set of the LSTM model based on the online learning algorithm is obtained; and the input data set is preprocessed, that is, the input data set is normalized, and the normalized data is subjected to a sliding window operation; Second, the LSTM model is used as a prediction model, and the word vector and its occurrence number in the corresponding time interval are input into the trained LSTM model, and the word vector and its occurrence number that will appear in the future period of time are output; The output results of the prediction model are summed, all the word vectors and their occurrence numbers that will appear in the future period of time are summarized, and the corresponding fields or tables with a query number greater than a threshold value are sequentially sorted in descending order of the occurrence number, and the query hot data is defined; Finally, the output query hot data of the LSTM model is subjected to word embedding reverse solution, the word vector corresponding to the word text in the vocabulary table is found, and the word vector in the output is finally saved in the form of word text.

6. The method for Hudi asynchronous data clustering based on hot spot prediction according to claim 1, characterized in that, An online learning cycle is added to the LSTM model, comprising: First, new data points are obtained from the real-time acquired historical query record, and the new data points are transmitted to the LSTM model for forward propagation, and the loss of the forward propagation is calculated; Second, the gradient of the loss function is propagated to each weight parameter in the reverse direction using the back propagation algorithm, and the gradient is the partial derivative of the loss function with respect to the weight; and the weights of the LSTM model are updated using the stochastic gradient descent; Finally, the learning rate scheduling method is used to dynamically adjust the learning speed. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method for Hudi asynchronous data clustering based on hot spot prediction according to any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method for Hudi asynchronous data clustering based on hot spot prediction according to any one of claims 1-6.

9. A system for Hudi asynchronous data clustering based on hotspot prediction, the system comprising: Comprise: The SQL query statement parsing module is configured to obtain and parse the SQL query statement; The model construction module is configured to construct and train the LSTM model, and add an online learning loop in the LSTM model; The prediction module is configured to predict the hot query field and the hot query table by using the trained LSTM model based on the online learning algorithm; The data clustering module is configured to automatically execute asynchronous data clustering analysis multiple times based on the prediction of the hot query field and the hot query table by using the trained LSTM model based on the online learning algorithm; including: Local python connects the Linux virtual machine; Local python connects the MySQL library in the Linux virtual machine; The query hot field and the table involved in the final output of the LSTM model based on the online learning algorithm are respectively summarized into a new file, and are named as a hot field data table and a hot table data table; The hot field data table and the hot table data table are saved to the MySQL database, and are taken as new data files to be imported into the Hudi by Flink to generate a plurality of small data files; And they are written into a new partition file group, when the submission times of data writing reach a plurality of times, python automatically sends Linux commands to the Hudi to generate a clustering analysis execution plan; At the same time of generating the clustering analysis execution plan, the plan is read and asynchronous data clustering is executed.

Images