Data storage method based on PaaS cloud platform and data lake technology

Abstract

The invention discloses a data storage method based on a PaaS cloud platform and a data lake technology. The data storage method comprises the steps that original data of various sources in the enterprise operation process is imported into a data lake on the PaaS cloud platform through an ETL tool; cleaning the original data in the data lake by using a big data processing framework on the PaaS cloud platform to obtain standardized data; storing the standardized data in a disk of a B + tree partition index deployed by the PaaS cloud platform; when the data is queried from the disk, a data query comprehensive evaluation index is established according to the data query efficiency ratio, the index selection cost and the index size, and the B + tree partition index is optimized according to the data query comprehensive evaluation index. According to the method, the data is stored in the PaaS cloud platform through a data lake technology, so that the quality of the stored data, the access and query efficiency of the data and the security of data storage and transmission are improved.

Description

Technical Field

[0001] The present invention belongs to the field of data lake technology, and specifically relates to a data storage method based on PaaS cloud platform and data lake technology. Background Art

[0002] Data lake is a data storage and analysis architecture that has emerged in the era of big data. It aims to address the limitations and challenges of traditional data warehouses. In the past, enterprises often stored data in various isolated systems, resulting in data silos and data integration difficulties. With the continuous growth of data scale and the diversification of data types, the traditional centralized data warehouse architecture has become difficult to adapt to the needs of modern data processing. Therefore, data lake technology came into being.

[0003] The concept of data lake was first proposed by James Dixon in 2010. He likened the data lake to a boundless storage pool that can accommodate various structured and unstructured data without pre-defining the structure and mode of the data. Unlike traditional data warehouses, data lakes accept raw data input and save it in its original form, retaining the integrity and flexibility of the data. This architecture allows enterprises to store large amounts of data in a centralized manner and provide a unified data source for data analysis and mining.

[0004] With the popularization of the Internet and the acceleration of digital transformation, enterprises and organizations are facing more and more data, including structured data, semi-structured data and unstructured data. Traditional data warehouses often cannot meet the storage and processing needs of these data because they need to define the data model and structure in advance and are not flexible and scalable enough. The emergence of distributed storage and processing frameworks provides the infrastructure for the implementation of data lakes. These frameworks allow data lakes to store and process large-scale data on hundreds or thousands of servers, with high fault tolerance and scalability. In addition, the development of cloud computing has also facilitated the deployment and management of data lakes, allowing enterprises to quickly build and expand the infrastructure of data lakes and flexibly adjust storage and computing resources according to demand.

[0005] However, since the data lake accepts raw data from various data sources, the control and management of data quality becomes a complex task. The data lake may contain poor data quality, missing values, inconsistencies, and erroneous data, which will cause trouble for subsequent analysis and decision-making, and reduce the credibility and effectiveness of the data lake. In addition, existing data lake technologies also face challenges in data partitioning indexing and secure storage and transmission, which seriously affects the performance of the data lake and the efficiency of data management. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention provides a data storage method based on PaaS cloud platform and data lake technology, which stores data in the PaaS cloud platform through data lake technology, thereby improving the quality of stored data, data access and query efficiency, and the security of data storage and transmission.

[0007] To achieve the above technical objectives, the present invention adopts the following technical solution: a data storage method based on PaaS cloud platform and data lake technology, specifically comprising the following steps:

[0008] Step S1: Use ETL tools to import raw data from various sources in the enterprise operation process into the data lake on the PaaS cloud platform;

[0009] Step S2: Use the big data processing framework on the PaaS cloud platform to clean the raw data in the data lake to obtain standardized data;

[0010] Step S3: Store the standardized data in a disk of a B+ tree partition index deployed on the PaaS cloud platform;

[0011] Step S4: when querying data from the disk, a data query comprehensive evaluation index is established according to the data query efficiency ratio, index selection cost and index size, and the B+ tree partition index is optimized according to the data query comprehensive evaluation index.

[0012] Furthermore, the process of cleaning the original data in step S2 includes: deduplication of original data, filling of missing values ​​of original data, detection and processing of outliers of original data, standardization of format and encoding of original data, consistency check and repair of original data, integrity check and processing of original data, and conversion and integration of original data.

[0013] Furthermore, step S3 includes the following sub-steps:

[0014] Step S3.1, determine the index key according to the characteristics of the standardized data and the query requirements;

[0015] Step S3.2: Use the index key to partition and store the standardized data, and write the partition information into the index file.

[0016] Furthermore, the process of establishing the data query comprehensive evaluation index is as follows:

[0017] PSE=W q *QWE+W c *(1-ISC norm )+W s *(1-IS norm )

[0018] Among them, PSE represents the comprehensive evaluation index of data query, QWE represents the data query efficiency improvement ratio, and W q represents the weight of QWE, ISC norm represents the standardized index selection cost, ISC represents the index selection cost, T baseline Indicates the base time, W c ISC norm The weight of IS norm represents the normalized index size, IS table index size, S bseline Indicates the index base size, W s Indicates IS norm The weight, W q +W c +W s =1.

[0019] Furthermore, the calculation process of the data query efficiency improvement ratio QWE is:

[0020] QER=(T_all-T_partition) / T_all

[0021] Among them, T_all represents the time required to query all data when B+ tree partition index is not performed, and T_partition represents the time required to query all data when B+ tree partition index is performed.

[0022] Furthermore, the calculation process of the index selection cost ISC is:

[0023] ISC=C_seek+C_read*P_hit

[0024] Among them, C_seek represents the time required to find the index key value, C_read represents the time to read data from the disk, and P_hit represents the probability of the query hitting the partition.

[0025] Furthermore, the calculation process of the index size IS is as follows:

[0026] IS=(R_n*R_size) / B_size

[0027] Among them, R_n represents the number of records in the index, R_size represents the space occupied by each record in the index, and B_size represents the size of the corresponding node in the B+ tree.

[0028] Furthermore, in step S4, the specific process of optimizing the B+ tree partition index according to the data query comprehensive evaluation index is as follows: if the calculated data query comprehensive evaluation index value is less than 0.6, the B+ tree partition index needs to be optimized to determine whether the index fill rate of the partition where the query data is located is less than 0.5. If so, the partition where the query data is located is merged with its adjacent partitions until the data query comprehensive evaluation index is greater than or equal to 0.6; otherwise, the partition where the query data is located is split by binary search until the data query comprehensive evaluation index is greater than or equal to 0.6.

[0029] Furthermore, the index fill rate of the partition where the query data is located represents the ratio of the actual data volume in the partition to the maximum capacity of the partition.

[0030] Furthermore, in the process of standardized data storage, asymmetric encryption is used to encrypt and save the data, and when the data is queried from the disk, the data is encrypted and transmitted using symmetric encryption.

[0031] Compared with the prior art, the present invention has the following beneficial effects: the data storage method based on the PaaS cloud platform and data lake technology of the present invention stores data in the data lake of the PaaS cloud platform, ensuring that all data can be stored in the data lake; in addition, in order to improve the quality and accuracy of the data in the data lake, the original data in the data lake is cleaned, so that the data stored in the data lake can better provide guidance for enterprise analysis and decision-making; further, the present invention also optimizes the B+ tree partition index used by the data lake through the data query comprehensive evaluation index, comprehensively considers the impact of multiple factors on data query in the data lake, realizes stable data partitioning and efficient retrieval capabilities, improves data access and query efficiency, and copes with the problem of poor performance of existing data lakes in complex query and multidimensional analysis scenarios. Therefore, the data storage method of the present invention improves the quality of stored data, the access and query efficiency of data, and the security of data storage and transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of the data storage method based on the PaaS cloud platform and data lake technology of the present invention;

[0033] Figure 2 The present invention is a flowchart for optimizing the B+ tree partition index according to the comprehensive evaluation index of data query. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0035] like Figure 1This is a flow chart of a data storage method based on the PaaS cloud platform and data lake technology of the present invention. The data storage method specifically includes the following steps:

[0036] Step S1: Use ETL tools to import raw data from various sources such as databases, log files, sensors, and social media during enterprise operations into the data lake on the PaaS cloud platform to ensure that the raw data can be smoothly imported from various data sources into the data lake.

[0037] Step S2, using the big data processing framework on the PaaS cloud platform to clean the raw data in the data lake, obtain standardized data, improve data quality and accuracy, in order to meet the challenges in data quality and data governance, so that the data can better support the analysis and decision-making of the enterprise. The process of cleaning the raw data in the present invention includes: deduplication of raw data, missing value filling of raw data, detection and processing of abnormal values ​​of raw data, format and coding standardization of raw data, consistency check and repair of raw data, integrity check and processing of raw data, and conversion and integration of raw data, thereby eliminating duplicate records in the raw data, reducing redundancy, and being able to improve data specifications and improve data quality.

[0038] The raw data imported into the data lake may contain a large number of duplicate records, which will affect the accuracy of data analysis. Therefore, the raw data needs to be deduplicated. The specific process is: use a hash algorithm to detect and identify duplicate data records in the raw data, formulate deduplication rules based on business logic, such as retaining the latest or earliest records, and apply deduplication rules to delete duplicate raw data, thereby eliminating duplicate records in the raw data, reducing redundancy, and improving data quality.

[0039] The specific process of filling missing values ​​in raw data is: by scanning the raw data, detecting the records and fields where the missing values ​​are located, and selecting appropriate filling methods according to the raw data type and business needs, such as mean filling, median filling, previous and next value filling, or using machine learning models for prediction filling. While filling missing values, a flag can be added to the raw data to record which values ​​are filled for subsequent analysis and processing. By filling missing values ​​in raw data, deviations or errors caused by missing data during the analysis process can be avoided.

[0040] The specific process of standardizing the format and encoding of raw data is as follows: standardize the date field into the YYYY-MM-DD format, standardize the currency unit into the same unit, standardize the encoding methods that may exist in different data sources, such as character encoding, geographic location encoding, etc., and convert raw data of different units in a unified manner to ensure the consistency of data format and facilitate subsequent data processing and analysis.

[0041] The specific process of outlier detection and processing of raw data is: use statistical methods such as standard deviation method, box plot method or machine learning algorithm to detect outliers in data, analyze the detected outliers, and decide whether to delete, replace or correct them. Outliers can be replaced with the mean or median of adjacent data. Flags can also be added to the raw data to record which values ​​are processed outliers, identify and process outliers in the data, and ensure the authenticity and reliability of the data.

[0042] The specific process of consistency check and repair of raw data is as follows: formulate consistency check rules for raw data based on business logic. For example, the information of a customer in different data tables should be consistent. Through consistency check rules, check the raw data, find out inconsistent data records, repair inconsistent data records, ensure data consistency, ensure the consistency of raw data between different tables and different sources, and improve the overall data quality.

[0043] The specific process of checking and processing the integrity of raw data is as follows: formulate raw data integrity rules according to business needs, such as foreign key constraints, uniqueness constraints, etc. Apply integrity rules to check raw data to ensure that the integrity constraints of raw data are not violated; repair raw data that violates integrity constraints, such as filling in missing related data, so as to ensure the integrity of raw data and avoid business logic errors caused by incomplete raw data.

[0044] The specific process of raw data conversion and integration is as follows: according to business needs, make necessary conversions on raw data, such as data type conversion, unit conversion, etc., integrate data from different data sources, eliminate data silos, and form a unified and comprehensive data set.

[0045] At the same time, the present invention can record every operation step of the original data in the cleaning process, including: data modification, deletion and addition operations, and regularly audit the cleaning log to ensure the traceability and transparency of the cleaning process.

[0046] Step S3, store the standardized data in the disk of the B+ tree partition index deployed on the PaaS cloud platform. The B+ tree is a multi-way balanced search tree with the following characteristics: all leaf nodes are located at the same layer, and the leaf nodes contain all keyword information and pointers to the corresponding data; non-leaf nodes are only used for indexing and do not contain actual data information; adjacent leaf nodes are connected to each other through pointers to facilitate range queries. The B+ tree partition index divides the data into multiple partitions, each partition contains a part of the data, and each partition can be processed and queried independently. When querying, only the required partitions need to be scanned instead of scanning the entire data set, which can reduce the amount of data scanning, thereby greatly improving data query efficiency. Specifically includes the following sub-steps:

[0047] Step S3.1, determine the index key according to the characteristics of the standardized data and the query requirements;

[0048] Step S3.2: Use the index key to partition and store the standardized data, and write the partition information into the index file.

[0049] Step S4: With the popularity of data lakes and the continuous increase in data volume, static B+ tree partition indexing technology is prone to imbalanced data partitioning, resulting in a large number of small files or cold data partitions, and prolonging query response time. Therefore, data query has become an increasingly important issue. In order to maintain the high performance of the index, when querying data from the disk, a comprehensive data query evaluation index is established based on the data query efficiency ratio, index selection cost, and index size, and the B+ tree partition index is optimized based on the comprehensive data query evaluation index. The closer the value of the comprehensive data query evaluation index is to 1, the better the B+ tree partition indexing technology is in improving the stability and efficiency of data query. By comprehensively considering the impact of multiple factors on data query in the data lake, the data partitions should be merged or split based on the comprehensive data query evaluation index to improve data access and query efficiency, and achieve stable data partitioning and efficient retrieval capabilities to cope with the problem of poor performance in complex query and multidimensional analysis scenarios. Figure 2 If the calculated data query comprehensive evaluation index value is less than 0.6, it is necessary to optimize the B+ tree partition index to determine whether the index fill rate of the partition where the query data is located is less than 0.5. If so, the node of the partition where the query data is located may increase the height of the tree due to too few data items, thereby affecting the query efficiency. The partition where the query data is located is merged with its adjacent partitions until the data query comprehensive evaluation index is greater than or equal to 0.6; otherwise, the partition where the query data is located is too large. Too large a partition will cause each query to process more data items, reducing the query speed. Therefore, the partition where the query data is located is split by binary search until the data query comprehensive evaluation index is greater than or equal to 0.6; by merging or splitting the partition where the query data is located, the balance of the B+ tree partition index can be ensured and the data query time can be reduced. Among them, the index fill rate of the partition where the query data is located represents the proportion of the actual data volume in the partition to the maximum capacity of the partition.

[0050] The query efficiency improvement ratio QWE is used to measure the improvement degree of the data query speed by the B+ tree partition index technology. The higher the query efficiency improvement ratio QWE is, the more obvious the improvement effect of the B+ tree partition index technology on the data query speed is. The calculation process of the data query efficiency improvement ratio QWE is as follows:

[0051] QER=(T_all-T_partition) / T_all

[0052] Among them, T_all represents the time required to query all data when B+ tree partition index is not performed, and T_partition represents the time required to query all data when B+ tree partition index is performed.

[0053] The index selection cost ISC is used to evaluate the query cost when using the B+ tree partition index technology. The lower the index selection cost ISC, the lower the query cost of using the index technology. The calculation process of the index selection cost ISC is as follows:

[0054] ISC=C_seek+C_read*P_hit

[0055] Among them, C_seek represents the time required to find the index key value, C_read represents the time to read data from the disk, and P_hit represents the probability of the query hitting the partition.

[0056] The index size IS is calculated by the number of records in the index and the space occupied by each record in the index. A reasonable index size can improve query efficiency while reducing the storage space occupied by the index. The calculation process of the index size IS is as follows:

[0057] IS=(R_n*R_size) / B_size

[0058] Among them, R_n represents the number of records in the index, R_size represents the space occupied by each record in the index, and B_size represents the size of the corresponding node in the B+ tree.

[0059] In order to better meet the stability and efficiency requirements of the data lake under the PaaS cloud platform, a comprehensive data query evaluation index combining the query efficiency improvement ratio, index selection cost and index size is introduced. The stability and efficiency of data query are evaluated by the data query comprehensive evaluation index. The higher the value of the data query comprehensive evaluation index, the better the effect of B+ tree partition index technology in improving data query stability and efficiency. For partitions with a data query comprehensive evaluation index value less than 0.6, there may be problems with the partition being too large or too small. By dynamically adjusting the partition where the query data is located, the performance requirements in complex query and multidimensional analysis scenarios are met, ensuring the stability of data partitions and the high efficiency of retrieval.

[0060] The process of establishing the comprehensive evaluation index of data query in the present invention is as follows:

[0061] PSE=W q *QWE+W c *(1-ISC norm )+W s *(1-IS norm )

[0062] Among them, PSE represents the comprehensive evaluation index of data query, QWE represents the data query efficiency improvement ratio, and W q represents the weight of QWE, ISC norm represents the standardized index selection cost, ISC represents the index selection cost, T baseline Indicates the base time, W c ISC norm The weight of IS norm represents the normalized index size, IS table index size, S bseline Indicates the index base size, W s Indicates IS norm The weight, W q +W c +W s =1, maintain the validity of the comprehensive evaluation indicators of data query.

[0063] In one technical solution of the present invention, asymmetric encryption is used to encrypt and save data during standardized data storage, and when data is queried from the disk, symmetric encryption is used to encrypt and transmit data, thereby achieving secure storage and transmission of data and protecting the confidentiality and integrity of the data.

[0064] The data storage method based on PaaS cloud platform and data lake technology of the present invention stores data in PaaS cloud platform through data lake technology, and dynamically optimizes the size of B+ tree partition through comprehensive evaluation index of data query, thereby improving the quality of stored data, data access and query efficiency, and security of data storage and transmission, and can obtain the data required for query in the shortest time, which can help enterprises better cope with challenges in data management and analysis and improve the availability and value of data.

[0065] In a technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program, wherein the computer program enables a computer to execute the data storage method based on the PaaS cloud platform and data lake technology.

[0066] In a technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data storage method based on the PaaS cloud platform and data lake technology is implemented.

[0067] In a technical solution of the present invention, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the data storage method based on the PaaS cloud platform and data lake technology.

[0068] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0069] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0070] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A data storage method based on PaaS cloud platform and data lake technology, characterized in that: The specific steps include: Step S1: Use ETL tools to import raw data from various sources in the enterprise operation process into the data lake on the PaaS cloud platform; Step S2: Use the big data processing framework on the PaaS cloud platform to clean the raw data in the data lake to obtain standardized data; Step S3: Store the standardized data in a disk of a B+ tree partition index deployed on the PaaS cloud platform; Step S4: when querying data from the disk, a data query comprehensive evaluation index is established according to the data query efficiency ratio, index selection cost and index size, and the B+ tree partition index is optimized according to the data query comprehensive evaluation index.

2. According to claim 1, a data storage method based on PaaS cloud platform and data lake technology is characterized in that: The process of cleaning the raw data in step S2 includes: deduplication of raw data, filling of missing values ​​of raw data, detection and processing of outliers of raw data, format and coding standardization of raw data, consistency check and repair of raw data, integrity check and processing of raw data, and conversion and integration of raw data.

3. According to claim 1, a data storage method based on PaaS cloud platform and data lake technology is characterized in that: Step S3 includes the following sub-steps: Step S3.1, determine the index key according to the characteristics of the standardized data and the query requirements; Step S3.2: Use the index key to partition and store the standardized data, and write the partition information into the index file.

4. The data storage method based on PaaS cloud platform and data lake technology according to claim 1 is characterized in that: The process of establishing the data query comprehensive evaluation index is as follows: PSE=W q *QWE+W c *(1-ISC norm )+W s *(1-IS norm ) Among them, PSE represents the comprehensive evaluation index of data query, QWE represents the data query efficiency improvement ratio, and W q represents the weight of QWE, ISC norm represents the standardized index selection cost, ISC represents the index selection cost, T baseline Indicates the base time, W c ISC norm The weight of IS norm represents the normalized index size, IS table index size, S bseline Indicates the index base size, W s Indicates IS norm The weight, W q +W c +W s =1.

5. The data storage method based on PaaS cloud platform and data lake technology according to claim 4 is characterized in that: The calculation process of the data query efficiency improvement ratio QWE is as follows: QER=(T_all-T_partition) / T_all Among them, T_all represents the time required to query all data when B+ tree partition index is not performed, and T_partition represents the time required to query all data when B+ tree partition index is performed.

6. The data storage method based on PaaS cloud platform and data lake technology according to claim 4 is characterized in that: The calculation process of the index selection cost ISC is: ISC=C_seek+C_read*P_hit Among them, C_seek represents the time required to find the index key value, C_read represents the time to read data from the disk, and P_hit represents the probability of the query hitting the partition.

7. The data storage method based on PaaS cloud platform and data lake technology according to claim 4 is characterized in that: The calculation process of the index size IS is as follows: IS=(R_n*R_size) / B_size Among them, R_n represents the number of records in the index, R_size represents the space occupied by each record in the index, and B_size represents the size of the corresponding node in the B+ tree.

8. The data storage method based on PaaS cloud platform and data lake technology according to claim 3 is characterized in that: The specific process of optimizing the B+ tree partition index according to the data query comprehensive evaluation index in step S4 is as follows: if the calculated data query comprehensive evaluation index value is less than 0.6, the B+ tree partition index needs to be optimized to determine whether the index fill rate of the partition where the query data is located is less than 0.

5. If so, merge the partition where the query data is located with its adjacent partitions until the data query comprehensive evaluation index is greater than or equal to 0.6; otherwise, split the partition where the query data is located by binary search until the data query comprehensive evaluation index is greater than or equal to 0.

6.

9. The data storage method based on PaaS cloud platform and data lake technology according to claim 8, characterized in that: The index fill rate of the partition where the query data is located represents the ratio of the actual data volume in the partition to the maximum capacity of the partition.

10. The data storage method based on PaaS cloud platform and data lake technology according to claim 1, characterized in that: In the process of standardized data storage, asymmetric encryption is used to encrypt and save data. When querying data from the disk, symmetric encryption is used to encrypt and transmit data.

Images