Zipper table-based data storage method and device, network equipment and storage medium
By using the combination of historical zipper list and Hudi components in the data storage, data is acquired and updated, and the problem of high resource consumption in the existing technology is solved, and efficient and convenient data processing and storage is achieved.
Patent Information
- Application Number
- CN202411423319.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-01-14
AI Technical Summary
When using zipper lists for data storage, network resources and disk resources consume a lot, resulting in poor convenience and timeliness of data storage.
Obtain historical zipper data through historical zipper lists, and use the data lake to obtain mirror data of incremental data. Use Hudi components to update historical zipper data to achieve efficient update of zipper lists and reduce resource consumption.
It improves data processing efficiency and convenience, supports real-time processing, reduces the resource utilization of data accumulation, has efficient incremental data processing capabilities, and reduces hardware costs.
Smart Images
Figure CN119311693A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data storage method, device, network equipment and storage medium based on a zipper table. Background Art
[0002] With the development of science and technology, more and more services are provided to users, and improving service quality has become the focus of users. Among them, zipper tables are used for data storage, and the zipper table can, for example, maintain the historical state of the table. For example, the calculation of the zipper table can be implemented through the hive component, but the network resources and disk resources required in this method are large, making the convenience of data storage poor and the timeliness poor. Summary of the invention
[0003] The present disclosure provides a data storage method, device, network device and storage medium based on a zipper table, which can improve data processing efficiency and convenience without processing the entire amount of data. The technical solution of the present disclosure is as follows:
[0004] According to a first aspect of an embodiment of the present disclosure, a data storage method based on a zipper table is provided, comprising:
[0005] Obtain historical zipper data through the historical zipper table;
[0006] Obtain incremental data corresponding to historical zipper data, and use the data lake to obtain mirror data corresponding to the incremental data;
[0007] According to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0008] According to some embodiments, obtaining incremental data corresponding to historical zipper data includes:
[0009] The incremental data corresponding to the historical zipper data is acquired according to the incremental data acquisition method corresponding to the historical zipper data.
[0010] According to some embodiments, acquiring the incremental data corresponding to the data set according to the incremental data acquisition method corresponding to the historical zipper data includes:
[0011] When the incremental data corresponding to the historical zipper data is acquired in a JDBC manner, the incremental data corresponding to the historical zipper data is acquired in a micro-batch form;
[0012] or
[0013] Through the Spark data processing component or the Flink data processing component, log data is used to obtain incremental data corresponding to the historical zipper data, and the incremental data is written into the data lake, wherein the incremental data is in the form of a stream, and the data storage format of the incremental data includes at least one of parquet and orc.
[0014] According to some embodiments, the updating of the historical zipper data using the hudi component corresponding to the data lake according to the comparison result between the historical zipper data and the mirror data to obtain an updated zipper table includes:
[0015] Obtaining data types in a comparison result between the historical zipper data and the mirror data;
[0016] The historical zipper data is updated by adopting an update method corresponding to the data type and a hudi component corresponding to the data lake to obtain an updated zipper table.
[0017] According to some embodiments, updating the historical zipper data to obtain an updated zipper table includes:
[0018] Acquire update information of the historical zipper table, wherein the update information includes update time and update method of the historical zipper table;
[0019] The historical zipper data is updated using the update information to obtain an updated zipper table.
[0020] According to some embodiments, updating the historical zipper data to obtain an updated zipper table includes:
[0021] Obtaining application scenarios corresponding to the historical zipper table;
[0022] A target read-write mode corresponding to the application scenario is obtained, and the read-write mode of the history zipper table is set to the target read-write mode.
[0023] According to a second aspect of an embodiment of the present disclosure, there is provided a data storage device based on a zipper table, comprising:
[0024] A data acquisition unit, used for acquiring historical zipper data through a historical zipper table;
[0025] The data acquisition unit is further used to acquire incremental data corresponding to the historical zipper data, and use the data lake to acquire mirror data corresponding to the incremental data;
[0026] The zipper table updating unit is used to update the historical zipper data according to the comparison result between the historical zipper data and the mirror data by using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0027] According to a third aspect of an embodiment of the present disclosure, a network device is provided, including:
[0028] processor;
[0029] a memory for storing instructions executable by the processor;
[0030] The processor is configured to execute the instructions to implement the data storage method based on a zipper table as described in any one of the aforementioned aspects.
[0031] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of a network device, the network device is enabled to execute the zipper table-based data storage method described in any one of the preceding aspects.
[0032] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the preceding aspects when executed by a processor.
[0033] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0034] In some or related embodiments, historical zipper data is obtained through a historical zipper table; incremental data corresponding to the historical zipper data is obtained, and the data lake is used to obtain mirror data corresponding to the incremental data; based on the comparison result of the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table. Therefore, the update of the zipper table can be implemented by using a combination of hadoop and hudi components, which can increase the capacity of the data, and the use of hudi components can support real-time processing of data, reduce the occupation of resources by data accumulation, and the hudi component has efficient incremental data processing capabilities, without the need to process the full amount of data, which can reduce resource consumption, and the hudi component can be compatible with a variety of data processing components, which can improve data processing efficiency and convenience.
[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0037] Figure 1 is a flow chart showing a data storage method based on a zipper table according to an exemplary embodiment;
[0038] Figure 2 is a flow chart showing a data storage method based on a zipper table according to an exemplary embodiment;
[0039] Figure 3 is a schematic diagram showing an example of a data storage method based on a zipper table according to an exemplary embodiment;
[0040] Figure 4 is a block diagram of a data storage device based on a zipper table according to an exemplary embodiment;
[0041] Figure 5 The invention is a block diagram of a network device according to an exemplary embodiment. DETAILED DESCRIPTION
[0042] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0043] The embodiments of the present disclosure propose a data storage method, device, network device and storage medium based on a zipper table. In some embodiments, the data storage method based on a zipper table and the terms such as information processing method, communication method, etc. can be replaced with each other, the data storage device based on a zipper table and the terms such as information processing device, communication device, etc. can be replaced with each other, and the terms such as information processing system, communication system, etc. can be replaced with each other.
[0044] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.
[0045] In each embodiment of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.
[0046] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.
[0047] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular form, such as "a", "an", "the", "above", "said", "aforementioned", "this", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun after the article may be understood as a singular expression or a plural expression.
[0048] In the embodiments of the present disclosure, “plurality” refers to two or more.
[0049] In some embodiments, the terms “at least one,” “one or more,” “a plurality of,” “multiple,” etc. may be used interchangeably.
[0050] In some embodiments, "at least one of A and B", "A and / or B", "A in one case, B in another case", "in response to one case A, in response to another case B", etc., may include the following technical solutions according to the situation: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, execution is selected from A and B (A and B are selectively executed); in some embodiments, A and B (both A and B are executed). When there are more branches such as A, B, C, etc., the above is also similar.
[0051] In some embodiments, devices and equipment may be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. In some cases, they may also be understood as "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", etc.
[0052] In some embodiments, "terminal" or "terminal device" can be referred to as "user equipment (UE)", "user terminal" "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, etc.
[0053] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0055] Figure 1 is a flow chart showing a data storage method based on a zipper table according to an exemplary embodiment. Figure 1 As shown, the following steps are included:
[0056] In step S11, historical zipper data is obtained through the historical zipper table;
[0057] According to some embodiments, the execution subject of the embodiments of the present disclosure may be, for example, a network device. The number of network devices is not limited. For example, the network device may be a device that executes the solution alone, or may be a plurality of network devices that collaboratively execute the solution. The name of the network device is not limited. For example, the name of the network device may also be a server, or a server cluster.
[0058] In some embodiments, the zipper table may be, for example, a table for maintaining the historical state of data in the table and the latest state data. The zipper table may remove a portion of unchanged records according to different zipper granularities, and may easily restore the data records at the zipper time point. The name of the zipper table is not limited, for example, the zipper table may also be called a track table.
[0059] According to some embodiments, the historical zipper table can be used to indicate that the zipper table was established before the current time point, or it can be used to maintain data within the historical time period. The historical zipper table does not specifically refer to a fixed zipper table. For example, when the historical time period corresponding to the historical zipper table changes, the historical zipper table can also change accordingly. For example, when a certain data in the historical zipper table changes, the historical zipper table can also change accordingly.
[0060] In some embodiments, the zipper data may be, for example, data stored in a zipper table. The zipper data does not specifically refer to a fixed data. The zipper data may also be, for example, a zipper data set, or at least one zipper data, etc., which is not limited in the embodiments of the present disclosure.
[0061] According to some embodiments, the historical zipper data may be used to indicate data stored in a historical zipper table, for example. The historical zipper data may also be referred to as a historical zipper data set, etc., which is not limited in the embodiments of the present disclosure.
[0062] In some embodiments, for example, the historical zipper data may be obtained through a historical zipper table. For example, when the historical zipper table is obtained, the historical zipper data in the historical zipper table may be read.
[0063] In step S12, incremental data corresponding to the historical zipper data is obtained, and mirror data corresponding to the incremental data is obtained using the data lake;
[0064] In some embodiments, the incremental data may be used, for example, to indicate data that has changed or been updated corresponding to the historical zipper data. The incremental data does not specifically refer to a fixed data. For example, when the acquisition duration corresponding to the incremental data changes, the incremental data may also change accordingly. For example, when the data that has changed or been updated changes, the incremental data may also change accordingly.
[0065] According to some embodiments, a data lake can be, for example, a repository or system that stores data in its original format, which can store data as is without structuring the data in advance. Among them, a data lake can store structured data, semi-structured data, unstructured data, binary data, etc., and is based on the Hadoop ecosystem and supports the technical framework of data deletion and incremental update of HDFS.
[0066] In some embodiments, the mirror data may be used to indicate data that is identical to the incremental data. The mirror data does not specifically refer to a fixed data. For example, when the incremental data changes, the mirror data may also change accordingly. For example, when the method for obtaining the mirror data changes, the mirror data may also change accordingly.
[0067] According to some embodiments, incremental data corresponding to historical zipper data is obtained, and a data lake is used to obtain mirror data corresponding to the incremental data.
[0068] In step S13, according to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0069] In some embodiments, the comparison result may be, for example, a result obtained after comparing the historical zipper data and the mirror data. The comparison result does not specifically refer to a fixed result. For example, when the comparison method of the historical zipper data and the mirror data changes, the comparison result may also change accordingly. For example, when the historical zipper data or the mirror data changes, the comparison result may also change accordingly.
[0070] According to some embodiments, the hudi component is an open source data storage layer that can update, delete, and incrementally process data sets on Hadoop and HDFS. Different processing scenarios can correspond to different hudi components.
[0071] According to some embodiments, the updated zipper table may be, for example, a zipper table obtained after updating the data in the historical zipper table. The updated zipper table does not specifically refer to a fixed zipper table. For example, when the historical zipper table changes, the updated zipper table may also change accordingly. For example, when the updated historical zipper data changes, the updated zipper table may also change accordingly.
[0072] In some embodiments, according to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0073] In some or related embodiments, historical zipper data is obtained through a historical zipper table; incremental data corresponding to the historical zipper data is obtained, and the data lake is used to obtain the mirror data corresponding to the incremental data; according to the comparison result of the historical zipper data and the mirror data, the hudi component corresponding to the data lake is used to update the historical zipper data to obtain an updated zipper table. Therefore, the update of the zipper table can be realized by using a set of hadoop and hudi components, which can increase the capacity of the data, and the use of hudi components can support real-time processing of data, reduce the occupation of resources by data accumulation, and the hudi component has efficient incremental data processing capabilities, which can reduce resource consumption, and can be compatible with a variety of data processing components, which can improve data processing efficiency and convenience. In addition, a smaller amount of data can be processed during the data transmission and calculation stages, and all changed data can be retained, making the data more real and valuable, and reducing hardware costs.
[0074] Figure 2 is a flow chart showing a data storage method based on a zipper table according to an exemplary embodiment. Figure 2 As shown, the following steps are included:
[0075] In step S21, historical zipper data is obtained through the historical zipper table;
[0076] The specific process is as described above and will not be repeated here.
[0077] According to some embodiments, Figure 3 is a schematic diagram showing an example of a data storage method based on a zipper table according to an exemplary embodiment. Figure 3As shown, the incremental data can be acquired in real time or in other ways, which is not limited in the embodiments of the present disclosure. Real-time acquisition can improve the timeliness of data processing, reduce the time it takes for the downstream system to acquire incremental data, and improve data processing efficiency. When the historical zipper table is acquired, the data can be initialized. For example, when the zipper table is made for the first time, all data in the library is synchronized to the table, and the time fields start_date and end_date are added. Start_date is assigned the data business time, and end_date is assigned '9999-12-31' to form historical zipper data.
[0078] In step S22, the incremental data corresponding to the historical zipper data is acquired according to the incremental data acquisition method corresponding to the historical zipper data;
[0079] According to some embodiments, the incremental data acquisition method can be used to indicate a method for acquiring incremental data, and the incremental data acquisition method does not specifically refer to a fixed method. For example, when the current data storage scenario changes, the incremental data acquisition method can also change accordingly. For example, when a modification instruction for the incremental data acquisition method is received, the incremental data acquisition method can also change accordingly.
[0080] In some embodiments, the incremental data corresponding to the historical zipper data is acquired according to the incremental data acquisition method corresponding to the historical zipper data.
[0081] According to some embodiments, acquiring the incremental data corresponding to the data set according to the incremental data acquisition method corresponding to the historical zipper data includes:
[0082] When the incremental data corresponding to the historical zipper data is acquired in a JDBC manner, the incremental data corresponding to the historical zipper data is acquired in a micro-batch form;
[0083] or
[0084] Through the Spark data processing component or the Flink data processing component, the incremental data corresponding to the historical zipper data is obtained using log data, and the incremental data is written into the data lake, wherein the incremental data is in the form of a stream, and the data storage format of the incremental data includes at least one of parquet and orc. Therefore, the incremental data can be obtained according to different incremental data acquisition methods, thereby improving the accuracy of incremental data acquisition.
[0085] In some embodiments, the micro-batch format may be limited to the number of acquired data, or when a certain amount of data is acquired, when incremental data is acquired, or when the number acquired within a certain period of time is less than a quantity threshold, the data of that amount may be eliminated. The quantity threshold does not specifically refer to a fixed threshold. For example, when a modification instruction for the quantity threshold is received, the quantity threshold may also change accordingly.
[0086] In some embodiments, the Spark data processing component may be, for example, a component for processing data, and the component may also be referred to as a framework, a model, a processing system, etc.
[0087] According to some embodiments, the log data may be acquired in the form of a data stream, for example.
[0088] According to some embodiments, the data storage format can be used to indicate the format used for data storage. The data storage format can be related to the application scenario of the downstream system, for example, it can also correspond to the big data platform, for example, it can also correspond to the demand information, and the embodiments of the present disclosure are not limited to this. Among them, the data storage format can include at least one of parquet and orc. The data storage format does not specifically refer to a fixed format. For example, the data storage format can be parquet, and the data storage format can also be orc. Among them, the data storage format of the embodiment of the present disclosure is not limited, and can also be the format supported by other hudi components.
[0089] In step S23, the data lake is used to obtain the mirror data corresponding to the incremental data;
[0090] The specific process is as described above and will not be repeated here.
[0091] In step S24, according to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0092] The specific process is as described above and will not be repeated here.
[0093] Among them, the syntax corresponding to the hudi component is richer, including but not limited to upsert, update, delete, commit, merge on read, copy on write and other merge calculation methods. Therefore, the hudi component can improve the accuracy of data analysis during analysis, and can perform real-time calculations on data. It can also adjust the syntax and update methods according to different scenarios. Among them, the hudi component can be deployed based on the hadoop big data cluster, and can be expanded according to the expansion of the hadoop cluster, which improves the convenience of hudi component layout. The data lake Hudi component supports real-time insertion, update and deletion operations of data, so that the data warehouse built on the data lake can process real-time data like a database, and provides efficient incremental data processing capabilities, supporting efficient incremental data writing and reading on large data sets.
[0094] In some embodiments, the Hadoop cluster is based on the distributed file system HDFS (Hadoop Distributed File System), and the storage capacity can be easily expanded by adding more servers. Therefore, the hudi component can be used to process data sets at the PB level or even larger. In addition, fault tolerance is taken into consideration when Hadoop is designed. In HDFS, data is divided into 128MB or 256MB blocks by default (adjustable according to the configuration), and these data blocks are copied to multiple nodes. Even if some nodes fail, the data can still be accessed through copies of other nodes. Therefore, the high fault tolerance of haddop is more suitable for big data scenarios, which can achieve 24-hour data availability and reduce the situation where short-term data cannot be accessed. Then, Hadoop supports a variety of data storage formats, such as SequenceFile, HDFS, Avro, etc., which enables it to store different types of data, including structured data, semi-structured data and unstructured data. Therefore, the scalability, high fault tolerance and flexible data storage format of the system are improved, and it is compatible with massive data sources such as mysql, oracle, and pg.
[0095] According to some embodiments, the updating of the historical zipper data using the hudi component corresponding to the data lake according to the comparison result between the historical zipper data and the mirror data to obtain an updated zipper table includes:
[0096] Obtaining data types in a comparison result between the historical zipper data and the mirror data;
[0097] The historical zipper data is updated by using an update method corresponding to the data type and a hudi component corresponding to the data lake to obtain an updated zipper table. Therefore, the historical zipper data can be updated according to the data type, thereby improving the accuracy of zipper data update and data storage.
[0098] In some embodiments, the data type may include, for example, type I data, type U data, and type UN data. Type I data is used to indicate data that has never appeared in the zipper table; type U data is used to indicate old data that has appeared in the table and needs to be updated; and type UN data is used to indicate that the primary key value is the same as the old data in the zipper table.
[0099] According to some embodiments, when updating the historical zipper table, for example, the data in the table may be processed. For Class I data, the time fields start_date and end_date are added, where start_date is assigned the data business time, and end_date is assigned '9999-12-31'. For Class U data, start_date is modified to the data business time, and end_date is assigned '9999-12-31'. For Class UN data, end_date is modified to the business time of Class U data with the same primary key, thereby forming a closed chain operation in real time or periodically.
[0100] According to some embodiments, updating the historical zipper data to obtain an updated zipper table includes:
[0101] Acquire update information of the historical zipper table, wherein the update information includes update time and update method of the historical zipper table;
[0102] The historical zipper data is updated using the update information to obtain an updated zipper table.
[0103] According to some embodiments, the update time may be used to indicate the time to update the historical zipper table. The update time may include, for example, updating the historical zipper table when the incremental data meets the requirements, or updating the historical zipper table once every preset time. The embodiments of the present disclosure are not limited to this.
[0104] Among them, you can use hudi's merge syntax when updating: zip the same data according to the primary key, and update different data.
[0105] According to some embodiments, updating the historical zipper data to obtain an updated zipper table includes:
[0106] Obtaining application scenarios corresponding to the historical zipper table;
[0107] The target read / write mode corresponding to the application scenario is obtained, and the read / write mode of the historical zipper table is set to the target read / write mode. Therefore, the zipper table used can be optimized by selecting copy on write or merge on read to meet the requirements of fast writing or fast reading.
[0108] In some related embodiments, according to the incremental data acquisition method corresponding to the historical zipper data, the incremental data corresponding to the historical zipper data is acquired, and the data lake is used to acquire the mirror data corresponding to the incremental data. Therefore, the correspondence between the data acquisition method and the incremental data can be improved, and the zipper table can be calculated offline in real time, which can save network resources and disk resources, and improve the accuracy and convenience of incremental data acquisition.
[0109] Figure 4 is a block diagram of a data storage device based on a zipper table according to an exemplary embodiment. Figure 4 , the zipper table-based data storage device 400 comprises:
[0110] The data acquisition unit 401 is used to acquire historical zipper data through the historical zipper table;
[0111] The data acquisition unit 401 is further used to acquire incremental data corresponding to historical zipper data, and use the data lake to acquire mirror data corresponding to the incremental data;
[0112] The zipper table updating unit 402 is used to update the historical zipper data according to the comparison result between the historical zipper data and the mirror data by using the hudi component corresponding to the data lake to obtain an updated zipper table.
[0113] According to some embodiments, the data acquisition unit 401 is used to acquire the incremental data corresponding to the historical zipper data, and is specifically used to:
[0114] The incremental data corresponding to the historical zipper data is acquired according to the incremental data acquisition method corresponding to the historical zipper data.
[0115] According to some embodiments, the data acquisition unit 401 is used to acquire the incremental data corresponding to the data set according to the incremental data acquisition method corresponding to the historical zipper data, specifically to:
[0116] When the incremental data corresponding to the historical zipper data is acquired in a JDBC manner, the incremental data corresponding to the historical zipper data is acquired in a micro-batch form;
[0117] or
[0118] Through the Spark data processing component or the Flink data processing component, log data is used to obtain incremental data corresponding to the historical zipper data, and the incremental data is written into the data lake, wherein the incremental data is in the form of a stream, and the data storage format of the incremental data includes at least one of parquet and orc.
[0119] According to some embodiments, the zipper table updating unit 402 is used to update the historical zipper data using the hudi component corresponding to the data lake according to the comparison result of the historical zipper data and the mirror data, and obtain the updated zipper table, specifically for:
[0120] Obtaining data types in a comparison result between the historical zipper data and the mirror data;
[0121] The historical zipper data is updated by adopting an update method corresponding to the data type and a hudi component corresponding to the data lake to obtain an updated zipper table.
[0122] According to some embodiments, the zipper table updating unit 402 is used to update the historical zipper data to obtain an updated zipper table, specifically for:
[0123] Acquire update information of the historical zipper table, wherein the update information includes update time and update method of the historical zipper table;
[0124] The historical zipper data is updated using the update information to obtain an updated zipper table.
[0125] According to some embodiments, the zipper table updating unit 402 is used to update the historical zipper data to obtain an updated zipper table, specifically for:
[0126] Obtaining application scenarios corresponding to the historical zipper table;
[0127] A target read-write mode corresponding to the application scenario is obtained, and the read-write mode of the history zipper table is set to the target read-write mode.
[0128] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0129] In some or related embodiments, a data acquisition unit is used to acquire historical zipper data through a historical zipper table; the data acquisition unit is also used to acquire incremental data corresponding to the historical zipper data, and use a data lake to acquire mirror data corresponding to the incremental data; a zipper table update unit is used to update the historical zipper data using a hudi component corresponding to the data lake according to a comparison result between the historical zipper data and the mirror data, to obtain an updated zipper table. Therefore, the update of the zipper table can be implemented by using a combination of hadoop and hudi components, which can increase the capacity of the data, and the use of hudi components can support real-time processing of data, reduce the occupation of resources by data accumulation, and the hudi component has efficient incremental data processing capabilities, which can reduce resource consumption, and can be compatible with a variety of data processing components, which can improve data processing efficiency and convenience.
[0130] Figure 5 is a block diagram of a network device 500 provided in an embodiment of the present disclosure. For example, the network device 500 may be provided as a network side device. Figure 5 The network device 500 includes a processing component 522, which further includes at least one processor, and a memory resource represented by a memory 532 for storing instructions executable by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform any method of the aforementioned method applied to the network side device.
[0131] The network device 500 may also include a power supply component 527 configured to perform power management of the network device 500, a wired or wireless network interface 550 configured to connect the network device 500 to a network, and an input / output (I / O) interface 558. The network device 500 may operate based on an operating system stored in the memory 532, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM or the like.
[0132] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0133] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable zipper-list-based data storage device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0134] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0136] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0137] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0139] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A data storage method based on a zipper table, characterized in that: include: Obtain historical zipper data through the historical zipper table; Obtain incremental data corresponding to historical zipper data, and use the data lake to obtain mirror data corresponding to the incremental data; According to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table.
2. The method according to claim 1, characterized in that The step of obtaining incremental data corresponding to historical zipper data includes: The incremental data corresponding to the historical zipper data is acquired according to the incremental data acquisition method corresponding to the historical zipper data.
3. The method according to claim 2, characterized in that The acquiring of the incremental data corresponding to the data set according to the incremental data acquiring method corresponding to the historical zipper data includes: When the incremental data corresponding to the historical zipper data is acquired in a JDBC manner, the incremental data corresponding to the historical zipper data is acquired in a micro-batch form; or Through the Spark data processing component or the Flink data processing component, log data is used to obtain incremental data corresponding to the historical zipper data, and the incremental data is written into the data lake, wherein the incremental data is in the form of a stream, and the data storage format of the incremental data includes at least one of parquet and orc.
4. The method according to claim 1, characterized in that: According to the comparison result between the historical zipper data and the mirror data, the historical zipper data is updated using the hudi component corresponding to the data lake to obtain an updated zipper table, including: Obtaining data types in a comparison result between the historical zipper data and the mirror data; The historical zipper data is updated by adopting an update method corresponding to the data type and a hudi component corresponding to the data lake to obtain an updated zipper table.
5. The method according to claim 1, characterized in that The updating of the historical zipper data to obtain an updated zipper table includes: Acquire update information of the historical zipper table, wherein the update information includes update time and update method of the historical zipper table; The historical zipper data is updated using the update information to obtain an updated zipper table.
6. The method according to claim 1, characterized in that The updating of the historical zipper data to obtain an updated zipper table includes: Obtaining application scenarios corresponding to the historical zipper table; A target read-write mode corresponding to the application scenario is obtained, and the read-write mode of the history zipper table is set to the target read-write mode.
7. A data storage device based on a zipper table, characterized in that: include: A data acquisition unit, used for acquiring historical zipper data through a historical zipper table; The data acquisition unit is further used to acquire incremental data corresponding to the historical zipper data, and use the data lake to acquire mirror data corresponding to the incremental data; The zipper table updating unit is used to update the historical zipper data according to the comparison result between the historical zipper data and the mirror data by using the hudi component corresponding to the data lake to obtain an updated zipper table.
8. A network device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data storage method based on a zipper table as described in any one of claims 1 to 6.
9. A storage medium, when instructions in the storage medium are executed by a processor of a network device, enables the network device to execute the zipper list-based data storage method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the zipper table-based data storage method according to any one of claims 1 to 6.