Method and device for quickly generating zipper table
By building a zipper cache and updating data using primary key matching and difference comparison, the existing zipper list generation methods are solved, and the effect of quickly generating zipper lists is achieved, which is suitable for large data processing.
Patent Information
- Application Number
- CN202411673517.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-25
AI Technical Summary
The existing zipper list generation method is inefficient when processing large data volumes, resulting in high time cost and requires a large amount of memory space, making it difficult to deal with excessive data volumes.
Data is temporarily stored by building a zipper cache, and the zipper cache data is updated through primary key matching and differential comparison when reading the data, reducing the reprocessing of all data, focusing only on the changing data parts. Use zipper cache to process data per day, and batch zipper the read data of the previous day to reduce frequent data operation overhead.
It greatly improves the generation efficiency of zipper lists, reduces memory usage, can quickly generate zipper lists, adapt to large data processing, and reduces the system's time and memory costs.
Smart Images
Figure CN119513102A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method and device for quickly generating a zipper table. Background Art
[0002] In the field of data processing, zipper tables are used as a data storage method for processing slowly changing dimensions. By recording the historical changes of data, just like a zipper, you can open it to see the data status at each stage. In business systems, the data in dimension tables is not static. For example, customer address, contact information and other information may change over time. Zipper tables can effectively record these changes and reflect the effective interval of each data version by recording the effective time and expiration time of the data. However, during the generation process, the existing zipper table generation method needs to compare the source data and the target data (if it is an incremental update) or the previous and next versions of the data (in the case of a full update) to determine the changes in the data. This means that each row of data needs to query the database to determine whether the start time and end time of the data exist within the time interval, resulting in the problem of inefficiency and high time cost in the generation method of zipper tables for large amounts of data.
[0003] In addition, for the task of full update, all the data needs to be loaded into memory or disk before generating the zipper table, which will consume a lot of memory space. Once the amount of data is too large, it will be impossible to process it. Therefore, there is an urgent need for a method that can quickly generate a zipper table without taking up too much memory space. Summary of the invention
[0004] The embodiment of the present application provides a method and device for quickly generating a zipper table. By constructing a zipper cache to store the full data of the first day, and using the zipper cache to process the subsequently read data on a daily basis, memory usage is greatly reduced and the generation efficiency of the zipper table is improved.
[0005] In a first aspect, an embodiment of the present application provides a method for quickly generating a zipper list, the method comprising:
[0006] Selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in a day-by-day order from early to late;
[0007] Construct a zipper cache, read data sequentially in the blocking queue, monitor the date information of each data during the reading process, and when the next day's data is read, perform zipper processing on the data read on the previous day in batches. The zipper processing method is:
[0008] If the zipper cache is empty, the read data is input into the zipper cache to obtain the zipper cache data; if the zipper cache is not empty, the primary key of each piece of data in the read data is matched with the zipper cache data to obtain a difference result; based on the difference result, a first data set and a second data set are obtained, wherein the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data; the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data;
[0009] Build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, transfer the latest zipper cache data to the zipper database, and generate a zipper table based on the zipper database.
[0010] In a second aspect, an embodiment of the present application provides a zipper table fast generation device, comprising:
[0011] An output module, used for selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in a day-by-day order from early to late;
[0012] A construction module is used to construct a zipper cache, sequentially read data in the blocking queue, monitor the date information of each data during the reading process, and when the data of the next day is read, perform zipper processing on the data read on the previous day in batches, wherein the zipper processing method is as follows: if the zipper cache is empty, input the read data into the zipper cache to obtain zipper cache data; if the zipper cache is not empty, match the zipper cache data with the primary key of each data in the read data to obtain a difference result, and obtain a first data set and a second data set based on the difference result, wherein the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data, and the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data;
[0013] A generation module is used to build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, the latest zipper cache data is transferred to the zipper database, and a zipper table is generated based on the zipper database.
[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a method for quickly generating a zipper list.
[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for quickly generating a zipper list.
[0016] The main contributions and innovations of the present invention are as follows:
[0017] The embodiment of the present application temporarily stores data by constructing a zipper cache. In the process of reading data, the zipper cache data is updated through primary key matching and difference comparison. This method avoids reprocessing all data and only focuses on the changed data part, which greatly improves the data processing efficiency. When reading the next day's data, this solution batches the read data of the previous day for zipper processing. Compared with processing data one by one, batch processing can reduce frequent data operation overhead, such as database connection and query times, thereby improving the overall data processing speed. When matching data differences, the embodiment of the present application uses each piece of data in the read data to match the corresponding data with the same primary key in the zipper cache, which can accurately find the changed data, so that the update of the zipper data is more in line with the actual business logic.
[0018] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 is a flow chart of a method for quickly generating a zipper table according to an embodiment of the present application;
[0021] Figure 2 is a structural block diagram of a zipper table fast generation device according to an embodiment of the present application;
[0022] Figure 3 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0024] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0025] Embodiment 1
[0026] The embodiment of the present application provides a method and device for quickly generating a zipper table, which greatly reduces memory usage and improves the generation efficiency of the zipper table by building a zipper cache to store the full data of the first day and using the zipper cache to process the subsequent read data on a daily basis. Specifically, refer to Figure 1 , the method comprises: selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in a day-by-day order from early to late;
[0027] Construct a zipper cache, read data sequentially in the blocking queue, monitor the date information of each data during the reading process, and when the next day's data is read, perform zipper processing on the data read on the previous day in batches. The zipper processing method is:
[0028] If the zipper cache is empty, the read data is input into the zipper cache to obtain the zipper cache data; if the zipper cache is not empty, the primary key of each piece of data in the read data is matched with the zipper cache data to obtain a difference result; based on the difference result, a first data set and a second data set are obtained, wherein the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data; the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data;
[0029] Build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, transfer the latest zipper cache data to the zipper database, and generate a zipper table based on the zipper database.
[0030] In some specific embodiments, the source database stores full data, including a large amount of duplicate data. For example, the table on the first day stores Zhang San's data and Li Si's data. When Zhang San's data is changed on the second day, the table on the second day will store the changed Zhang San's data and Li Si's data, resulting in duplicate storage of Li Si's data, wasting storage space.
[0031] In this solution, a reading thread is loaded, and the reading thread reads the data to be processed in the source database through SQL streaming and outputs it to the blocking queue.
[0032] Specifically, since the data in the source database is sorted from early to late in units of days, the reading order of the reading thread is also read from early to late in units of days. Similarly, the order in the blocking queue is also sorted from early to late in units of days.
[0033] In some specific embodiments, during the process of reading the blocking queue data, if it is monitored that the reading date changes, that is, the data of the next day is read, it means that the data of the previous day has been read, and all the read data of the previous day is processed at this time.
[0034] Specifically, this solution ensures daily processing of data by monitoring dates, greatly reducing memory usage and speeding up the generation time of zipper tables.
[0035] During the zipper processing of the read data, if it is detected that the zipper cache is empty, it means that the read data is the first day's data in the data to be processed, and the read data is input into the zipper cache.
[0036] Specifically, since the zipper table is a data table that records data changes, when the read data is the first day's data, the read data is the latest data relative to the first day. For example, the start time of the read data is November 13, 2024, and the end time is December 31, 9999, which means that the read data is data created on November 13, 2024.
[0037] In this solution, in the step of "matching the primary key of each piece of data in the read data with the zipper cache data to obtain a difference result", each piece of data in the read data is matched with the corresponding data with the same primary key in the zipper cache to obtain a difference result, and the difference result represents two corresponding data with the same primary key but different start time and / or end time.
[0038] Specifically, since the data in the blocking queue is full data, the read data and the data in the zipper cache will contain a large amount of data with the same start time and end time, that is, these data have not changed, so the cache hit method is used to find out the data that has changed.
[0039] Furthermore, the data in the first data set and the data in the second data set are corresponding data. For example, the data in the first data set is a data in the zipper cache with a start time of November 13, 2024 and an end time of December 31, 9999. Then, there will also be a data in the second data set with a start time of November 14, 2024 and an end time of December 31, 9999. After replacing the data in the first data set with the data in the second data set, it can be ensured that the data in the zipper cache is always the latest data. When all subsequent read data are judged, it can be guaranteed that the data changes in the read data can be accurately found to construct a zipper table.
[0040] In some embodiments, in the step of "using the data in the second data set to replace the corresponding data in the first data set to obtain new zipper cache data", if the start time of the data in the second data set is equal to the start time of the corresponding data in the first data set, then the end time of the data in the second data set is set to the end time of the corresponding data in the first data set.
[0041] In some cases, historical data may contain certain errors. When errors are found in historical data, it must be refreshed. That is, in order to ensure the accuracy of the zipper cache data, even if the start time of the data in the second data set is the same as the start time of the corresponding data in the first data set, it must be replaced.
[0042] In some embodiments, in the step of "using the data in the second data set to replace the corresponding data in the first data set to obtain new zipper cache data", if the start time of the data in the second data set is later than the start time of the corresponding data in the first data set, the end time of the data in the first data set is set to the start time of the corresponding data in the second data set.
[0043] Specifically, if the start time of the data in the second data set is later than the start time of the corresponding data in the first data set, it means that the data in the second data set is the latest data, so the end time of the data in the first data set is set to the start time of the corresponding data in the second data set, which is used to indicate that the end time of the data in the first data set is the start time of the corresponding data in the second data set, thereby recording the data changes.
[0044] Furthermore, by setting the end time of the data in the first data set to the start time of the corresponding data in the second data set and then storing it in the zipper database, it can be ensured that all the changed data are stored in the zipper database. For example, there is a data in the first data set with a start time of November 13, 2024 and an end time of December 31, 9999. When the data is replaced with a data with a start time of November 14, 2024, the end time of the data in the first data set is replaced with November 14, 2024. When querying the changed data subsequently, you only need to query the data with an end time of November 14, 2024 to get all the data changed on November 14, 2024.
[0045] In some embodiments, if the start time of the data in the second data set is less than the start time of the corresponding data in the first data set, the corresponding data in the first data set is transferred to the zipper database.
[0046] In some specific embodiments, data in the read data that cannot be matched with the same primary key in the zipper cache is used to form a third data set. When the amount of data in the third data set is greater than a set upper limit, a fourth data set is obtained by matching the primary key of each data in the third data set in the source database, and the fourth data set is written to the zipper cache, and then the third data set is zippered.
[0047] Furthermore, in the step of "matching in the source database based on the primary key of each data in the third data set", the results of traversing the source database for the data with the same primary key as each data in the third data set but with a start time earlier than the corresponding data form a fourth data set.
[0048] Specifically, there may be a large amount of data in the full table that has expired, but these data may have been restarted in the read data, such as canceling an account and then reissuing a card in banking business. Therefore, if a large amount of data with the same primary key cannot be matched in the zipper cache, it means that a large amount of data has expired, so these data need to be matched in the source database, so that the historical status of these data can be accurately known to build the zipper table.
[0049] In some specific embodiments, if there is still unprocessed data after the third data set is zipped, the source database is traversed based on the primary key of each data in the unprocessed data to obtain a result whose start time is earlier than or equal to the start time of the corresponding data and whose end time is later than the start time of the corresponding data as the fifth data set, and the fifth data set is written into the zipper cache, and then the unprocessed data is zipped.
[0050] For example, if the data being processed now has a start time of October 1, and the primary key field id=1, and the data with the primary key field id=1 in the database, a start time earlier than or equal to October 1, and an end time later than October 1, then the end time of the data queried in the source database will be set to October 1, and the end time of the current data will be set to the end time of the data queried in the source database
[0051] In some embodiments, if there is still unprocessed data, and the start time of the latest data in the source database is later than the unprocessed data, the end time of the unprocessed data is set to the start time of the latest data in the source database; if the start time of the latest data in the source database is not later than the unprocessed data, the end time of the unprocessed data is set to the maximum end time, and the unprocessed data is written to the zipper cache.
[0052] Embodiment 2
[0053] Based on the same idea, refer to Figure 2 , the present application also proposes a zipper table rapid generation device, comprising:
[0054] An output module, used for selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in a day-by-day order from early to late;
[0055] A construction module is used to construct a zipper cache, sequentially read data in the blocking queue, monitor the date information of each data during the reading process, and when the data of the next day is read, perform zipper processing on the data read on the previous day in batches, wherein the zipper processing method is as follows: if the zipper cache is empty, input the read data into the zipper cache to obtain zipper cache data; if the zipper cache is not empty, match the zipper cache data with the primary key of each data in the read data to obtain a difference result, and obtain a first data set and a second data set based on the difference result, wherein the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data, and the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data;
[0056] A generation module is used to build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, the latest zipper cache data is transferred to the zipper database, and a zipper table is generated based on the zipper database.
[0057] Embodiment 3
[0058] This embodiment also provides an electronic device, referring to Figure 3 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.
[0059] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0060] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 404 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (Static Random-Access Memory, abbreviated as SRAM) or a dynamic random access memory (Dynamic Random Access Memory, abbreviated as DRAM), wherein the DRAM can be a fast page mode dynamic random access memory 404 (Fast Page Mode Dynamic Random Access Memory, abbreviated as FPMDRAM), an extended data output dynamic random access memory (Extended Date Out Dynamic Random Access Memory, abbreviated as EDODRAM), a synchronous dynamic random access memory (Synchronous Dynamic Random-Access Memory, abbreviated as SDRAM), etc.
[0061] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .
[0062] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any one of the methods for quickly generating a zipper list in the above embodiments.
[0063] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .
[0064] The transmission device 406 can be used to receive or send data via a network. The specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0065] The input / output device 408 is used to input or output information. In this embodiment, the input information may be the full amount of data in the source database, and the output information may be the generation result of the zipper table.
[0066] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program:
[0067] Selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in a day-by-day order from early to late;
[0068] Construct a zipper cache, read data sequentially in the blocking queue, monitor the date information of each data during the reading process, and when the next day's data is read, perform zipper processing on the data read on the previous day in batches. The zipper processing method is:
[0069] If the zipper cache is empty, the read data is input into the zipper cache to obtain the zipper cache data; if the zipper cache is not empty, the primary key of each piece of data in the read data is matched with the zipper cache data to obtain a difference result; based on the difference result, a first data set and a second data set are obtained, wherein the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data; the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data;
[0070] Build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, transfer the latest zipper cache data to the zipper database, and generate a zipper table based on the zipper database.
[0071] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.
[0072] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.
[0073] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 3Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.
[0074] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0075] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for quickly generating a zipper table, characterized in that: The following steps are involved: Selecting data to be processed in the source database, and outputting the data to be processed to a blocking queue in order from early to late on a daily basis; Construct a zipper cache, read data sequentially in the blocking queue, monitor the date information of each data during the reading process, and when reading the next day's data, perform zipper processing on the data read on the previous day in batches. The zipper processing method is: If the zipper cache is empty, the read data is input into the zipper cache to obtain the zipper cache data; if the zipper cache is not empty, a difference result is obtained based on the primary key of each data in the read data and the zipper cache data; based on the difference result, a first data set and a second data set are obtained, where the first data set is data in the zipper cache data that differs from the read data, and the second data set is data in the read data that differs from the zipper cache data; the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data; Build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set and then transfer it to the zipper database. After processing all the data in the blocking queue, transfer the latest zipper cache data to the zipper database, and generate a zipper table based on the zipper database.
2. A method for quickly generating a zipper table according to claim 1, characterized in that: A reading thread is loaded, and the reading thread reads the data to be processed in the source database through SQL streaming and outputs it to the blocking queue.
3. The method for quickly generating a zipper table according to claim 1, wherein: In the step of "matching the primary key of each piece of data in the read data with the zipper cache data to obtain a difference result", each piece of data in the read data is matched with the corresponding data with the same primary key in the zipper cache to obtain a difference result, and the difference result represents two corresponding data with the same primary key but different start time and / or end time.
4. A method for quickly generating a zipper table according to claim 1, characterized in that: In the step of "using the data in the second data set to replace the corresponding data in the first data set to obtain new zipper cache data", if the start time of the data in the second data set is later than the start time of the corresponding data in the first data set, the end time of the data in the first data set is set to the start time of the corresponding data in the second data set.
5. The method for quickly generating a zipper table according to claim 1, wherein: The data in the read data that cannot be matched with the same primary key in the zipper cache are combined into a third data set. When the amount of data in the third data set is greater than the set upper limit, a fourth data set is obtained by matching the primary key of each data in the third data set in the source database, and the fourth data set is written to the zipper cache, and then the third data set is zipped.
6. A method for quickly generating a zipper table according to claim 5, characterized in that: In the step of "matching in the source database based on the primary key of each data item in the third data set", the results of traversing the source database for data items with the same primary key as each data item in the third data set but with a start time earlier than that of the corresponding data constitute a fourth data set.
7. A method for quickly generating a zipper table according to claim 1, characterized in that: If unprocessed data still exists after the zipper processing is performed on the third data set, the source database is traversed based on the primary key of each data item in the unprocessed data to obtain a result with a start time less than or equal to the corresponding data and an end time later than the corresponding data as a fifth data set, and the fifth data set is written into the zipper cache, and then the unprocessed data is zippered.
8. A zipper table rapid generation device, characterized in that: include: An output module is used to select data to be processed in the source database and output the data to be processed to a blocking queue in order from early to late on a daily basis; A construction module is used to construct a zipper cache, sequentially read data in the blocking queue, monitor the date information of each data during the reading process, and when the data of the next day is read, perform zipper processing on the data read on the previous day in batches, wherein the zipper processing method is as follows: if the zipper cache is empty, input the read data into the zipper cache to obtain zipper cache data; if the zipper cache is not empty, match the zipper cache data with the primary key of each data in the read data to obtain a difference result; based on the difference result, obtain a first data set and a second data set, wherein the first data set is data in the zipper cache data that differs from the read data; the second data set is data in the read data that differs from the zipper cache data; and the data in the second data set is used to replace the corresponding data in the first data set to obtain new zipper cache data; A generation module is used to build a zipper database, modify the end time of each data in the first data set to the start time of each corresponding data in the second data set, and then transfer it to the zipper database. After processing all the data in the blocking queue, the latest zipper cache data is transferred to the zipper database, and a zipper table is generated based on the zipper database.
9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the method for quickly generating a zipper list according to any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes a method for quickly generating a zipper list according to any one of claims 1 to 7.