Method, system, device, and medium for cold-hot data merge

Abstract

The application discloses a cold and hot data merging method, system, device and medium, the method comprising: based on setting the primary key extraction function, obtaining the hot key set and the cold key set from the hot library data list and the cold library data list; calculating the difference of the hot key set and the cold key set to obtain the cold key set; when the cold key set is in a non-empty value state, performing dimension coarse judgment test on the dimension value corresponding to the cold key set according to the set batch order; if there is an out-of-bound dimension value, continuing to perform dimension fine judgment test on the dimension value corresponding to the cold key set, and merging the hot library data list and the cold library data list after the dimension fine judgment test. The application performs two-level tests on the cold key set, first performs coarse judgment on the cold library unique data in batches to quickly judge whether all the cold library unique data is in the expected dimension range, then performs fine judgment on the out-of-bound data batch by batch, covers all effective data, avoids zero-dimension error data flow, and realizes the optimal balance of performance and accuracy.

Description

Technical Field

[0001] This invention relates to the field of distributed database storage and data processing, and in particular to a method, system, device, and medium for merging hot and cold data. Background Technology

[0002] In modern distributed database systems, separating hot and cold data is a common architectural design. The hot storage layer has high data freshness and can reflect the latest status in real time, but the data volume is incomplete, storing only recently active data, while archived data is migrated to the cold storage layer. The cold storage layer has low data freshness, and may contain expired or outdated data, but the data coverage is comprehensive and includes all historical data.

[0003] Hot storage is fresh but the data is incomplete, while cold storage is complete but the data is outdated. Searching only the hot storage will miss archived data, while searching only the cold storage will return expired data. Directly merging the hot and cold storage will introduce dirty data with dimension values ​​that do not match the current query scope. Performing dimension-by-dimensional validation on each piece of cold data in the cold storage will lead to extremely poor performance when the data volume is huge. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of low integrity and poor accuracy of cold and hot data merging in the prior art, and to provide a method, system, device and medium for cold and hot data merging.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution:

[0006] In a first aspect, the present invention provides a method for merging hot and cold data, the method comprising:

[0007] Based on the set primary key extraction function, the hot key set and cold key set are obtained from the hot storage data list and the cold storage data list respectively;

[0008] The difference between the hot key set and the cold key set is calculated to obtain the cold key set; the cold key set is used to characterize a set of data keys including several dimension values ​​where the cold storage data list exists and the hot storage data list does not exist.

[0009] When the cold key set is in a non-empty state, the dimension values ​​corresponding to the cold key set are subjected to coarse dimension judgment test according to the set batch order.

[0010] If no out-of-bounds dimension value exists, merge the hot storage data list and the cold storage data list;

[0011] If there are out-of-bounds dimension values, continue to perform dimension fine-grained checks on the dimension values ​​corresponding to the cold key set, and merge the hot storage data list and the cold storage data list that have passed the dimension fine-grained checks.

[0012] Preferably, merging the hot storage data list and the cold storage data list after passing the dimensionality-based precision test includes:

[0013] Retrieve the dimensional relationships corresponding to each primary key;

[0014] Perform dimensionality checks on all data items in the cold keyed set one by one.

[0015] When the dimensional relationship intersects with the set expected dimensional range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimensional precision test are then merged.

[0016] When the dimensional relationship does not intersect with the set expected dimensional range, the second data item is determined to be dirty data that is out of dimensional range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

[0017] Preferably, merging the hot storage data list and the cold storage data list after passing the dimensionality-based precision test further includes:

[0018] When the dimension association is null, the third data item is determined to be archived and has no dimension constraints, and is therefore saved.

[0019] Preferably, the method further includes:

[0020] A hot storage strategy is used to perform data synchronization queries on the hot storage layer and a cold storage strategy is used to perform data synchronization queries on the cold storage layer to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list;

[0021] And / or, when the second data item is determined to be dirty data that is out of dimension range and is removed, an alarm log containing the primary key, the dimension relationship, and the set expected dimension range is generated.

[0022] In a second aspect, the present invention provides a system for merging hot and cold data, the system comprising:

[0023] The acquisition module is used to retrieve the hot key set and cold key set from the hot storage data list and the cold storage data list respectively, based on the set primary key extraction function.

[0024] The calculation module is used to calculate the difference between the hot key set and the cold key set to obtain the cold key set; the cold key set is used to characterize the data key set including several dimension values ​​that exist in the cold storage data list and do not exist in the hot storage data list; when the cold key set is in a non-empty state, the first verification module is called.

[0025] The first inspection module is used to perform a coarse dimension check on the dimension values ​​corresponding to the cold key set according to a set batch order; if there are no out-of-bounds dimension values, the first merging module is called; if out-of-bounds dimension values ​​are stored, the second inspection module is called.

[0026] The first merging module is used to merge the hot storage data list and the cold storage data list;

[0027] The second inspection module is used to perform a dimension precision inspection on the dimension values ​​corresponding to the cold storage key subset if there are out-of-bounds dimension values, and to merge the hot storage data list and the cold storage data list that have passed the dimension precision inspection.

[0028] Preferably, the second inspection module includes an acquisition unit and an inspection unit;

[0029] The acquisition unit is used to acquire the dimensional association relationship corresponding to each primary key;

[0030] The inspection unit is used to perform dimensional inspection by traversing all data items of the cold key set one by one.

[0031] When the dimensional relationship intersects with the set expected dimensional range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimensional precision test are then merged.

[0032] When the dimensional relationship does not intersect with the set expected dimensional range, the second data item is determined to be dirty data that is out of dimensional range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

[0033] Preferably, the verification unit is further configured to determine that the third data item has been archived and is saved without dimensional constraints when the dimensional association relationship is null.

[0034] Preferably, the system further includes:

[0035] The query module is used to perform data synchronization queries on the hot storage layer using a hot storage strategy and on the cold storage layer using a cold storage strategy, in order to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list;

[0036] And / or,

[0037] The verification unit is also used to generate an alarm log containing the primary key, the dimension association relationship, and the set expected dimension range when it determines that the second data item belongs to dirty data that is out of dimension range and should be removed.

[0038] Thirdly, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method for merging hot and cold data as described above.

[0039] Fourthly, the present invention provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the method for merging hot and cold data as described above.

[0040] The positive and progressive effects of this invention are as follows: It provides a method, system, device, and medium for merging hot and cold data. The method includes: calculating the difference between the hot bond set and the cold bond set to obtain the cold bond set; when the cold bond set is non-nullable, performing a coarse dimension check on the dimension values ​​corresponding to the cold bond set according to a set batch order; if no out-of-bounds dimension values ​​exist, merging the hot data list and the cold data list; if out-of-bounds dimension values ​​exist, continuing to perform a fine dimension check on the dimension values ​​corresponding to the cold bond set, and merging the hot data list and the cold data list that has passed the fine dimension check. This invention, by performing a two-level check on the cold bond set, covers all valid data, avoids the inflow of zero-dimensional erroneous data, and ensures the integrity of data merging and the accuracy of verification results. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the first process of the method for merging hot and cold data in Embodiment 1 of the present invention.

[0042] Figure 2 This is a schematic diagram of the second process of the method for merging hot and cold data in Embodiment 1 of the present invention.

[0043] Figure 3 This is a schematic diagram of the first module of the system for merging hot and cold data in Embodiment 2 of the present invention.

[0044] Figure 4 This is a schematic diagram of the second module of the system for merging hot and cold data in Embodiment 2 of the present invention.

[0045] Figure 5 This is a schematic diagram of the hardware structure of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0046] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0047] Example 1

[0048] This embodiment provides a method for merging hot and cold data, such as Figure 1 As shown, the method includes:

[0049] S0. Use a hot storage strategy to perform data synchronization queries on the hot storage layer and a cold storage strategy to perform data synchronization queries on the cold storage layer to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list;

[0050] S1. Based on the set primary key extraction function, obtain the hot key set and cold key set from the hot storage data list and the cold storage data list respectively;

[0051] S2. Calculate the difference between the hot bond set and the cold bond set to obtain the cold bond set; the cold bond set is used to characterize the data bond set including several dimension values ​​where the cold storage data list exists but the hot storage data list does not.

[0052] S3. When the cold key set is in a non-null state, perform a coarse dimension check on the dimension values ​​corresponding to the cold key set according to the set batch order.

[0053] S41. If there are no out-of-bounds dimension values, merge the hot storage data list and the cold storage data list;

[0054] S42. If there are out-of-bounds dimension values, continue to perform dimension fine-grained checks on the dimension values ​​corresponding to the cold key set, and merge the hot storage data list and the cold storage data list that have passed the dimension fine-grained checks.

[0055] For step S0 above, the same query operation is performed on the hot storage layer and the cold storage layer respectively using the ReadStrategy.HOT_ONLY and ReadStrategy.COLD strategies to obtain the hot storage data list (hot) and the cold storage data list (cold). The hot storage data is fresh but incomplete, while the cold storage data is complete but may contain outdated or dimensionally changed data.

[0056] For steps S1-S3 above, the primary key extraction function keyExtractor is set to perform full lookup and difference calculation on both sides of the hot key data list hot and the cold key data list to extract the hot key set hotKeys and the cold key set coldKeys respectively. The difference between the hot key set hotKeys and the cold key set coldKeys is calculated as coldOnlyKeys=coldKeys−hotKeys to obtain the cold key set.

[0057] If the cold-only key set `coldOnlyKeys` is null, it means the hot storage data list already contains all the data from the cold storage data list. In this case, the hot storage data list is used as the base, and there is no need to import the cold storage data list; the hot and cold storage data lists can be directly and safely merged. If the cold-only key set `coldOnlyKeys` is not null, `coldOnlyKeys` is batched according to a set batch order for coarse dimension checks. For each batch of primary keys, the `roughCheck` function is called to query the dimension values ​​associated with the cold-only key set in batches. This method safely supplements the unique and non-repeating data in the cold storage when the hot storage data is incomplete, significantly improving the data integrity of the merged hot and cold data.

[0058] For steps S41-S42 above, if the coarse dimension check result does not show any out-of-bounds dimension values, it indicates that all cold storage-specific data is within the current query range and can be safely merged. The `mergeByKeyGeneric(hot, cold, keyExtractor)` method is called to merge hot and cold storage data by primary key, prioritizing the retention of hot storage data. If the coarse dimension check result shows out-of-bounds dimension values, it indicates that some cold storage-specific data is within the current query range, and a fine dimension check is initiated. During the fine check process, dirty data is filtered out and the cold storage data list is updated. This approach adopts a coarse-to-fine, on-demand degradation strategy. Most queries use normal scenarios where all dimensions of the cold storage-specific data are within the expected range, and a successful coarse check allows for safe merging without any fine check queries; the performance overhead is only one coarse batch query. A few queries use abnormal scenarios where fine check is triggered only when out-of-bounds dimensions are found during coarse check. The scope of fine check is much smaller than the cold storage key set. Even if the entire cold storage key set requires fine check, the performance is not equivalent to the traditional line-by-line verification scheme, and there is no performance degradation.

[0059] In one embodiment, such as Figure 2 As shown, step S42 includes:

[0060] S421. Obtain the dimension relationships corresponding to each primary key;

[0061] S422. Iterate through all data items in the cold keyed set and perform dimensionality checks;

[0062] S4231. When there is an intersection between the dimension relationship and the set expected dimension range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimension fine judgment test are merged.

[0063] S4232. When there is no intersection between the dimension relationship and the set expected dimension range, the second data item is determined to be dirty data that is out of the dimension range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

[0064] S4233. When the dimension relationship is null, determine that the third data item has been archived and has no dimension constraints and save it.

[0065] For steps S421-S422 above, fineCheck is called batch by batch on the coldOnlyKeys set to obtain the dimensional relationship Map for each primary key. <K,Set <d>In this approach, K represents the primary key or unique identifier type, and D represents the dimension value type. Each record in `coldOnlyKeys` is iterated over, and further dimensional checks are performed on all data items based on the dimension relationships corresponding to their primary keys. This two-level dimensional validation intercepts dirty data, preventing the introduction of dirty data during direct merging and significantly improving the accuracy of data merging.

[0066] Regarding steps S4231-S4233 above, when the dimension relationship intersects with the set expected dimension range, it indicates that the data dimension is still within the current query range, and the data item can be retained; when the dimension relationship does not intersect with the set expected dimension range, it indicates that the data dimension is completely outside the current query range, and the messy data item can be deleted before updating the cold storage data list; when the dimension relationship is an empty set, it indicates that the data item has not been archived in the hot storage, the data item is retained, and the cold storage data list is not updated. This method performs a two-level check on the keyed set of cold storage data. First, it performs a batch coarse judgment to quickly determine whether all the unique data of the cold storage is within the expected dimension range, and then performs a detailed judgment on each batch of out-of-bounds data, covering all valid data and achieving the optimal balance between performance and accuracy.

[0067] In one embodiment, step S4232 further includes:

[0068] When the second data item is determined to be dirty data due to out-of-bounds dimension and is removed, an alarm log is generated containing the primary key, dimension relationship, and the set expected dimension range.

[0069] Store the alarm logs related to dirty data that crosses the dimension in the cold keyed set. This will facilitate the identification of dirty data and the secondary verification of out-of-bounds dimension data with minimal data query cost in subsequent large-scale data scenarios, and significantly improve the accuracy and effectiveness of the cold and hot data merging results.

[0070] This embodiment provides a method for merging hot and cold data. The difference between the hot and cold key sets is calculated to obtain the cold key set. When the cold key set is non-empty, a coarse dimension check is performed on the corresponding dimension values ​​according to a set batch order. If no out-of-bounds dimension values ​​exist, the hot and cold storage data lists are merged. If out-of-bounds dimension values ​​exist, a fine dimension check is performed on the corresponding dimension values ​​of the cold key set, and the hot storage data list and the cold storage data list that has passed the fine dimension check are merged. By performing a two-level check on the cold key set, a coarse check is first performed in batches to quickly determine whether all unique cold storage data is within the expected dimension range. Then, a fine check is performed on each batch of out-of-bounds data, covering all valid data and preventing the inflow of zero-dimensional erroneous data, achieving an optimal balance between performance and accuracy.

[0071] Example 2

[0072] This embodiment provides a system for merging hot and cold data, such as Figure 3 As shown, the system includes: an acquisition module 210, a calculation module 220, a first verification module 230, a first merging module 240, and a second verification module 250.

[0073] The query module 200 is used to perform data synchronization queries on the hot storage layer using a hot storage strategy and on the cold storage layer using a cold storage strategy, in order to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list.

[0074] The acquisition module 210 is used to obtain the hot key set and cold key set from the hot storage data list and the cold storage data list respectively based on the set primary key extraction function;

[0075] The calculation module 220 is used to calculate the difference between the hot key set and the cold key set to obtain the cold key set; the cold key set is used to represent the data key set including several dimension values ​​where the cold storage data list exists and the hot storage data list does not exist; when the cold key set is in a non-empty state, the first verification module 230 is called.

[0076] The first inspection module 230 is used to perform a coarse dimension check on the dimension values ​​corresponding to the cold key set according to the set batch order; if there are no out-of-bounds dimension values, the first merging module 240 is called; if out-of-bounds dimension values ​​are stored, the second inspection module 250 is called.

[0077] The first merging module 240 is used to merge the hot storage data list and the cold storage data list;

[0078] The second inspection module 250 is used to perform a dimension fine-grained inspection on the dimension values ​​corresponding to the cold storage key subset if there are out-of-bounds dimension values, and to merge the hot storage data list and the cold storage data list that has passed the dimension fine-grained inspection.

[0079] The query module 200 performs the same query operation on the hot storage layer and the cold storage layer using the ReadStrategy.HOT_ONLY and ReadStrategy.COLD strategies respectively, to obtain the hot storage data list (hot) and the cold storage data list (cold). The hot storage data is fresh but incomplete, while the cold storage data is complete but may contain outdated or dimensionally changed data.

[0080] The acquisition module 210 performs a full lookup and difference calculation on both sides of the hot and cold key data lists by setting the primary key extraction function keyExtractor, and extracts the hot key set hotKeys and the cold key set coldKeys respectively. The calculation module 220 calculates the difference between the hot key set hotKeys and the cold key set coldKeys, coldOnlyKeys=coldKeys−hotKeys, to obtain the cold key set.

[0081] If the cold-only key set `coldOnlyKeys` is null, it means the hot storage data list already contains all the data from the cold storage data list. In this case, the hot storage data list is used as the baseline, and there is no need to import the cold storage data list. If the cold-only key set `coldOnlyKeys` is not null, the first verification module 230 performs a coarse dimension check on `coldOnlyKeys` in batches according to a set batch order. For each batch of primary keys, the `roughCheck` function is called to query the dimension values ​​associated with the cold-only key set in batches. This method safely supplements the unique and non-repeating data in the cold storage when the hot storage data is incomplete, significantly improving the data integrity of the merged hot and cold data.

[0082] If the coarse dimension check result shows no out-of-bounds dimension values, it indicates that all cold storage-specific data is within the current query range. The first merging module 240 safely merges the hot storage data list and the cold storage data list. It calls `mergeByKeyGeneric(hot, cold, keyExtractor)` to merge the hot storage data and cold storage data by primary key, prioritizing the retention of hot storage data. If the coarse dimension check result shows out-of-bounds dimension values, it indicates that some cold storage-specific data is within the current query range. The second check module 250 then performs a detailed dimension check, filtering out dirty data and updating the cold storage data list during the check. This approach employs a strategy of coarse-to-fine analysis and downgrading as needed. The vast majority of queries use conventional scenarios where all dimensions of the unique cold storage data fall within the expected range. Once the coarse assessment passes, the data can be safely merged without any fine-assessment queries. The performance overhead is only one coarse-assessment batch query. A few queries use abnormal scenarios where fine-assessment is triggered only when the coarse assessment finds out-of-bounds dimensions. The scope of fine-assessment is much smaller than the set of keyed cold storage data. Even if the entire set of keyed cold storage data requires fine-assessment, the performance is not equivalent to the traditional row-by-row verification scheme, and there is no performance degradation.

[0083] In one embodiment, such as Figure 4 As shown, the second inspection module 250 includes an acquisition unit 251 and an inspection unit 252;

[0084] Unit 251 is used to obtain the dimensional relationships corresponding to each primary key;

[0085] Inspection unit 252 is used to perform dimensionality checks on all data items of the cold key set one by one;

[0086] When there is an intersection between the dimensional relationship and the set expected dimensional range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimensional precision test are merged.

[0087] When the dimensional relationship does not intersect with the set expected dimensional range, the second data item is determined to be dirty data that is out of dimensional range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

[0088] When the dimension relationship is null, the third data item is determined to be archived and has no dimension constraints, and is therefore saved.

[0089] Unit 251 cold-only-key sets (coldOnlyKeys) are retrieved by batch-calling fineCheck to obtain the dimensional relationship Map for each primary key. <K,Set <d>In this approach, K represents the primary key or unique identifier type, and D represents the dimension value type. Verification unit 252 iterates through each record in `coldOnlyKeys`, performing further dimensional checks on all data items based on the dimension relationships corresponding to their primary keys. This two-level dimensional verification intercepts dirty data, preventing the introduction of dirty data during direct merging and significantly improving the accuracy of data merging.

[0090] When the correlation relationship determined by test unit 252 intersects with the set expected dimension range, it indicates that the data dimension is still within the current query range, and the data item can be retained. When the correlation relationship determined by test unit 252 does not intersect with the set expected dimension range, it indicates that the data dimension is completely outside the current query range, and the messy data item can be deleted before updating the cold storage data list. When the correlation relationship determined by test unit 252 is an empty set, it indicates that the data item has not been archived in the hot storage, and the data item is retained without updating the cold storage data list. This method performs a two-level test on the keyed set of cold storage data. First, it performs a coarse judgment in batches to quickly determine whether all the unique data of the cold storage is within the expected dimension range. Then, it performs a fine judgment on each batch of out-of-bounds data, covering all valid data and achieving the optimal balance between performance and accuracy.

[0091] In one embodiment, such as Figure 4 As shown, the system also includes:

[0092] The inspection unit 252 is also used to generate an alarm log containing the primary key, dimension relationship and set expected dimension range when the second data item is determined to be dirty data that is out of dimension range and is removed.

[0093] The verification unit 252 stores the alarm logs related to dirty data that intercepts out-of-bounds dimensions in the cold keyed set. This facilitates the identification of dirty data and the completion of secondary verification of out-of-bounds dimension data with minimal data query cost in subsequent large-scale data scenarios, significantly improving the accuracy and effectiveness of the cold and hot data merging results.

[0094] This embodiment provides a system for merging hot and cold data. A calculation module calculates the difference between the hot and cold key sets to obtain the cold key set. When the cold key set is non-empty, a first verification module performs a coarse dimension check on the dimension values ​​corresponding to the cold key set according to a set batch order. If no out-of-bounds dimension values ​​exist, the second verification module merges the hot and cold storage data lists. If out-of-bounds dimension values ​​exist, a fine dimension check is performed on the dimension values ​​corresponding to the cold key set, and the hot storage data list and the cold storage data list that has passed the fine dimension check are merged. By performing a two-level check on the cold key set, a first coarse check quickly determines whether all unique cold storage data is within the expected dimension range, and then a fine check is performed on each out-of-bounds batch, covering all valid data, avoiding the inflow of zero-dimensional erroneous data, and achieving an optimal balance between performance and accuracy.

[0095] Example 3

[0096] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for merging hot and cold data as described in Embodiment 1. Figure 5 The electronic device 60 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0097] Electronic device 60 may be in the form of a general-purpose computing device, such as a server device. Components of electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0098] Bus 63 includes a data bus, an address bus, and a control bus.

[0099] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0100] The memory 62 may also include a program / utility 625 having a set (at least one) of program modules 624, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0101] The processor 61 executes various functional applications and data processing by running computer programs stored in the memory 62, such as the method for merging hot and cold data in Embodiment 1 of the present invention.

[0102] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 65. Furthermore, the model-generated electronic device 630 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of the model-generated electronic device 60 via bus 63. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0103] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0104] Example 4

[0105] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for merging hot and cold data in Embodiment 1.

[0106] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0107] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the method for merging hot and cold data in Embodiment 1.

[0108] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0109] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.< / d> < / d>

Claims

1. A method for merging hot and cold data, characterized in that, The method includes: Based on the set primary key extraction function, the hot key set and cold key set are obtained from the hot storage data list and the cold storage data list respectively; The difference between the hot key set and the cold key set is calculated to obtain the cold key set; the cold key set is used to characterize a set of data keys including several dimension values ​​where the cold storage data list exists and the hot storage data list does not exist. When the cold key set is in a non-empty state, the dimension values ​​corresponding to the cold key set are subjected to coarse dimension judgment test according to the set batch order. If no out-of-bounds dimension value exists, merge the hot storage data list and the cold storage data list; If there are out-of-bounds dimension values, continue to perform dimension fine-grained checks on the dimension values ​​corresponding to the cold key set, and merge the hot storage data list and the cold storage data list that have passed the dimension fine-grained checks.

2. The method for merging hot and cold data as described in claim 1, characterized in that, The step of merging the hot storage data list and the cold storage data list after dimensionality-based precision testing includes: Retrieve the dimensional relationships corresponding to each primary key; Perform dimensionality checks on all data items in the cold keyed set one by one. When the dimensional relationship intersects with the set expected dimensional range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimensional precision test are then merged. When the dimensional relationship does not intersect with the set expected dimensional range, the second data item is determined to be dirty data that is out of dimensional range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

3. The method for merging hot and cold data as described in claim 2, characterized in that, The step of merging the hot storage data list and the cold storage data list after dimensionality-based precision testing also includes: When the dimension association is null, the third data item is determined to be archived and has no dimension constraints, and is therefore saved.

4. The method for merging hot and cold data as described in claim 2, characterized in that, The method further includes: A hot storage strategy is used to perform data synchronization queries on the hot storage layer and a cold storage strategy is used to perform data synchronization queries on the cold storage layer to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list; And / or, when the second data item is determined to be dirty data that is out of dimension range and is removed, an alarm log containing the primary key, the dimension relationship, and the set expected dimension range is generated.

5. A system for merging hot and cold data, characterized in that, The system includes: The acquisition module is used to retrieve the hot key set and cold key set from the hot storage data list and the cold storage data list respectively, based on the set primary key extraction function. The calculation module is used to calculate the difference between the hot key set and the cold key set to obtain the cold key set; the cold key set is used to characterize the data key set including several dimension values ​​that exist in the cold storage data list and do not exist in the hot storage data list; when the cold key set is in a non-empty state, the first verification module is called. The first inspection module is used to perform a coarse dimension check on the dimension values ​​corresponding to the cold key set according to a set batch order; if there are no out-of-bounds dimension values, the first merging module is called; if out-of-bounds dimension values ​​are stored, the second inspection module is called. The first merging module is used to merge the hot storage data list and the cold storage data list; The second inspection module is used to perform a dimension precision inspection on the dimension values ​​corresponding to the cold storage key subset if there are out-of-bounds dimension values, and to merge the hot storage data list and the cold storage data list that have passed the dimension precision inspection.

6. The system for merging hot and cold data as described in claim 5, characterized in that, The second inspection module includes an acquisition unit and an inspection unit; The acquisition unit is used to acquire the dimensional association relationship corresponding to each primary key; The inspection unit is used to perform dimensional inspection by traversing all data items of the cold key set one by one. When the dimensional relationship intersects with the set expected dimensional range, the first data item is determined to be within the query range and saved. The hot storage data list and the cold storage data list after passing the dimensional precision test are then merged. When the dimensional relationship does not intersect with the set expected dimensional range, the second data item is determined to be dirty data that is out of dimensional range and is removed. After updating the cold storage data list, it is merged with the hot storage data list.

7. The system for merging hot and cold data as described in claim 6, characterized in that, The verification unit is also used to determine that the third data item has been archived and has no dimensional constraints when the dimensional relationship is null.

8. The system for merging hot and cold data as described in claim 6, characterized in that, The system also includes: The query module is used to perform data synchronization queries on the hot storage layer using a hot storage strategy and on the cold storage layer using a cold storage strategy, in order to obtain the hot storage data list and the cold storage data list; when the query result of either strategy is empty, the hot storage data list or the cold storage data list is an empty list; And / or, the verification unit is further configured to generate an alarm log containing the primary key, the dimension association relationship, and the set expected dimension range when the second data item is determined to be dirty data that has exceeded the dimension limit and is therefore removed.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method of merging hot and cold data as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method of merging hot and cold data as described in any one of claims 1-4.

Images