A data dynamic storage method and device based on limited fixed sub-tables

Abstract

The present invention relates to the technical field of data storage and processing, and provides a data dynamic storage method and device based on limited fixed sub-tables, comprising: obtaining a plurality of predefined variables, including the number of sub-tables, the effective data storage period, the sub-table prefix name and the anchor date. Constructing a ring storage structure, and initializing the ring storage structure using the number of sub-tables and the sub-table prefix name. Determining data operation and data cleaning sub-table indexes according to the current data operation date, the number of sub-tables, the effective data storage period and the anchor date. Based on the data operation and data cleaning sub-table indexes, obtaining the current database operation sub-table name and the current database cleaning sub-table name from the initialized ring storage structure, so as to perform data operation and data cleaning on the corresponding sub-tables. The present invention can conveniently determine the data cleaning sub-table index according to the effective data storage period and the current data operation date, thereby centrally cleaning up expired data and realizing dynamic storage and cleaning of data.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and processing, and in particular to a data dynamic storage method and device based on limited fixed sub-tables. Background Art

[0002] When a large amount of data is stored in a database in a financial information system, it is usually necessary to set a certain preservation validity period. After the expiration, the invalid data must be cleaned up to maintain the storage space and the number of sub-tables within a reasonable range. While meeting such storage requirements, the database waterline must be lowered after the invalid data is cleaned up to support dynamic adjustment of the data preservation period, which brings great difficulties to the design of data sub-table storage. The existing technical solution supports the distribution of data in a fixed number of sub-tables in a certain dimension (modulus, hash, etc.). This solution only addresses the performance pressure problem when a large amount of data is stored in a single table, and the data table is sub-tabled to reduce the storage pressure of the single table. However, the data storage is scattered, and it is impossible to centrally clean up expired data. The data cleaning time complexity is O(N), which is inefficient and cannot effectively lower the waterline. This technical solution extends the fixed number (such as 31 days, 12 months) sub-table strategy, which also cannot adapt to the needs of dynamic adjustment of the validity period and lacks flexibility. Summary of the invention

[0003] The present invention provides a data dynamic storage method and device based on limited fixed sub-tables, which is used to solve the defects of the prior art that data storage cannot adapt to the needs of dynamic adjustment of validity period and lacks flexibility, and can realize dynamic management of data without frequent creation and deletion of sub-tables, thereby reducing the complexity of data cleaning and improving cleaning efficiency. The technical solution proposed by the present invention is as follows:

[0004] In a first aspect, the present invention provides a data dynamic storage method based on a limited fixed sub-table, comprising:

[0005] Acquire a plurality of predefined variables; wherein the plurality of variables include the number of sub-tables, the effective storage period of data, the prefix name of the sub-table and the anchor date;

[0006] Constructing a ring storage structure, and initializing the ring storage structure using the number of sub-tables and the sub-table prefix name to obtain an initialized ring storage structure;

[0007] Obtaining a current data operation date, and determining a data operation sub-table index and a data cleaning sub-table index according to the current data operation date, the number of sub-tables, the data valid storage period, and the anchor date;

[0008] Based on the data operation sub-table index and the data cleaning sub-table index, a current database operation sub-table name and a current database cleaning sub-table name are obtained from the initialized ring storage structure;

[0009] Data operation and data cleaning are performed on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

[0010] Optionally, the data operation sub-table index is determined in the following manner:

[0011] Determine the difference in days based on the current data operation date and the anchor date;

[0012] Determine the time span of the data stored in each sub-table in days according to the effective storage period of the data and the number of sub-tables;

[0013] Determine data grouping according to the day difference and the time span days;

[0014] Calculate the data operation sub-table index according to the data grouping and the number of sub-tables;

[0015] The sub-table corresponding to the sub-table name of the current database operation is used as the operation table, so that each sub-table stores data of a size equal to the number of days of the time span.

[0016] Optionally, the data cleaning sub-table index is determined in the following manner:

[0017] Determine the data archiving date based on the current data operation date and the effective data storage period;

[0018] The data archiving date is used to replace the current data operation date in the process of determining the data operation sub-table index, and the step of determining the data operation sub-table index is re-executed, and the execution result is used as the data cleaning sub-table index.

[0019] Optionally, the time span is determined by the following formula:

[0020]

[0021] Where s is the time span in days, expireDays is the effective storage period of data, and tableCount is the number of sub-tables. This is a rounding operation.

[0022] Optionally, the data operation sub-table index is determined by the following formula:

[0023] I=g%tableCount+1

[0024] Where I is the data operation table index, g is the data group, and tableCount is the number of tables.

[0025] Optionally, the method further comprises:

[0026] The operation of cleaning up the sub-tables is triggered once every time interval of the time span, so that the data stored in at least one sub-table are all expired and invalid data.

[0027] In a second aspect, the present invention further provides a data dynamic storage device based on a limited fixed sub-table, comprising the following modules:

[0028] A variable acquisition module, used to acquire a plurality of predefined variables; wherein the plurality of variables include the number of sub-tables, the valid storage period of data, the prefix name of the sub-table and the anchor date;

[0029] An initialization module is used to construct a ring storage structure, and initialize the ring storage structure using the number of sub-tables and the prefix name of the sub-table to obtain an initialized ring storage structure;

[0030] An index determination module, used to obtain a current data operation date, and determine a data operation sub-table index and a data cleaning sub-table index according to the current data operation date, the number of sub-tables, the data valid storage period and the anchor date;

[0031] A table name determination module, used for acquiring a current database operation sub-table name and a current database cleanup sub-table name from the initialized annular storage structure based on the data operation sub-table index and the data cleanup sub-table index;

[0032] The sub-table operation module is used to perform data operation and data cleaning on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

[0033] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the method for dynamic data storage based on a finite fixed partition table as described in the first aspect above is implemented.

[0034] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data dynamic storage method based on a finite fixed partition table as described in the first aspect above.

[0035] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data dynamic storage method based on a finite fixed partition table as described in the first aspect above.

[0036] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0037] The data dynamic storage method and device based on limited fixed sub-tables provided by the present invention can conveniently determine the data cleaning sub-table index according to the data effective storage period and the current data operation date through pre-defined multiple variables (such as the number of sub-tables, the effective storage period of data, etc.) and the construction of a ring storage structure, so as to centrally clean up expired data and realize dynamic storage and cleaning of data. Moreover, the introduction of the ring storage structure simplifies the complexity of data management. Through the circular storage method, dynamic management of data can be realized without frequently creating and deleting sub-tables, which reduces the complexity of data cleaning and improves cleaning efficiency. In addition, through the pre-defined multiple variables (such as the number of sub-tables, the effective storage period of data, etc.) and the dynamic calculation index method, it is possible to flexibly adapt to changes in the data validity period. Even if the data validity period changes, there is no need to redesign the storage structure or algorithm.

[0038] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is a flow chart of the data dynamic storage method based on limited fixed sub-tables provided by the present invention.

[0042] Figure 2 It is a logical structure diagram of the ring storage structure provided by the present invention.

[0043] Figure 3 It is a schematic diagram of data distribution after applying the method of the present invention.

[0044] Figure 4 It is a structural schematic diagram of a data dynamic storage device based on limited fixed sub-tables provided by the present invention.

[0045] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Combine the following Figure 1-Figure 4 The present invention describes a data dynamic storage method and device based on limited fixed sub-tables.

[0048] Reference Figure 1 As shown, the data dynamic storage method based on limited fixed sub-tables includes the following:

[0049] Step S110: Acquire multiple predefined variables.

[0050] In the data dynamic storage method based on finite fixed sub-tables, in order to achieve effective management and optimized storage of data, the present invention defines the following four key variables, which play a vital role in the calculation of obtaining the target sub-table of the database in the sub-table algorithm and the initialization process of the specific data structure.

[0051] Variable 1 is the number of sub-tables for storing data (hereinafter referred to as sub-table number), recorded as tableCount, and the type of this variable is integer. This variable represents the total number of sub-tables used to store data. In order to ensure the uniformity of sub-tables and the convenience of data management, tableCount>2 is required. tableCount supports dynamic modification to adapt to changes in data volume and adjustments in storage requirements.

[0052] Variable 2 is the validity period of the data (hereinafter referred to as the valid storage period of data), recorded as expireDays, and the type of this variable is integer. This variable indicates the valid number of days for data to be stored in the subtable. Data that exceeds this validity period will be considered expired data and needs to be cleaned up. expireDays also supports dynamic modification to flexibly respond to possible changes in the validity period of data.

[0053] Variable 3 is the prefix name of the sub-table, denoted as tablePrefix, such as table. The variable type is string. This variable represents the common prefix of all sub-table names. By adding different suffixes (such as numbers 1, 2, ..., N), specific sub-table names can be generated. For example, if the actual value of variable 1 (tableCount) is N, the names of the sub-tables are "table1", "table2", ..., "tableN" in turn. tablePrefix does not need to be modified frequently, so it can be regarded as a static variable in most cases.

[0054] Variable 4 is the anchor date, denoted as anchorDate, and the variable type is date. This variable represents a selected date that is before the current date. This date is used to determine in which sub-table the data should be stored. anchorDate can be set to the date when the function is put into use, or any other suitable date that is not too far away from the current date. Although anchorDate does not need to be modified frequently in most cases, it may need to be adjusted in some special cases (such as when the system is reset or redeployed).

[0055] For the two variables tableCount and expireDays that support dynamic modification, the system needs to provide corresponding interfaces or mechanisms so that they can be adjusted according to actual needs at runtime. This can be achieved through configuration files, database settings, or API calls. When dynamically modifying these variables, you need to ensure the stability of the system and the consistency of the data. For example, when modifying tableCount, you need to recalculate the sub-table indexes of all data and adjust the data storage and cleanup strategies accordingly. Similarly, when modifying expireDays, you also need to re-evaluate the expiration status of the data and perform necessary data cleanup operations. After defining the above variables, you can use them to implement dynamic storage and cleanup of data.

[0056] Step S120: construct a ring storage structure, and use the sub-table number tableCount and the sub-table prefix name tablePrefix to initialize the ring storage structure to obtain an initialized ring storage structure.

[0057] Create a string array of length tableCount, which will be used to store the table names of the sub-tables. Each element of the array is initialized to an empty string or to a specific placeholder as required. Use variables tableCount and tablePrefix to generate the table names of the sub-tables. Through a loop, fill the generated table names into the corresponding positions of the circular array in sequence. For example, if tablePrefix is ​​"table", the generated table names will be "table1", "table2", ..., "tableN". In order to easily access the table names in the circular array, an index mapping mechanism needs to be implemented. Index mapping can be implemented based on modulo operations to ensure that when the index exceeds the array length, it can loop back to the beginning of the array. Create a method (for example, getTableNameByIndex) that receives an index as a parameter and returns the corresponding table name. Inside the method, use the index mapping mechanism to calculate the actual array index and return the table name at that index.

[0058] After defining the TableRingBuffer class, you can use the variables tableCount and tablePrefix to initialize a tableRingBuffer instance. This process will form a logical structure, which is represented in memory as a circular array that contains the table names of all sub-tables. The logical structure formed is as follows Figure 2 As shown in the figure, this logical structure appears as a continuous array in memory, but through the index mapping mechanism, it can be regarded as a ring structure, in which each element can be accessed through a circular index. In this way, dynamic storage and cleaning of data can be achieved within a limited number of sub-tables.

[0059] Step S130, obtaining the current data operation date, and determining the data operation sub-table index and the data cleaning sub-table index according to the current data operation date, the number of sub-tables, the data valid storage period and the anchor date.

[0060] Get the current data operation date nowDate from the system clock or a specific time service. This is the key information for calculating the shard table index. Based on the current data operation date, the number of shard tables, the effective data storage period, and the anchor date, use certain calculation logic to determine which shard table the data should be stored in (data operation shard table index) and which shard table the data should be cleaned up (data cleanup shard table index). If the calculated index is a negative number, it needs to be converted to a positive number (by adding an integer multiple of the number of shard tables). If the number of shard tables changes (for example, a new shard table is added), the shard table indexes of all existing data need to be recalculated and the storage strategy adjusted.

[0061] Step S140: acquiring the current database operation sub-table name and the current database cleanup sub-table name from the initialized ring storage structure based on the data operation sub-table index and the data cleanup sub-table index.

[0062] The data operation sub-table index and data cleaning sub-table index calculated in step S130 are used as the index of the ring storage structure. The string values ​​corresponding to the index positions are read from the ring storage structure, and these values ​​are the names of the current database operation sub-table and the current database cleaning sub-table. These names will be used for subsequent data operations and data cleaning processes. If the index exceeds the range of the ring storage structure, an exception is thrown or an error log is recorded. If the ring storage structure is empty or not initialized correctly, subsequent operations cannot be performed, and an error needs to be returned or initialization operations need to be performed.

[0063] Step S150: perform data operation and data cleaning on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

[0064] According to the obtained current database operation sub-table name and current database cleanup sub-table name, perform data insertion, update, deletion and other operations on the corresponding sub-table, and clean up expired or invalid data.

[0065] Specifically, according to the name of the current database operation sub-table, connect to the corresponding database sub-table. Perform data insertion, update or deletion operations. Record operation logs for auditing and troubleshooting. According to the name of the current database cleanup sub-table, connect to the corresponding database sub-table. Perform data deletion operations to clean up expired or invalid data. You can optionally record cleanup logs to track the results of the cleanup process. If data operations and data cleanup need to ensure atomicity and consistency, they should be included in the same database transaction. Before the transaction is committed, check whether all operations are executed successfully. If an error occurs, roll back the transaction to maintain data consistency.

[0066] The data dynamic storage method based on limited fixed sub-tables provided by the present invention can conveniently determine the data cleaning sub-table index according to the data effective storage period and the current data operation date through pre-defined multiple variables (such as the number of sub-tables, the effective storage period of data, etc.) and the construction of a ring storage structure, so as to centrally clean up expired data and realize dynamic storage and cleaning of data. Moreover, the introduction of the ring storage structure simplifies the complexity of data management. Through the circular storage method, dynamic management of data can be realized without frequently creating and deleting sub-tables, thereby reducing the complexity of data cleaning and improving cleaning efficiency. In addition, through the pre-defined multiple variables (such as the number of sub-tables, the effective storage period of data, etc.) and the method of dynamically calculating the index, it is possible to flexibly adapt to the change of the data validity period. Even if the data validity period changes, there is no need to redesign the storage structure or algorithm.

[0067] The existing technology also dynamically expands the data table in a certain time dimension (by day, month, year, etc.). The data of the same time dimension are all concentrated in the same sub-table, and a specific algorithm needs to be implemented to clean up the data of a specific validity period. In addition, it is necessary to dynamically create and delete tables during system operation to meet data storage requirements rather than circular use. In a multi-data center scenario, data synchronization tools are required to support dynamic creation and deletion of tables that can be automatically synchronized between multiple data centers. This has certain limitations, and creating tables is a time-consuming operation with low efficiency. It is not suitable to create tables during system operation.

[0068] The data dynamic storage method based on limited fixed sub-tables provided by the present invention avoids the need for frequent creation and deletion of tables through a ring storage structure and a limited number of sub-tables. It uses a fixed number of sub-tables for cyclic storage, which simplifies the operation and reduces the complexity of the system. It is aimed at the defects of the difficulty and cost of development of data cleaning with a specific validity period that requires self-implementation of a specific algorithm. By means of pre-defined variables and dynamically calculated indexes, dynamic storage and cleaning of data can be conveniently realized without the need for complex algorithm design.

[0069] In an optional embodiment, the calculation logic of the data operation sub-table index may be: calculate the difference in days between the current data operation date nowDate and the anchor date anchorDate. Use the difference in days to perform a modulo operation on the number of sub-tables tableCount to obtain the data operation sub-table index. This index indicates in which sub-table the current data should be stored. The calculation logic of the data cleaning sub-table index may be: calculate a cleaning threshold date, which is the current date minus the valid storage period of the data. Calculate the difference in days between the cleaning threshold date and the anchor date. Use the difference in days to perform a modulo operation on the number of sub-tables to obtain the data cleaning sub-table index. This index indicates in which sub-table the data has expired and should be cleaned.

[0070] The data operation sub-table index and the data cleaning sub-table index are determined in the following manner:

[0071] S210: Determine the difference in days according to the current data operation date nowDate and the anchor date anchorDate, recorded as △d, that is, △d=nowDate-anchorDate. This difference indicates how many days have passed since the anchor date.

[0072] S220, determine the time span days s of the data stored in each sub-table according to the data valid storage period expireDays and the number of sub-tables tableCount. The time span days can be calculated by dividing the data valid storage period by the number of sub-tables (or adjusting appropriately according to actual needs). This value indicates how many days of data each sub-table will store.

[0073] S230, determine which data group the data should belong to based on the calculation result of S210, i.e., the difference in days △d, and the calculation result of S220, i.e., the number of days in the time span s. This can be achieved by dividing the difference in days by the number of days in the time span and rounding (or performing operations such as rounding or taking the remainder according to actual needs). The result indicates which time period the data should be allocated to. The result is rounded down and recorded as g, then .

[0074] S240, calculating the data operation sub-table index I according to the data group g and the sub-table number tableCount. This can be achieved by performing a modulo operation on the data group value and the sub-table number. The result is an integer between 0 and (sub-table number - 1), indicating in which sub-table the data should be stored.

[0075] S250, using the value I calculated in S240 as the index subscript, obtain the name of the current database operation sub-table through tableRingBuffer. Use this sub-table as the operation table for subsequent data insertion, update or deletion operations. Use the sub-table corresponding to the current database operation sub-table name as the operation table, so that each sub-table stores data of the size of the time span days.

[0076] S260. Determine the data archiving date according to the current data operation date and the data effective storage period.

[0077] First, get the current date from the system clock or a reliable time service, which will be the starting point for determining the data archiving date. The data valid storage period (expireDays) refers to the number of days that the data is considered valid from the date when the data is generated or a specific event occurs. For example, if the data valid storage period is 30 days, then the data is valid for the next 30 days from the date when the data is generated. In order to determine the data archiving date, it is necessary to calculate the data valid storage period (expireDays) from the current data operation date. This involves date subtraction, that is, subtracting the valid days from the current date, and the result is the data archiving date.

[0078] S270, replace the current data operation date snowDate in S210 with the data archiving date, re-execute steps S210-S240, and use the execution result of S240 as the data cleaning sub-table index. Use the data cleaning sub-table index as the index subscript, and obtain the previous sub-table corresponding to the I value through tableRingBuffer, which is the name of the table that can be cleaned.

[0079] By grouping data by time span and assigning them to different sub-tables, you can ensure that data is evenly distributed among the sub-tables. This helps avoid the problem of performance degradation in some sub-tables due to excessive data volume. Each sub-table only stores data within a specific time period, making data management and maintenance simpler and more intuitive. You can quickly locate the sub-table where the data to be queried or cleaned is located based on the time range. Since the data is stored in the sub-tables in chronological order, when performing a time range query, you can only access the relevant sub-tables, thereby improving query efficiency. This method is easy to expand and the number of sub-tables can be increased or decreased according to actual needs. When the amount of data increases, you only need to increase the number of sub-tables and recalculate the time span days and the sub-table index for data operations. You can regularly clean up expired data based on the effective storage period of the data without traversing the entire database to find and delete invalid data, making the data cleaning process simpler and more efficient.

[0080] In an optional embodiment, the time span in days is determined by the following formula:

[0081]

[0082] Where s is the time span in days, expireDays is the effective storage period of data, and tableCount is the number of sub-tables. This is a rounding operation.

[0083] By setting the time span in days appropriately, you can ensure that data is evenly distributed among multiple shards. This helps prevent a shard from becoming a hotspot due to excessive data volume, which can affect the overall performance of the database. Balanced data distribution means that query operations can be executed in parallel among multiple shards more efficiently, reducing query latency and improving system throughput. In addition, a reasonable time span in days ensures that the amount of data in each shard is neither too much nor too little, thus avoiding waste of storage resources. This helps reduce database operating costs and improve resource utilization.

[0084] The sharding strategy based on the time span days s can simplify the data cleaning process. By regularly cleaning up expired data (that is, data that exceeds the data effective storage period expireDays), you can ensure that only valid and latest data is stored in the database. The automated data cleaning process reduces the need for manual intervention and reduces maintenance costs. At the same time, because data cleaning is performed regularly, it is easier to predict and plan the storage needs of the database. When the amount of data increases, the storage capacity can be expanded by increasing the number of shards. Since the time span days s is inversely proportional to the number of shards tableCount, increasing the number of shards can automatically adjust the time span to adapt to the new amount of data.

[0085] In an optional embodiment, the ring storage structure (tableRingBuffer) of the present invention is used to store sub-tables, and each sub-table has a unique index subscript. Based on the ring storage structure, the present invention designs a sub-table algorithm, which uses a method based on modulo operation to calculate the index subscript of the target sub-table. The data operation sub-table index is determined by the following formula:

[0086] I=g%tableCount+1

[0087] Where I is the data operation table index, g is the data group, and tableCount is the number of tables.

[0088] Based on the calculation formula of the data operation sub-table index, the above steps S210-S240 are executed, and the current data operation date is mapped to an integer between 0 and tableCount-1 through a modulo operation (%), and the integer plus 1 is the index subscript of the target sub-table (i.e., the above operation table). In the above sub-table algorithm, no matter what the value of the current data operation date and the number of sub-tables tableCount is, the execution time of the modulo operation is fixed. Therefore, the time complexity of the algorithm is O(1). The time complexity O(1) means that the execution time of the algorithm is a constant, that is, it does not increase with the size of the input data. This means that no matter how many records there are in the database, or how long the time range is, the algorithm can calculate the index subscript of the target sub-table in a constant time. It is conducive to quickly locating the target sub-table in application scenarios that require efficient processing of large amounts of data.

[0089] In an optional embodiment, based on the original determination of the data operation sub-table index, an operation of regularly clearing the sub-table can be added. The method further includes:

[0090] S310: Trigger a sub-table cleanup operation every time interval of the time span, so that data stored in at least one sub-table are all expired and invalid data.

[0091] According to the number of days in the time span, set a cleanup cycle. This cycle should match the number of days in the time span to ensure that at least one sub-table can be cleaned up in each cycle. Whenever the cleanup cycle is reached, the system will automatically trigger an operation to clean up the sub-table. This can be achieved through a timer, event scheduler, or other mechanism. When triggering the cleanup operation, it is necessary to determine which sub-table should be cleaned up. This can be done by calculating the difference in days between the current date (i.e., the current data operation date mentioned above) and the anchor date, and determining it based on the number of days in the time span. After determining the sub-table to be cleaned up, perform the data cleanup operation. This includes deleting all data stored in the sub-table, or deleting part of the data according to business needs (for example, only deleting data that exceeds the valid storage period of the data). After the cleanup operation is completed, the relevant information in the ring storage structure needs to be updated to reflect the latest status of the sub-table.

[0092] The present invention realizes automatic management of data by regularly triggering the operation of cleaning up sub-tables. This reduces the need for manual intervention and improves the efficiency and accuracy of data management. The cost of data maintenance is reduced by automated cleaning operations. Regular cleaning of expired data can release storage resources, allowing the database to use storage space more efficiently. This helps to avoid the problems of data expansion and performance degradation and improve the overall performance of the database. The cleaning operation can ensure that only valid data is stored in the database, thereby maintaining data consistency. The method supports dynamic expansion and can increase or decrease the number of sub-tables according to business needs. When the amount of data increases, it is only necessary to increase the number of sub-tables and adjust the cleaning cycle without reallocating or migrating existing data.

[0093] Before data operation (addition, modification, search), the sub-table name obtained in S250 is used as the operation table, so that the data can be continuously distributed in each sub-table according to the date increase. Each sub-table stores the data in S230 with a time span of days s, thereby reducing the data volume of a single table and reducing the performance pressure of each sub-table.

[0094] Before data cleaning, use the table name obtained in S270 as the cleaning table. Cleaning is triggered every time span s calculated in S230. Through this algorithm, valid data storage data involves at most tableCount-1 sub-tables. From the realization that at least one sub-table is full of expired and invalid data, the expired and invalid data can be cleaned up with the table as the dimension through a more thorough data cleaning method (truncate) rather than deletion (delete), effectively lowering the database waterline, thereby solving the problem in the prior art that data storage is dispersed and cannot be cleaned up centrally to lower the waterline. And the cleaning device adapts to the validity period of data storage, solving the problem that the prior art cannot dynamically adjust after the validity period is initialized. For example, the following Figure 3 This is a common data distribution situation. In this case, all data in table2 is invalid data and the entire table can be cleaned up.

[0095] The present invention solves the performance problem and data cleaning problem of single-table storage in financial information systems by implementing efficient partitioning algorithms and data structures for scenarios with high requirements for large-scale data storage and cleaning based on databases. The present invention supports current mainstream databases (such as Mysql, Oracle, DB2, etc.), and the time complexity of calculating the partition table algorithm is O(1). It is transparent to the business application layer and does not require explicit calls from the business layer. Compared with fixed partition numbers such as 31 (days) and 12 (months), it supports dynamic adjustment of the number of partitions and storage validity period. Through good algorithm encapsulation, business personnel pay attention to the effective storage period of data, and technical personnel pay attention to the two parameters of the number of partitions, which realizes the decoupling of business needs and technical changes, and the solution is more flexible and adaptable to a wider range of scenarios. The time complexity of data cleaning is O(1), and the storage space waterline can be immediately reduced. In other current technical solutions, the data cleaning complexity is either O(N), or a specific cleaning algorithm needs to be implemented according to the specific number of partition tables and the data validity period to achieve the time complexity of O(1). It is impossible to dynamically adjust the storage validity period and use it out of the box. This solution is more flexible, efficient and convenient.

[0096] The present invention mainly aims at using a database to store a large amount of data, and it is necessary to clean up expired and invalid data to lower the database watermark. It provides a data partitioning storage and cleaning mechanism, and an improved partitioning algorithm based on consistent hashing solves the problems of scattered data storage, low efficiency in cleaning up expired data and inability to lower the watermark. At the same time, it avoids the problem of repeated creation and deletion of tables, and realizes the advantages of table reuse, dynamic adjustment of validity period, efficient partitioning storage, centralized cleaning of expired data, etc., which is more flexible and practical.

[0097] The following is a description of the data dynamic storage device based on finite fixed sub-tables provided by the present invention. The data dynamic storage device based on finite fixed sub-tables described below and the data dynamic storage method based on finite fixed sub-tables described above can be referenced to each other.

[0098] The data dynamic storage device based on limited fixed sub-table provided by the present invention refers to Figure 4 As shown, it includes the following modules:

[0099] The variable acquisition module 410 is used to acquire a plurality of predefined variables; wherein the plurality of variables include the number of sub-tables, the valid storage period of data, the prefix name of the sub-table and the anchor date;

[0100] Initialization module 420, used to construct a ring storage structure, and initialize the ring storage structure using the sub-table quantity and the sub-table prefix name to obtain an initialized ring storage structure;

[0101] An index determination module 430 is used to obtain a current data operation date, and determine a data operation sub-table index and a data cleaning sub-table index according to the current data operation date, the number of sub-tables, the data valid storage period, and the anchor date;

[0102] A table name determination module 440 is used to obtain a current database operation sub-table name and a current database cleanup sub-table name from the initialized ring storage structure based on the data operation sub-table index and the data cleanup sub-table index;

[0103] The sub-table operation module 450 is used to perform data operation and data cleaning on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

[0104] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the data dynamic storage method based on the finite fixed partition table.

[0105] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data dynamic storage method based on a finite fixed partition table provided by the above methods.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the data dynamic storage method based on a finite fixed partition table provided by the above methods.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data dynamic storage method based on a finite fixed sub-table, characterized in that: include: Acquire a plurality of predefined variables; wherein the plurality of variables include the number of sub-tables, the effective storage period of data, the prefix name of the sub-table and the anchor date; Constructing a ring storage structure, and initializing the ring storage structure using the number of sub-tables and the sub-table prefix name to obtain an initialized ring storage structure; Obtain the current data operation date, determine the difference in days according to the current data operation date and the anchor date; determine the time span days of the data stored in each sub-table according to the data valid storage period and the number of sub-tables; determine the data grouping according to the difference in days and the time span days; calculate the data operation sub-table index according to the data grouping and the number of sub-tables; wherein, use the sub-table corresponding to the current database operation sub-table name as the operation table, so that each sub-table stores data of a size of the time span days; determine the data archiving date according to the current data operation date and the data valid storage period; use the data archiving date to replace the current data operation date in the process of determining the data operation sub-table index, re-execute the step of determining the data operation sub-table index, and use the execution result as the data cleaning sub-table index; Based on the data operation sub-table index and the data cleaning sub-table index, a current database operation sub-table name and a current database cleaning sub-table name are obtained from the initialized ring storage structure; Data operation and data cleaning are performed on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

2. The data dynamic storage method based on limited fixed sub-tables according to claim 1 is characterized in that: The time span in days is determined by the following formula: ； Where s is the time span in days, expireDays is the effective storage period of data, and tableCount is the number of sub-tables. This is a rounding operation.

3. The data dynamic storage method based on limited fixed sub-tables according to claim 1 is characterized in that: The data operation table index is determined by the following formula: I=g%tableCount+1 Where I is the data operation table index, g is the data group, and tableCount is the number of tables.

4. The data dynamic storage method based on limited fixed sub-tables according to claim 1 is characterized in that: The method further comprises: The operation of cleaning up the sub-tables is triggered once every time interval of the time span, so that the data stored in at least one sub-table are all expired and invalid data.

5. A data dynamic storage device based on a finite fixed sub-table, characterized in that: include: A variable acquisition module, used to acquire a plurality of predefined variables; wherein the plurality of variables include the number of sub-tables, the valid storage period of data, the prefix name of the sub-table and the anchor date; An initialization module is used to construct a ring storage structure, and initialize the ring storage structure using the number of sub-tables and the prefix name of the sub-table to obtain an initialized ring storage structure; An index determination module is used to obtain the current data operation date, determine the difference in days according to the current data operation date and the anchor date; determine the time span days of the data stored in each sub-table according to the data effective storage period and the number of sub-tables; determine the data grouping according to the difference in days and the time span days; calculate the data operation sub-table index according to the data grouping and the number of sub-tables; wherein the sub-table corresponding to the current database operation sub-table name is used as the operation table, so that each sub-table stores data of a size of the time span days; determine the data archiving date according to the current data operation date and the data effective storage period; use the data archiving date to replace the current data operation date in the process of determining the data operation sub-table index, re-execute the step of determining the data operation sub-table index, and use the execution result as the data cleaning sub-table index; A table name determination module, used for acquiring a current database operation sub-table name and a current database cleanup sub-table name from the initialized annular storage structure based on the data operation sub-table index and the data cleanup sub-table index; The sub-table operation module is used to perform data operation and data cleaning on the corresponding sub-table according to the current database operation sub-table name and the current database cleaning sub-table name.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the data dynamic storage method based on a finite fixed sub-table as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data dynamic storage method based on a finite fixed partition table as claimed in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data dynamic storage method based on a finite fixed partition table as claimed in any one of claims 1 to 4 is implemented.

Images