A data archiving system based on data migration

Abstract

The application relates to the technical field of data storage and database management, in particular to a data archiving system based on data migration; the system comprises a database, a rule configuration, a data acquisition, a condition judgment, a data migration, a verification and a cleaning module; the system acquires data meeting time conditions, and determines a multi-dimensional archiving judgment basis; the core is to determine archiving conditions according to data creation time, record state and business type, and to perform consistency verification based on business identification or a hash value after migration; when the verification is consistent, the system safely cleans data from a production library; the application realizes a change from a single time condition to multi-dimensional comprehensive judgment, can accurately screen stable and completed data, and effectively avoids mismigration or misdeletion.

Description

Technical Field

[0001] This invention relates to the field of data storage and database management technology, specifically a data archiving system based on data migration. Background Technology

[0002] Existing production databases often use methods such as periodic deletion, full table migration, or time-based transfer when cleaning up historical business data; however, the above-mentioned existing technologies have the following drawbacks: Insufficient archiving criteria: Existing methods rely solely on a single time condition for filtering, failing to incorporate multi-dimensional constraints based on business type and record status. This leads to the direct and accidental migration or deletion of incomplete or unstable business data. Lack of control mechanisms during migration: Data migration lacks consistency verification between the production and archive databases, and there is a lack of independent control over migration batches, abnormal batches, and cleanup timing. This results in incomplete archiving writes, accidental deletion of source data, and difficulty in tracing anomalies. Furthermore, the deletion load on the source data interferes with online business operations. Lack of independent query channels for archived data: After data is archived, the lack of dedicated query interfaces and indexes for archived data means that the system still needs to return to the production database to perform queries when tracing historical data, failing to reduce the operational load on the online database. Summary of the Invention

[0003] The purpose of this invention is to provide a data archiving system based on data migration, addressing the following technical problems: existing production databases have shortcomings in historical data archiving determination, migration control mechanisms, and independent querying of archived data. This invention proposes a data archiving system based on multi-dimensional constraint archiving, relying on consistency checks to ensure migration security, and constructing an independent query channel to reduce online load. The objective of this invention can be achieved through the following technical solutions: The production database is used to store business data; the archiving rule configuration module is used to configure the preset archiving period, preset archiving type, archiveable status, and archived data structure; the data acquisition module is used to acquire business data that meets the preset archiving time conditions from the production database, the preset archiving time conditions including: the time interval between the current time and the creation time of the business data is greater than or equal to the preset archiving period; the condition judgment module is used to determine whether the business data meets the archiving conditions based on the creation time, record status, and business type of the business data, the archiving conditions including: the creation time of the business data reaches the preset archiving period, the record status is the archiveable status, and the business type belongs to the preset archiving type; The archive database is used to store business data migrated from the production database; the data migration module is used to write the business data into the archive database according to the archive data structure when the business data meets the archiving conditions; the data verification module is used to perform consistency verification on the corresponding business data in the production database and the archive database based on at least one of the business identifier and primary key, or based on the hash value calculated from the corresponding business data content; the data cleanup module is used to delete the business data from the production database after the data verification module confirms that the verification is consistent.

[0004] Preferably, the data acquisition module includes a log parsing unit and a data extraction unit. The log parsing unit is used to read the operation log of the production database, and the data extraction unit is used to extract business data that has reached the preset archiving period based on the data change time, data table identifier, and business record identifier recorded in the operation log.

[0005] Preferably, the data migration module includes a format conversion unit and a batch writing unit. The format conversion unit is used to convert the business data into a data storage format preset by the archive database, and the batch writing unit is used to write the converted business data into the archive database in batches according to a preset batch size.

[0006] Preferably, the data cleaning module includes a status marking unit and a timed deletion unit. The status marking unit is used to add a cleanup mark to the record status field of the production database for business data that has been successfully migrated and passed consistency verification. The timed deletion unit is used to delete the business data with the cleanup mark within a preset time range.

[0007] Preferably, it also includes a task monitoring module for monitoring the migration progress of the data migration module and generating a migration log containing migration batch identifiers, completion status, and failure information.

[0008] Preferably, it also includes an exception handling module, which is used to trigger a re-migration task for the corresponding abnormal batch and output alarm information when the task monitoring module detects that the migration task has timed out, data writing is abnormal, or consistency check is inconsistent.

[0009] Preferably, it also includes an archive query module, used to receive query requests from user terminals or external business systems, and retrieve archive data corresponding to the query request in the archive database.

[0010] Preferably, the archive query module includes an index building unit and a data retrieval unit. The index building unit is used to build a query index for the data in the archive database based on at least one of the business identifier, creation time, or primary key. The data retrieval unit is used to match and return the target archive data based on the query field in the query request and the query index.

[0011] Compared with the prior art, the present invention has the following beneficial effects: 1. This system uses a condition judgment module to perform multi-dimensional archiving judgment based on the creation time, record status, and business type of business data. Compared with existing technologies that rely solely on a single time condition, this method can accurately filter out data that has reached the preset archiving period, is archiveable, and belongs to the preset archiving type, thus avoiding the direct and accidental migration or deletion of incomplete or unstable business data. 2. This system performs consistency verification through the data verification module before the data cleanup module performs deletion. Combined with the status marking unit and the timed deletion unit, data that has been verified to be consistent is marked as to be cleaned and deleted within a preset time range. This mechanism prevents incomplete archiving and accidental deletion of source data, decouples migration and cleanup, and avoids cleanup actions interfering with online business. 3. This system provides an independent query channel for the archive database through the archive query module; it uses the index building unit to build query indexes based on business identifiers, creation time, or primary keys, and the data retrieval unit can directly match target archive data in the archive database; this design eliminates the need to trace historical data back to the production database, reducing the operating load of the production database. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the functional modules of a data archiving system based on data migration, provided in an embodiment of this disclosure. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0015] like Figure 1 As shown, a data archiving system based on data migration includes: The production database is used to store business data, which includes creation time, record status, and business type. The archiving rules configuration module is used to configure the preset archiving period, preset archiving type, archiveable status, and archived data structure; The data acquisition module is used to acquire business data that meets the preset archiving time conditions from the production database. The preset archiving time conditions include: the time interval between the current time and the creation time of the business data is greater than or equal to the preset archiving period. The condition judgment module is used to determine whether the business data meets the archiving conditions based on the creation time, record status, and business type of the business data. The archiving conditions include: the time interval between the current time and the creation time of the business data reaches the preset archiving period, the record status of the business data is archiveable, and the business type of the business data belongs to the preset archiving type. Archive databases are used to store business data migrated from production databases; The data migration module is used to write business data into the archive database according to the archive data structure when the business data meets the archiving conditions. The data verification module is used to verify the consistency of corresponding business data in the production database and the archive database based on at least one of the business identifier and primary key, or based on the hash value calculated from the corresponding business data content. The data cleanup module is used to delete business data from the production database after the data verification module confirms that the data is consistent.

[0016] The data acquisition module includes: a log parsing unit, used to read the operation logs generated by the production database; The data extraction unit is used to extract business data that has reached the preset archiving period based on the data change time, data table identifier, and business record identifier recorded in the operation log.

[0017] The data migration module includes: a format conversion unit, used to convert business data into a data storage format pre-defined by the archive database; and a batch writing unit, used to write the converted business data into the archive database in batches according to a preset batch size.

[0018] The system is deployed in a business environment that continuously generates large-scale transaction flows. The production database handles online transaction writes, while the archive database is used to store historical transaction data that has reached the retention period. Since online transactions and archive migration share storage resources, the archiving process is not triggered at a fixed time. Instead, it first converges the range of data to be migrated based on the archiving period, archiving type, archiveable status, and archive data structure pre-saved in the archiving rule configuration module, and then the subsequent modules complete the migration, verification, and cleanup. The archiving rule configuration module first saves the rule set corresponding to the business transaction. The archiving period is used to limit the time boundary for data to enter the archiving candidate set. The archiving type is used to limit the business category participating in archiving, such as specific business types such as payment transaction, refund transaction or transfer transaction. The archiveable status is used to exclude records that are still being processed, frozen or have subsequent business associations. The archived data structure is used to constrain the table structure, field mapping relationship and data organization method in the archived database. Once the above rules are read within the system cycle, they serve as the common basis for the data acquisition module and the condition judgment module, ensuring that the extraction of candidate data is consistent with the determination of archiving conditions, without requiring the data migration module to maintain independent rules separately. The data acquisition module forms a candidate record set for the production database within each system cycle. For data that meets the preset archiving time condition, the system compares the creation time of the business data with the time boundary corresponding to the current cycle and only includes data whose creation time is earlier than the time boundary in the candidate range. After the candidate range is formed, the condition judgment module continues to read the status field and business type field of each record and compares them with the archiveable status and preset archive type in the archiving rules. Only when the creation time reaches the archiving cycle, the record status is archiveable, and the business type falls into the preset archive type will the system add the record to the migration set. After this processing, the time condition, status condition, and business type condition are processed in layers to avoid sending data that may still be modified by online business directly into the archive database. In this embodiment, the data acquisition module uses log parsing to generate a candidate set to reduce the additional scanning load on the production database. The production database's operation log records the data change time, data table identifier, and business record identifier. The log parsing unit continuously reads the operation log according to the system cycle and sorts out the record index of records that have been written or changed in the archiving cycle. The data extraction unit retrieves the corresponding business data from the production database based on the data change time, data table identifier, and business record identifier. When using this method, the candidate range no longer relies on repeated traversal of the entire table. Instead, the operation log first provides clues about the records to be checked, and then the condition judgment module performs the archiving condition judgment. If there are temporary missing items or incomplete log reading in a certain period, the records that cannot be confirmed in that period will not directly enter the migration process, but will be retained until the next system period to continue parsing, so as to avoid extracting incorrect data when the identifier is incomplete. Once the set to be migrated is generated, the data migration module writes the corresponding business data into the archive database. This process first converts the field names, field order, and data types according to the archive data structure provided by the archive rule configuration module, so that the business records in the production database are consistent with the storage structure in the archive database. If the source field has null values, default values, or the status field is inconsistent with the constraints in the archive structure, the format conversion unit generates the corresponding storage value according to the established mapping relationship. Specifically, the established mapping relationship is defined by a pre-configured field mapping dictionary, which records the conversion rules between the source data type and the target data type, such as converting the source variable-length character type to the target fixed-length string type, and the null value filling strategy; when the source field is empty and the target archive structure requires it to be non-empty, the format conversion unit assigns a value according to the default placeholder specified for the field in the mapping dictionary, such as '-1' or 'unknown'. When the encoding system of the source status field is different from the archive structure, the unified translation of the status code is completed by looking up the key-value pairs in the dictionary mapping table; for abnormal records that cannot be mapped, they will not be written to the current batch and will be kept in the pending set to avoid affecting the continuous archiving of qualified data. In the implementation of the embodiment, the data migration module includes a format conversion unit and a batch writing unit. After the format conversion unit completes the structural adaptation, the batch writing unit divides the converted business data into multiple writing batches according to the preset batch size, and then writes them into the archive database in sequence. The preset batch size is determined based on the available archive input / output bandwidth of the production database, the preset single write time window, and the average number of bytes per record. In the quantization calculation, the system is pre-configured with the total network bandwidth. Total disk throughput capacity of the production database Total disk throughput of archive database By collecting real-time data on the current disk read / write throughput of the production database Current disk read / write throughput of the archive database And network idle status, quantifying the network idle status as the proportion of currently available network. The system dynamically evaluates and sets the available archiving input / output bandwidth allowed for archiving operations using a preset quantization mapping formula. The quantization mapping formula is:

[0019] in, This indicates taking the minimum value of each item within the parentheses; This is the constant for the dimensional conversion between network bandwidth and disk throughput. and To measure the disk throughput of the production database. and Disk throughput metrics for archived databases; The preset safety buffer coefficient has a value range of [value range missing]. And limit the expected time window for writing a single batch to 1. Therefore, the maximum tolerable input / output consumption for a single archiving session is the product of the two, and then this consumption is divided by the average data size of a single candidate record. The preset batch size is derived, i.e. To ensure resource controllability; After writing in batches, each batch retains an independent business identifier, primary key range, or a hash value correspondence generated by the source end recording non-null fields and the target end storing fields. This facilitates the subsequent data verification module to perform consistency verification batch by batch. If the writing of a certain batch is interrupted, the system only retains that batch as incomplete, while the other batches that have been successfully written continue to enter the verification process, avoiding the failure of the entire cycle's archiving results due to a single batch's anomaly. After receiving the write result in the archive database, the data verification module performs consistency verification on the corresponding business data in the production database and the archive database. The verification basis can be at least one of the business identifier and primary key, or it can be a hash value calculated from the corresponding business data content. In practice, for stable and unique primary key log records, the corresponding records before and after archiving can be located first using the business identifier and primary key, and then the record contents can be compared. For business data with many fields and large batches, the hash value of the corresponding data content can be calculated directly and then compared. When performing hash calculation, the system extracts all non-empty field values ​​of the corresponding record, concatenates them in alphabetical order of field names to form the target string, and uses a preset hash algorithm to calculate the corresponding hash value of the target string. To prevent boundary ambiguity when concatenating values ​​of non-empty fields directly adjacent to each other, for example, if field A has a value of '10' and field B has a value of '0 success', the result of concatenating them directly is the same as if field A has a value of '100' and field B has a value of 'success'. The system inserts a preset special separator, such as '', between the values ​​of each non-empty field. Furthermore, before concatenating, the system uses a preset escape character to escape the separator contained in the source data itself, or it uses a structured sequence of field byte length + special separator + field value for concatenation. For example, if the business data contains an amount field of '100', a status field of 'success', and a serial number primary key of 'serial number one', the system will concatenate the corresponding values ​​into the target string '100 success T001' according to the lexicographical order of the fields, and use a hash algorithm, such as Secure Hash Algorithm 256, to calculate and output a fixed-length hash value, which will serve as the unique identifier of the record. When the hash values ​​calculated for the same business identifier in the production database and the archive database are consistent, it is determined that the business data is consistent before and after the migration, thereby avoiding the performance overhead caused by field-by-field comparison; the business data corresponding to the batches with consistent verification are marked as pending cleanup and the mark is passed to the data cleanup module; if the verification results are inconsistent, the corresponding records are kept in the production database as they are and are not deleted, and are returned to the set to be verified or migrated, so as to ensure that the original data in the production database is always retained before the archiving results are confirmed; After receiving the verification result, the data cleanup module deletes the business data from the production database. This deletion action is premised on the completion of verification, thereby decoupling data migration from source cleanup. The data migration module is only responsible for completing the data writing, the data verification module is only responsible for confirming the consistency before and after archiving, and the data cleanup module only performs deletion on the confirmed data. The three are connected through batch status or record status. If a batch does not obtain a verification result for a long time after writing, or if the corresponding record in the archive database cannot be located during the verification process, the batch will not enter the cleanup stage, and the corresponding data in the production database will continue to be retained, thereby avoiding the accidental deletion of source data due to the interruption of the status chain. In the above-mentioned transaction log archiving scenario, the production database continuously receives new transaction writes, while the archiving system migrates data that has reached the retention period and whose business status is stable to the archiving database according to the system cycle; online transaction records are first generated and status changed in the production database, and candidate data is formed by the log parsing unit or direct query method, and then the condition judgment module excludes records that do not meet the archiving conditions; Data that meets the criteria is converted into a format and written to the archive database in batches. After verification, the corresponding records in the production database are deleted by the data cleaning module. Thus, online business uses the production database to process the current transaction, while historical transaction is gradually transferred from the production database to the archive database, achieving data archiving without changing the business master data processing method. In this embodiment, the data cleanup module includes: a status marking unit, used to add a cleanup mark to the record status field corresponding to the business data in the production database for business data that has been successfully migrated and passed consistency verification; The timed deletion unit is used to delete business data marked as pending cleanup within a preset time range.

[0020] The system also includes a task monitoring module, which monitors the migration progress of the data migration module and generates migration logs containing migration batch identifiers, completion status, and failure information.

[0021] The system also includes an exception handling module, which is used to trigger a re-migration task for the corresponding abnormal batch and output alarm information when the task monitoring module detects that the migration task has timed out, data writing is abnormal, or consistency check is inconsistent.

[0022] As the scale of transaction record archiving continues to expand, relying solely on the migration, verification, and deletion order of the implementation can complete the basic archiving. However, with a large number of batches and continuous archiving cycles, if the deletion and migration actions are tightly coupled, problems such as excessively concentrated deletion timing, difficulty in tracking failed batches, and mixing of abnormal and normal batches can easily occur. Therefore, this implementation introduces status flags, migration logs, and abnormal batch retry mechanisms in the cleanup and task control stages to keep the records to be deleted in the production database separate from the records currently being processed by the data migration module. For business data that has been written and confirmed to be consistent by the data verification module, the data cleanup module does not delete it immediately. Instead, the status marking unit adds a cleanup mark to the record status field of the production database. After adding a cleanup mark, the data that has been archived and confirmed is first retained in the production database as a status change. Subsequent deletion operations only need to be performed on the marked records, thus creating a clear boundary between migration confirmation and physical deletion. After the status marker is formed, the timed deletion unit reads the business data marked as to be cleaned and deletes it within a preset time range. The preset time range is used to constrain the execution window of the cleanup action so that the deletion load does not completely overlap with the archive writing. In specific implementation, the preset time range is determined based on the business load curve analysis results of the production database. The low period when the average daily online transaction concurrency of the system is lower than the preset busy threshold is selected, such as 2:00 am to 4:00 am, as the execution window, or it is dynamically opened when the CPU and memory usage of the production database are both below the safe baseline. In the dynamic activation determination, the timed deletion unit deploys millisecond-level high-frequency probes through the performance monitoring interface of the production database to collect CPU utilization and memory usage in real time for executing burst threshold circuit breaking. At the same time, it uses a preset period to aggregate and calculate the moving average of N consecutive positive integer collection periods to control the start and stop of the deletion window. The dynamic deletion window is only triggered when the moving average of CPU utilization is continuously lower than or equal to the preset first safety baseline, such as 30%, and the moving average of memory usage is continuously lower than or equal to the preset second safety baseline, such as 40%. If, during the execution of a deletion operation, the real-time collected value of any performance metric exceeds the corresponding safety baseline, the system will not forcibly interrupt the current transaction because there is a physical delay in the atomic deletion command being executed at the underlying database engine. Instead, it will immediately intercept subsequent deletion commands that have not yet been issued and record the processed primary key as a breakpoint to ensure the system resources required for online transactions. In the specific processing, the timed deletion unit first retrieves records in the production database whose record status field already has a cleanup mark, and then deletes them step by step in combination with the batch information corresponding to the generation of the mark. When performing physical deletion, the timed deletion unit adopts atomic operations based on status verification, specifically issuing a conditional deletion command to the database: the deletion condition not only binds the current primary key of the record, but also forcibly binds the cleanup mark value when it entered the cleanup set and the corresponding timestamp or version number. If the database returns zero affected rows after the instruction is executed, it indicates that the status field of a record has been modified again by the business system during the deletion window, or the record has been re-marked as non-deletable. In this case, the record is removed from the current deletion set and will be re-evaluated in subsequent cycles to avoid accidental deletion of the cleanup action due to business data entry, status rollback, or other reasons. The embodiment further includes a task monitoring module; this module tracks the processing progress of each migration batch when the data migration module performs batch writing and generates migration logs; the migration logs include at least a migration batch identifier, completion status, and failure information, wherein the migration batch identifier is used to map the migration data, verification results, and cleanup flags to the same processing unit, the completion status is used to distinguish progress such as written, pending verification, verified, flagged, or deleted, and the failure information records situations such as writing anomalies, batch interruptions, or verification inconsistencies; in this way, the data cleanup module does not need to backtrack the execution process of the original data migration module, but only needs to decide whether to continue subsequent actions based on the stable status results already saved in the migration logs; After the task monitoring module records the migration progress, the exception handling module can use the log information to identify abnormal batches. If the task monitoring module detects that the migration task times out, data write is abnormal, or consistency check is inconsistent, the exception handling module will trigger a re-migration task for the corresponding abnormal batch and output alarm information. The re-migration process does not involve repeating the process for all business data. Instead, it relocates the data range corresponding to the abnormal batch based on the migration batch identifier recorded in the migration log, and only re-extracts, rewrites, and re-verifies that batch. Before rewriting to the archive database, the system first performs a cleanup operation on any incomplete or erroneous data that may have already been written to the archive database based on the migration batch identifier and primary key. Alternatively, it directly uses a primary key-based overwrite update operation during rewriting to ensure that no duplicate records are generated in the archive database due to retries. This avoids the duplicate processing of batches that have already been archived and confirmed when a single batch fails. In the abnormal branch, regarding the above detection conditions, the time threshold for migration task timeout is dynamically set by the task monitoring module in combination with the average time of historical normal batches and the standard deviation of write timeout. Since network latency jitter is in the millisecond range and batch write time is in the second or minute range, using network latency jitter with differences to amplify write timeout will introduce calculation errors. The system uses a statistical inference method to set the timeout threshold. Specifically, at the end of each archiving cycle, the system calculates the average write time of the most recently completed positive integer M normal batches. and standard deviation And introduce the system's load tolerance factor. Typically, the value is 2 to 3, using the formula. The timeout threshold for the current period is calculated. If the migration task times out, the system first uses the task monitoring module to confirm whether the batch has actually completed writing but the status has not been returned. For batches that have completed writing but have not updated the status in time, the task monitoring module first records the completion status and then hands it over to the data verification module for further processing, instead of directly triggering repeated migration. If it is confirmed that the batch has not been written, the exception handling module restores it to the pending migration state; if a data write exception occurs, the original data of the corresponding record in the production database remains unchanged and is processed again by the re-migration task; if the consistency check is inconsistent, it means that the corresponding result in the archive database cannot be directly used for cleaning. In addition to triggering the re-migration, the system also keeps the original pending cleaning mark from being generated to prevent data that has not passed the check from being mistakenly entered the deletion stage; alarm information can be output in association with the migration batch identifier and failure information so that the operation and maintenance side can locate the problem range according to the batch granularity, but the alarm does not change the retention status of the original data in the production database; In the case of transaction record archiving, the above structure enables the system to operate as follows: After transaction records that meet the archiving conditions are migrated in batches, the task monitoring module continuously registers the execution progress of each batch; for batches that are verified to be consistent, their corresponding records are first marked as pending cleanup by the status marking unit, and then deleted by the timed deletion unit within a preset time range; if a batch experiences timeout, abnormality, or inconsistency during the migration, writing, or verification phase, the exception handling module restarts the archiving task of that batch separately according to the batch identifier recorded in the migration log and outputs the corresponding alarm; thus, normal batches continue to be executed according to the migration, verification, marking, and deletion process, while abnormal batches remain in an independent retry link, without affecting the archiving progress of other historical transactions; In this embodiment, the system also includes an archive query module, which is used to receive query requests from user terminals or external business systems and retrieve archive data corresponding to the query requests from the archive database.

[0023] The archive query module includes: an index building unit, used to build a query index for data in the archive database based on at least one of the business identifier, creation time, or primary key; and a data retrieval unit, used to match and return the target archive data based on the query fields in the query request and the query index.

[0024] After transaction records are migrated to the archive database, if the archived data is only used for storage and lacks a query channel, then when user terminals or external business systems need to trace historical transactions, they may still return to the production database to perform historical queries, weakening the effect of the archiving action in reducing the query load on the online database. Therefore, this implementation method sets up an independent query entry on the archive database side, so that the archived historical data can be retrieved directly in the archive database. After receiving a query request from a user terminal or external business system, the archive query module first parses the query fields carried in the request and maps the query fields to the corresponding data items in the archive database. The query fields can correspond to business identifiers, creation times, primary keys, or combinations thereof. Since the archive database has already stored historical business data migrated from the production database, the query action no longer depends on the original table records in the production database, but directly completes the matching and return in the archive database. The embodiment further includes an index building unit and a data retrieval unit; the index building unit builds a query index for data in the archived database based on at least one of the following: business identifier, creation time, or primary key; The index building unit creates a clustered index on the primary key in the archive database to ensure access continuity at the physical storage level; at the same time, it creates a secondary index with a multi-way balanced tree structure for the combination of the business identifier and the creation time field; when creating the secondary index, the date prefix of the creation time accurate to the day is used as the first-level index node, the business identifier is used as the second-level index node, and the leaf nodes store the primary key value of the corresponding archive data record. Creating query indexes based on the business entity, generation time period, and unique record identifier can avoid query requests traversing the entire dataset and improve the response efficiency of the archive database to historical queries. During index construction, after the data migration module completes batch writing, the system can synchronously supplement the corresponding index entries for the newly entered batches of archived data, ensuring that the data content remains associated with the index records. The index stores the matching relationship between archived data and business identifiers, creation times, or primary keys. When subsequent query requests arrive, the data retrieval unit first locates the corresponding index entry based on the query fields, and then uses the index entry to retrieve the target archived data. In this way, the query entry point and the migration entry point are associated in the archived database through the index, without requiring the query module to re-parse the business table structure in the production database. In abnormal branches, if the query request only contains some of the query fields, the data retrieval unit will perform matching in the existing index based on the existing fields; if the key fields that can locate the index are missing, the system will return a no-hit result or prompt for supplementary query conditions, instead of rolling back to the production database to perform a full search; if an archive batch has been written to the archive database but its index has not yet been built, the corresponding records of that batch can be temporarily not returned before the index is built, or the locatable part can be returned based on the completed index range, thereby ensuring that the query results are consistent with the index status and avoiding the return of an incomplete data set; In the case of querying transaction history, after the external business system initiates a query request, the archive query module first identifies fields such as business identifier, creation time or primary key in the request, and then the data retrieval unit uses the query index pre-built by the index building unit to locate the corresponding archive data in the archive database and return the result; During retrieval, if the query request carries a creation time range and a business identifier, the data retrieval unit first matches the corresponding date range in the auxiliary index, then locates the set of all primary key values ​​corresponding to the business identifier by tracing down the multi-way balanced tree branch, and reads the complete archived data records in batches through the primary key mapping of the clustered index. In this way, the production database mainly maintains the current transaction processing and recent data read and write, while the tracing of historical transactions is transferred to the archived database, and the archived data remains queryable after migration.

[0025] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data archiving system based on data migration, characterized in that, include: A production database is used to store business data, which includes creation time, record status, and business type. The archiving rules configuration module is used to configure the preset archiving period, preset archiving type, archiveable status, and archived data structure; The data acquisition module is used to acquire business data that meets preset archiving time conditions from the production database. The preset archiving time conditions include: the time interval between the current time and the creation time of the business data is greater than or equal to the preset archiving period. The condition judgment module is used to determine whether the business data meets the archiving conditions based on the creation time of the business data, the recording status of the business data, and the business type of the business data. The archiving conditions include: the time interval between the current time and the creation time of the business data reaches the preset archiving period, the recording status of the business data is the archiveable status, and the business type of the business data belongs to the preset archiving type. An archive database is used to store business data migrated from the production database; The data migration module is used to write the business data into the archive database according to the archive data structure when the business data meets the archive conditions. The data verification module is used to perform consistency verification on the corresponding business data in the production database and the archive database based on at least one of the business identifier and primary key, or based on the hash value calculated from the corresponding business data content. The data cleanup module is used to delete the business data from the production database after the data verification module confirms that the verification is consistent.

2. The data archiving system based on data migration according to claim 1, characterized in that, The data acquisition module includes: a log parsing unit, used to read the operation logs generated by the production database; The data extraction unit is used to extract business data that has reached the preset archiving period based on the data change time, data table identifier, and business record identifier recorded in the operation log.

3. A data archiving system based on data migration according to claim 1, characterized in that, The data migration module includes: a format conversion unit for converting the business data into a pre-defined data storage format of the archive database; and a batch writing unit for writing the converted business data into the archive database in batches according to a preset batch size.

4. A data archiving system based on data migration according to claim 1, characterized in that, The data cleaning module includes: a status marking unit, used to add a cleanup mark to the record status field corresponding to the business data that has been successfully migrated and passed the consistency check in the production database; The timed deletion unit is used to delete business data marked with the cleaning mark within a preset time range.

5. A data archiving system based on data migration according to claim 1, characterized in that, It also includes a task monitoring module, which is used to monitor the migration progress of the data migration module and generate migration logs containing migration batch identifiers, completion status and failure information.

6. A data archiving system based on data migration according to claim 5, characterized in that, It also includes an exception handling module, which is used to trigger a re-migration task for the corresponding abnormal batch and output alarm information when the task monitoring module detects that the migration task has timed out, data writing is abnormal, or consistency check is inconsistent.

7. A data archiving system based on data migration according to claim 1, characterized in that, It also includes an archive query module, which is used to receive query requests from user terminals or external business systems and retrieve archive data corresponding to the query request in the archive database.

8. A data archiving system based on data migration according to claim 7, characterized in that, The archive query module includes: an index building unit, used to build a query index for data in the archive database based on at least one of business identifier, creation time, or primary key; and a data retrieval unit, used to match and return target archive data based on the query field in the query request and the query index.

Images