A data migration method for a black-box big data service system

Abstract

The application provides a data migration method for a black-box big data business system, and relates to the technical field of financial science and technology data processing.The method comprises the following steps: according to database layer table activity information of the black-box big data business system, classifying and confirming the migration range of the tables in the black-box big data business system; deploying a trigger at the database layer of the black-box big data business system, tracking the real-time changes of the database caused by transactions through the trigger, and obtaining tracking results; based on the tracking results, analyzing the correlation between the business tables and the data distribution law in the black-box big data business system; and according to the data distribution law, statistically analyzing the business table fields of the black-box big data business system to obtain field statistical characteristics.The application can guarantee the integrity and accuracy of data migration, and reduce the switching and production risk of the black-box big data business system and the target system.

Description

Technical Field

[0001] This invention relates to the field of financial technology data processing technology, and in particular to a data migration method for black-box big data business systems. Background Technology

[0002] In the process of system upgrades and replacements in the fintech field, data migration is a core element in ensuring business continuity between the old and new systems.

[0003] Black-box big data business systems, due to their technological secrecy and highly encapsulated application-layer code and interfaces, generally suffer from the following technical defects: They cannot directly obtain the relationships and data distribution patterns between business tables. This defect becomes a key bottleneck restricting the quality of cross-system data migration. On the one hand, due to a lack of clear understanding of the relationships between business tables, existing migration solutions struggle to accurately define the migration scope: they cannot effectively distinguish the business attributes and migration necessity of hot, cold, and static tables, easily leading to the omission of critical business data or the mis-migration of redundant and obsolete data, directly compromising the integrity of data migration and consequently affecting the continuity of the new system's business lifecycle. On the other hand, limited by the unknown nature of data distribution patterns, a large amount of unstructured XML data in black-box systems is difficult to parse effectively: it is impossible to clearly define the nesting logic and mapping rules of data nodes, nor to identify the numerical characteristics, enumeration ranges, and abnormal data conditions of fields. This hinders the conversion of unstructured data to structured data, and the meaning and usage of fields between the old and new systems cannot be precisely aligned, ultimately resulting in insufficient accuracy of the migrated data, severely impacting the functional implementation and business continuity of the target system.

[0004] In summary, existing data migration methods fail to provide effective solutions to the technical barriers of black-box systems and cannot simultaneously meet the dual requirements of migration integrity and accuracy. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a data migration method for black-box big data business systems, which can ensure the integrity and accuracy of the migrated data, reduce the risk of switching and production between the black-box big data business system and the target system, and provide general and reliable technical support for the upgrading and replacement of black-box systems in the financial technology field.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A data migration method for black-box big data business systems, the method comprising:

[0008] Based on the database layer table activity information of the black box big data business system, the tables in the black box big data business system are classified and the migration scope is confirmed.

[0009] Deploy triggers at the database layer of the black-box big data business system to track real-time database changes caused by transactions and obtain tracking results; based on the tracking results, analyze the relationships between business tables and the data distribution patterns in the black-box big data business system.

[0010] Based on the data distribution patterns, statistical analysis was performed on the business table fields of the black-box big data business system to obtain the field statistical characteristics;

[0011] Based on the aforementioned relationships and data distribution patterns, and with reference to field statistical characteristics, unstructured data in XML format in the black-box big data business system is converted into structured data.

[0012] Based on the confirmed migration scope, the aforementioned relationships, and the transformed structured data, design a migration intermediate table connecting the black-box big data business system and the target system;

[0013] By designing an intermediate migration table, the meaning and usage of fields between the black-box big data business system and the target system are aligned to obtain the field alignment results.

[0014] Based on the design of the intermediate migration table and field alignment results, perform full-cycle data verification from the black-box big data business system to the target system and generate a verification report.

[0015] In the target system refactoring project, the design of the migration intermediate table, field alignment results, and verification reports will be coordinated with the application development process of the target system to support the functional testing, data verification, and system switchover of the target system.

[0016] The above-described solution of the present invention has at least the following beneficial effects:

[0017] By employing technical means such as classifying tables to confirm the migration scope, parsing data associations and distribution patterns using database-level triggers, converting XML unstructured data using multi-dimensional splitting, building data bridges using intermediate migration tables, eliminating system differences through field-level alignment, and conducting full-cycle quality checks in collaboration with the entire application development process, this approach effectively overcomes the technical problems caused by the technical blockade of black-box big data business systems, including unclear data logic, obstructed unstructured data migration, insufficient field compatibility, and difficulty in ensuring migration integrity and accuracy. This achieves the technical effects of accurately defining the migration scope, realizing efficient and compliant data migration, ensuring business continuity between the new system (the system to be switched to, i.e., the target system) and the old system (the source black-box big data business system), and reducing the risks of switching between the old and new systems. It provides universal and reliable technical support for upgrading black-box systems in the fintech field. Attached Figure Description

[0018] Figure 1This is a flowchart illustrating a data migration method for a black-box big data business system provided by an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0020] like Figure 1 As shown, an embodiment of the present invention proposes a data migration method for a black-box big data business system, the method comprising the following steps:

[0021] Step 1: Based on the database layer table activity information of the black box big data business system, classify the tables in the black box big data business system and confirm the migration scope.

[0022] Step 2: Deploy triggers at the database layer of the black-box big data business system to track real-time database changes caused by transactions and obtain tracking results; based on the tracking results, analyze the relationships between business tables and the data distribution patterns in the black-box big data business system.

[0023] Step 3: Based on the data distribution pattern, perform statistical analysis on the business table fields of the black-box big data business system to obtain the field statistical characteristics;

[0024] Step 4: Based on the aforementioned relationships and data distribution patterns, and referring to the statistical characteristics of the fields, convert the unstructured data in XML format in the black-box big data business system into structured data;

[0025] Step 5: Based on the confirmed migration scope, the aforementioned relationships, and the transformed structured data, design a migration intermediate table connecting the black-box big data business system and the target system;

[0026] Step 6: Through the designed migration intermediate table, align the field-level meanings and usages between the black-box big data business system and the target system to obtain the field alignment results;

[0027] Step 7: Based on the design of the intermediate migration table and field alignment results, perform a full-cycle data check from the black-box big data business system to the target system and generate a check report;

[0028] Step 8: In the target system's refactoring project, the design of the migration intermediate table, the field alignment results, and the verification report are coordinated with the target system's application development process to support the target system's functional testing, data verification, and system switching.

[0029] In this embodiment of the invention, the scope of data migration for black-box big data business systems can be accurately defined, effectively overcoming the challenge of unclear data association and distribution patterns caused by technological blockades. It achieves efficient conversion of unstructured XML data to structured data, eliminates differences between the black-box big data business system and the target system through intermediate migration tables and field-level alignment, and rigorously controls the integrity and accuracy of the migrated data throughout the entire lifecycle. Simultaneously, deep collaboration with the target system's application development process provides strong support for functional testing and data verification, facilitating a smooth switch between the black-box big data business system and the target system, ensuring business continuity, and reducing the risk of switching and deployment between the black-box big data business system and the target system. The target system is a standardized system that receives the migrated data from the black-box big data business system and supports core business operations. It possesses clear table structure design specifications, data storage standards, and business logic processing frameworks, adapting to the specifications of the migrated data and enabling the implementation of business functions. The target system needs to receive the migrated data through the mapping and splitting rules of the intermediate migration tables, complete the business acceptance after field alignment and data verification, and ultimately achieve a smooth switch with the black-box big data business system.

[0030] In a preferred embodiment of the present invention, step 1 above may include:

[0031] Step 1.1: Query the database system tables of the black-box big data business system to obtain the last data update time of all tables. Specifically, this includes: First, a brief description of the black-box big data business system: This refers to a big data processing system whose internal data processing logic and architectural details are not exposed externally. It only provides business data services through preset interfaces or database interactions. It typically adopts a distributed storage architecture, integrating multiple types of databases to support massive and diverse core business data storage and processing. It covers business tables and system auxiliary tables for various business scenarios such as user account opening and fund transactions, and is the core data source for this data migration work. Initiate a compliant access process by submitting a written application to the operation and maintenance management department of the black-box big data business system, specifying the purpose of the application as querying the last data update time of the tables and the coverage scope as all business-related tables of the black-box big data business system. After approval, legal access rights are obtained. Based on the actual database type deployed in the black-box big data business system, confirm its corresponding data dictionary tables or views and corresponding key fields: Specifically, for Oracle databases, the DBA-TAB-MODIFICATIONS data dictionary view is retrieved. For the LAST-MODIFIED field value, the MySQL database primarily retrieves the following: the LAST-UPDATE field value of the innodb-table-stats table and the UPDATE-TIME field value of the information-schema.tables table. Following the corresponding database operation specifications, the query operation is executed, traversing all registered and filed business tables and system auxiliary tables in the black-box big data business system, extracting the last data update time for each table. If, during the query process, some tables are found to have missing last update time fields or obviously abnormal field values, such as timestamps exceeding a reasonable range, the operation and maintenance personnel are immediately collaborated to supplement the accurate last data update time of the table by tracing back the database operation logs and checking the black-box big data business system configuration files. After the query is completed, the table name, business module, database type, last data update time, query operator, query timestamp, and the dictionary table / view name from which the data originates are uniformly entered into a pre-defined standardized data record table, forming an initial query dataset that can be traced throughout the entire process.

[0032] Step 1.2: Based on the last data update time, tables with no data changes in the past five years are classified as static tables, tables with no data changes in the past year are classified as cold tables, and tables with data changes in the past year are classified as hot tables. Specifically, this includes: obtaining the standard current time of the database server in the black-box big data business system, accurate to the day, as the time calculation benchmark to avoid classification errors caused by terminal time deviations; retrieving the initial query dataset and processing each table in order of name, calculating the interval in days between the last data update time and the current standard time for each table using date calculation logic; for tables with an interval exceeding 1825 days, calculated based on 365 days per year, further checking the database operation logs for the past five years to confirm whether... Tables with data additions, modifications, deletions, and table structure adjustments, such as adding or removing fields or changing table types, without any change records, are classified as static tables. For tables with an interval of 365 to 1825 days, the database operation logs for the past year are carefully reviewed; if no data change records are found, they are classified as cold tables. For tables with an interval of less than 365 days, if at least one data change operation is confirmed during the period, regardless of the amount of data, they are classified as hot tables. After classification, a table type field is added to the initial query dataset to indicate the static, cold, or hot table category for each table, and the classification criteria are supplemented with notes, including the interval and the results of the change record review, forming a classified table list dataset.

[0033] Step 1.3 involves identifying and updating static tables to identify non-parameter tables that are not being updated, classifying them as non-migration tables. This includes collecting all business documents from the black-box big data business system, such as the system function manual, core business process diagrams, and parameter configuration manuals. It also involves reviewing historical parameter table registration records retained by the operations and maintenance department to create a parameter identification reference document set. A special meeting is then held with business experts, relevant operations and maintenance personnel, and technical analysts to verify the table name, field meanings, and data usage of each static table based on the reference document set. This determines whether the table is a necessary parameter table supporting the core functions of the black-box big data business system, such as transaction settlement, user authentication, and rule configuration. The identified parameter tables are then added to the static parameter table list. The process was confirmed by business experts; subsequently, a static table update verification process was initiated. Under the premise of not affecting the normal operation of the black-box big data business system, a temporary verification mechanism was deployed to monitor all static tables in real time for 7 days, recording the table access frequency, data write operations, and table structure changes; at the same time, the database historical operation logs of the past five years were queried to trace the change records of static tables to ensure that no hidden updates were missed; combining the parameter table identification results and update verification results, static tables that were not included in the static parameter table list and had no update records, including those during the monitoring period and the historical retrospective period, were officially marked as non-migration tables after dual confirmation by the technical team and the business team, entered into the non-migration table list and noted the reason for the judgment, namely, non-parameter tables with no update records.

[0034] Step 1.4 involves analyzing the offline status of cold tables, removing offline tables, and confirming the necessity of migrating the remaining offline cold tables. This includes: retrieving historical operation and maintenance records from the black-box big data business system, such as system upgrade logs, table maintenance records, business function offline notification documents, and related meeting minutes, to establish an offline status verification document set; extracting the table name of each cold table, comparing it with the verification document set, and finding the corresponding business function description and offline approval record. If the document clearly states that the business function corresponding to the table has stopped service and there is no plan to reuse it, it is preliminarily determined to be an offline table; to ensure the accuracy of the results, verifying the actual operating status of the table with technical maintenance personnel, such as whether there are process calls or data interactions, and confirming the authenticity of the business function offline with the business manager. After verifying the authenticity and irreversibility of the data, the table is removed from the cold table migration candidate list. The table list dataset is then marked as offline and removed, along with the basis for removal. For cold tables that are not offline, their corresponding business scenarios, data flow paths, and relationships with other tables are analyzed to generate a cold table business analysis report. A migration assessment meeting is held with the business manager, target system developers, and the data migration team. Based on the target system's business plan, data storage architecture, and functional requirements, a comprehensive assessment is conducted from dimensions such as data availability, business necessity, and cost-effectiveness to determine whether each cold table needs to be migrated. Cold tables that pass the assessment are included in the cold table migration list, while those that fail are included in the cold table non-migration list. Detailed records of assessment opinions and decision-maker information are maintained for all cases.

[0035] Step 1.5 involves reviewing each hot table individually to confirm its migration. Based on the combined confirmation results of static, cold, and hot tables, the migration scope is ultimately confirmed. This includes: establishing a dedicated hot table review team comprised of technical personnel responsible for data logic analysis and business personnel responsible for business importance assessment, clearly defining their responsibilities; reviewing each hot table one by one according to the priority of the business modules in the black-box big data business system; technical personnel sharing the table's data update frequency, core field meanings, data volume, and dependencies with other tables with the team; business personnel assessing the impact of the table's data on the target system's business continuity by comparing it with the core business processes and high-frequency transaction scenarios of the black-box big data business system; for hot tables with dependencies, simultaneously verifying the classification and migration status of related tables to ensure the integrity of the data logic after migration; and finalizing the migration scope through team discussion. After reaching a consensus, the migration status of each hot table is determined, i.e., whether it needs to be migrated or not, and entered into the hot table migration confirmation list. Then, the confirmation results of the three types of tables are integrated: the static parameter table list of static tables is the table to be migrated, and the non-migration table list is the table not to be migrated; the cold table migration list of cold tables is the table to be migrated, and the non-migration cold table list is the table not to be migrated; the tables to be migrated from the hot table migration confirmation list of hot tables are included in the scope. All tables to be migrated are categorized and organized by business module to form a complete migration scope list containing information such as table name, table type, business module, migration status, confirmation basis, confirmer, and confirmation time. A list review meeting is organized with the migration team, business departments, and target system development departments to check the list content one by one, resolve any disputes, and finally form a signed and confirmed formal migration scope document, completing the final confirmation of the data migration scope of the black box big data business system.

[0036] In a preferred embodiment of the present invention, step 2 above may include:

[0037] Step 2.1: In the database layer of the black-box big data business system, create triggers for tracking data changes for the business tables. Specifically, this includes: first, collaborating with the black-box big data business system's operations and technical teams to conduct a pre-launch assessment of trigger creation, ensuring that trigger deployment will not consume excessive database resources or affect the transaction response speed and normal business operation of the black-box big data business system; second, based on the database type of the black-box big data business system and following the corresponding database's trigger creation specifications, planning the trigger creation scheme for each of the business tables within the final confirmed migration scope; all triggers created in this step are DML row-level triggers, which are automatically executed stored procedures closely related to data operations in the database, and their core consists of three parts: trigger event, trigger timing, and trigger body; the trigger event explicitly covers data addition, ... The three core operations—modification, deletion, and resubmission—are triggered by AFTER, meaning they are executed after the data operation is completed. This avoids interfering with the normal execution of the original business operations. The triggering mode uses row-level triggering to ensure that changes to each row of data are accurately captured. Furthermore, the old and new values ​​of fields before and after the change can be obtained through the pseudo-record function of the corresponding database. The core function of the trigger body is to capture key metadata related to the operation in a lightweight manner. It does not contain complex business logic and only focuses on the information required for data tracking, avoiding increasing the database load. The entire creation process of all triggers is verified in the test environment of the black-box big data business system. By simulating different types of data operations, it is confirmed that the triggers can respond accurately, capture information completely, and have no performance anomalies before being synchronized to the database layer of the production environment. This ensures that all triggers meet the system's security management requirements and data tracking needs.

[0038] Step 2.2: Capture data operation records on the business tables using triggers to form raw tracking logs. Specifically, this includes: when a user of the black-box big data business system initiates a business operation or a related system triggers data interaction, and the operation involves business tables within the migration scope, the corresponding business table's DML row-level trigger will respond in real time to the triggering mechanism; the trigger will completely capture all key information of the operation according to preset rules, specifically including the data operation type, i.e., clearly marked as add, modify, or delete; the complete name of the business table involved; the original and changed values ​​of key fields; if it is an add operation, only the new value is recorded; if it is a delete operation, only the old value is recorded; and so on. The system generates a unique transaction identifier to ensure that each operation is traceable individually; it includes an operation timestamp accurate to milliseconds; the identity identifier of the entity initiating the operation, such as the user account, system interface name, and operation execution result; and it synchronizes captured fragmented information to the log storage node in real time through a lightweight transmission mechanism, sorting and integrating it according to the rules of daily partitioning + transaction identifier segmentation. Each log record contains a fixed field order and format identifier. After storage, the system automatically performs log integrity verification to check whether fields are missing or whether the format is compliant, ultimately forming a well-structured and traceable original tracking log. At the same time, it sets hierarchical access permissions for the logs to prevent data leakage.

[0039] Step 2.3 involves parsing and cleaning the original tracking logs, extracting structured tracking records containing operation types, table names, key field values, and operation timestamps. Specifically, this includes: first, preprocessing the original tracking logs by merging the log files from each daily partition into a unified processing dataset, and simultaneously decompressing compressed log fragments to ensure data integrity and usability; then performing log cleaning, identifying and removing invalid data according to preset rules: duplicate records are determined by identical transaction identifiers, operation timestamps, and business table names; system exception logs are those with more than two missing fields or timestamps exceeding a reasonable range; and test operation logs are filtered and excluded based on the identity of the operation initiator. After cleaning, Each valid log entry is broken down to accurately extract core field information such as operation type, business table name, key field values, operation timestamp, transaction identifier, and operation execution result. The extracted information is then standardized in format: operation timestamps are standardized to year, month, day, hour:minute:second; millisecond format; the data types of key field values ​​are standardized, for example, redundant symbols are removed from numeric fields, and character fields are standardized in encoding format; and the logical consistency of fields is verified, for example, modification operations must have both old and new values, and transaction identifiers and operation timestamps must match the business flow sequence; finally, the processed information is stored according to business modules to form a structured tracking record with complete fields, unified format, and clear logic.

[0040] Step 2.4, based on structured tracking records, identifies multiple business tables that have undergone collaborative changes in business transactions by performing correlation analysis according to time series and transaction identifiers, in order to construct the correlation relationships between the business tables. Specifically, this includes: first, performing transaction identifier uniqueness verification on all structured tracking records, eliminating records corresponding to duplicate or invalid transaction identifiers, ensuring that only one complete set of records is retained for each business transaction; then, sorting the verified structured tracking records in ascending order by operation timestamp, using the unique transaction identifier as the core grouping basis, classifying all table operation records corresponding to the same transaction identifier into an independent transaction data group, fully retaining the order and correlation information of all table operations involved in each business transaction; and performing in-depth decomposition analysis on each transaction data group to identify collaborative change relationships: if multiple business tables under the same transaction identifier undergo collaborative changes... If the operation timestamps of tables are within a preset time window (usually set to 100 milliseconds) and are linked through key fields such as order number, user ID, and account number, then these business tables are identified as collaborative change tables. The change order, key field names, and corresponding values ​​of the collaborative change tables are recorded in detail; for example, the order number in table A must match the order number in table B. Cross-validation is performed by extracting data from multiple similar business transactions to eliminate occasional abnormal associations, such as temporary non-fixed table associations triggered in a single transaction. Stable and universal collaborative change logic between business tables is identified, and a list of business table associations is constructed, including table name, key fields, association type (one-to-one or one-to-many), and change sequence. The accuracy of the association logic is then jointly reviewed and confirmed by the technical and business teams.

[0041] Step 2.5 analyzes the frequency and distribution of field value changes in the structured tracking records. Based on the constructed relationships, it statistically analyzes the data distribution patterns under different business scenarios. Specifically, this includes: first, classifying the structured tracking records by business table name; traversing all records in each business table; and statistically analyzing the core characteristics of each field: for numeric fields, statistically analyzing the value range, data density (distribution ratio of different interval values), and change frequency; for classification identifier fields, statistically analyzing the value set, frequency of occurrence, and stability of each value, i.e., whether values ​​have been added or decreased in the past 30 days. Then, based on the business architecture document and core business process description of the black-box big data business system, the structured tracking records are classified according to business scenarios, such as user account opening, fund transfer, transaction settlement, account information modification, and bill generation, ensuring that each business... Each scenario corresponds to a clearly defined set of business tables. For each business scenario, the changes in field values ​​of the relevant business tables are analyzed in depth: the value constraints of key fields are determined, such as the transaction amount must be greater than 0; frequently occurring field value combinations are identified, such as the fixed matching relationship between document type and user type in the account opening scenario; the triggering conditions for field value changes are identified, such as the completion of associated fund transactions for account status changes; and the matching rules with fields in related tables are identified, such as the order number in the settlement table must be completely consistent with the order number in the transaction table, and the settlement amount must correspond to the transaction amount in the transaction table. The results of the pattern analysis for each scenario are summarized and refined, and abnormal patterns of individual special transactions are eliminated, while common features are retained. Finally, a complete data distribution pattern document is formed, categorized by business scenario and including a list of related tables, field feature descriptions, and field matching rules.

[0042] In a preferred embodiment of the present invention, step 3 above may include:

[0043] Step 3.1: Based on the data distribution patterns obtained from statistical analysis, identify the numeric fields and classification identifier fields in the business table fields. Specifically, this includes: first completing preliminary preparations by simultaneously retrieving the data distribution pattern document from the black-box big data business system, existing system business documents, and historical data dictionary descriptions to form a field analysis reference set; then, using the field values ​​recorded in the data distribution pattern document as the core basis, initiate the initial field type judgment process: for field values ​​that support arithmetic operations such as addition, subtraction, multiplication, and division, and whose values ​​exhibit continuous or semi-continuous distribution characteristics, while the field name contains characteristic words such as amount, quantity, duration, and interest rate, they are initially determined to be... Numeric fields; for fields with only a limited number of discrete values, each corresponding to a clear business category or status identifier, and whose names contain characteristic words such as type, status, code, and level, they are initially identified as category identifier fields; to avoid misjudgment due to the lack of field descriptions in the black-box system, a small amount of actual field data from structured tracking records is sampled for verification. For example, the numerical fields initially identified are verified to see if they can be used for valid arithmetic operations, and the category identifier fields initially identified are verified to see if their values ​​conform to the business classification logic. At the same time, business personnel are organized to cross-check the initial judgment results to finally determine the accurate field type classification results and form a preliminary list.

[0044] Step 3.2: For the identified numeric fields, analyze their numerical range, data density, and trends to form numerical distribution characteristics. Specifically, this includes: for each numeric field list, extracting all valid data from the structured tracking records, removing extreme outliers caused by anomalies in the black-box big data business system before extraction (e.g., field values ​​that are negative but should be positive in business logic); conducting numerical range analysis by traversing the valid data to determine the maximum and minimum values ​​of each field, clarifying the normal value boundaries of the fields; and dividing reasonable numerical intervals based on the corresponding business rules of the black-box big data business system during data density analysis (e.g., monetary fields are divided according to commonly used business rules). The data is divided into price ranges, and the frequency of field values ​​within each range and their proportion of the total valid data for that field are statistically analyzed to clearly present the concentration of data distribution across different ranges. When conducting trend analysis, field values ​​are grouped by time granularity (e.g., daily, weekly), and the changes in field values ​​within each time granularity are traced using operation timestamps. The fluctuation patterns of field values ​​over time are observed to determine whether the overall trend is upward, downward, or stable. Key time nodes of trend changes and possible business drivers are also recorded. The numerical range, the proportion of data density in each range, the time trend, and key node information are systematically summarized to form a unique numerical distribution characteristic report for each numerical field.

[0045] Step 3.3: For the identified classification identifier fields, analyze the value set, frequency of occurrence, and stability of the classification identifier fields to form an enumerated distribution feature. Specifically, this includes: for the determined list of classification identifier fields, traversing all values ​​of the corresponding fields in the structured tracking records one by one, forming a complete value set for the field through deduplication, and then verifying the rationality of the value set with the business documents of the black-box big data business system, eliminating invalid values ​​that obviously do not conform to business logic, such as garbled characters, meaningless characters, etc.; conducting frequency analysis, counting the number of times each value appears in the value set, calculating the proportion of each value to the total number of valid records for the field, identifying high-frequency and low-frequency values, and simultaneously labeling... Note the core business scenarios corresponding to high-frequency values; when conducting stability analysis, select the structured tracking records of the past 30 days as the core analysis sample, and at the same time retrieve the historical data of the past 6 months as auxiliary references to monitor whether there are new values ​​or disappearances of existing values ​​in the fields during the core sample period. Compare the auxiliary reference data to determine whether the value changes are temporary fluctuations or long-term trends, thereby determining the stability level of the field values; in addition, record the specific business meaning corresponding to each value, such as 01 for the status field corresponding to normal, 02 for pause, etc. Finally, summarize to form a classification identifier field enumeration distribution characteristic report containing the set of values, the frequency of occurrence of each value, the stability level, and the business meaning of the values.

[0046] Step 3.4: Combining the data distribution patterns under different business scenarios, perform scenario-based verification and correction of numerical distribution characteristics and enumeration distribution characteristics, generating scenario-based verification results. Specifically, this includes: based on the core business scenario division standards of the black-box big data business system, clarifying the boundary range of each business scenario and the corresponding business tables and fields; accurately classifying the numerical distribution characteristic report and enumeration distribution characteristic report according to business scenarios to ensure that each scenario corresponds to the relevant field distribution characteristics; conducting scenario-based verification for each business scenario: for numerical fields, verifying whether their actual values ​​in the scenario are completely within the determined numerical range, and whether the data density distribution matches the business volume distribution logic of the scenario, such as peak business... Check whether the value density of the amount field corresponding to the business period meets expectations; for the category identifier field, confirm whether its actual value in this scenario belongs to the determined value set, and whether the frequency of each value is consistent with the business operation rhythm of this scenario; if the verification finds that the actual performance of the field deviates from the generated distribution characteristics, immediately analyze the reasons for the deviation in combination with the specific business rules of this scenario. For numerical fields, the numerical interval division method can be adjusted and the value boundary can be corrected. For the category identifier field, scenario-specific value content can be added and the business meaning of the value can be corrected. After the correction is completed, verify again until it conforms to the scenario business logic, and finally generate a scenario-based verification result document containing the verification conclusion of each field, the analysis of the reasons for the deviation, and the correction explanation.

[0047] Step 3.5 integrates numerical distribution features, enumeration distribution features, and scenario-based verification results to generate field statistical features. Specifically, this includes: first, building a hierarchical field statistical feature integration framework, divided by business modules, business tables, and fields to ensure a clear structure and compatibility with the business architecture of the black-box big data business system; then, incorporating the numerical distribution feature reports for each numerical field and the enumeration distribution feature reports for each category identifier field into their respective business modules, business tables, and field levels within the framework; finally, associating and matching the scenario-based verification results with the corresponding field features in the framework, and marking the verification status for each field feature, i.e., through... After verification or correction, the data is approved, and the specific basis for feature correction, the differences before and after correction, and the applicable business scenarios are recorded in detail to ensure that the integrated information is traceable and verifiable. Finally, a completeness and consistency check is carried out: the completeness check confirms that each field of all business tables within the migration scope has generated corresponding statistical features and that no feature information is missing; the consistency check confirms that the feature descriptions of the same field in different business scenarios are consistent and the terminology is uniform; after the check is correct, the integrated content is systematically sorted and formatted, and finally a complete field statistical feature document covering all business table fields within the migration scope of the black box big data business system is generated.

[0048] In a preferred embodiment of the present invention, step 4 above may include:

[0049] Step 4.1: Based on the constructed association relationships, parse the nesting levels and reference relationships of data nodes in the XML format unstructured data to form a node relationship model. Specifically, this includes: first, conducting preliminary planning for XML data sample extraction, ensuring that the samples fully cover all core business scenarios in the black-box big data business system, such as user account opening, fund transactions, bill settlement, and account information modification. Simultaneously, it considers different data volumes, such as small-volume XML for a single user account opening record, large-volume XML for batch bill settlement, and XML data with different nesting complexities, such as basic transaction data XML with 3 levels of nesting, and complex settlement detail XML with 8 levels of nesting, ensuring the samples have sufficient representativeness and coverage. From the database or file storage nodes of the black-box big data business system, extract a sufficient number of XML format unstructured data samples according to the plan, establish sample sets according to business scenarios, and label the business scenario, data volume, and nesting level information corresponding to each sample.

[0050] The XML node parsing process is initiated, employing a layer-by-layer traversal method starting from the XML root node and sequentially parsing each level of child nodes. Detailed information about each node is recorded, including its name, unique identifier, nesting path (e.g., the root node, user information nodes, and the hierarchical path of basic information child nodes), the correspondence between parent and child nodes, and whether the node is a duplicate (e.g., duplicate transaction detail child nodes corresponding to multiple transaction details in a fund transaction scenario). Simultaneously, key fields used to link different nodes in the XML data are identified, such as the user ID in a user account opening scenario, the order number in a fund transaction scenario, and the account number in a bill settlement scenario. The names, data types, and value examples of these key fields are extracted and compared one by one with the business table relationships constructed in step 2.4. This verifies the consistency of key field names and value rules between the key fields and the related fields in the business table (e.g., whether the 16-character format of the order number in the XML matches the format of the order number in the business table). This ensures that the node association logic conforms to the business table association logic, and invalid association fields that do not conform to the business logic are removed.

[0051] Based on the above analysis and comparison results, the construction process of the node relationship model is initiated: First, the basic framework of the model is built, defining the core triplet structure of node attributes, relationship types, and association weights. The parsed node names, unique identifiers, and other information are entered into the framework as node attributes. The identified relationships such as inclusion, association, and parallel are used as basic relationship types. The initial association weights are set according to the matching degree of key fields: a perfect match weight is set to 1.0, a partial match weight is set to 0.6, and a no-match weight is set to 0.1. Next, XML sample parsing data from various business scenarios are imported, and the framework is filled in according to scenario categories. For example, in the user account opening scenario, the association between the user ID node and the account opening application form node is marked as the core association, with a weight of 1.0. In the bill settlement scenario, the association between the account number node and the settlement details node is marked as the core association, ensuring that the model is deeply bound to the business scenario.

[0052] The subsequent training process for the node relationship model was carried out: 70% of the XML data in the sample set was selected as the training set, and the remaining 30% was evenly divided into the validation set and the test set according to business scenarios; the training objective was set to improve the accuracy of node relationship recognition and the accuracy of associated field matching. The training process adopted an iterative optimization mechanism: First, the model was trained for the first round using user account opening scenario samples from the training set. By comparing the association relationships of user ID nodes and qualification review nodes identified by the model with the standard relationships manually labeled, the association weight parameters were adjusted, and misjudged association types were corrected, such as correcting parallel relationships to containment relationships; then, training samples from scenarios such as fund transactions and bill settlement were used for iterative training in turn, each... After each training round, the model's relationship recognition accuracy is calculated. When the accuracy remains stable above 95% for three consecutive rounds, the model enters the validation phase. Validation set samples are used to test the model's adaptability in different scenarios. For example, in a bill settlement scenario, deeply nested tax detail sub-nodes are tested to verify whether the model can accurately identify their inclusion relationship with the main settlement node. If recognition deviations occur, training data for such complex nested samples is supplemented to optimize the model's recognition logic for deeply nested nodes. Finally, a test set sample is used for final validation to ensure that the model's node relationship recognition accuracy is no less than 93% and the associated field matching accuracy is no less than 94% on new XML samples that were not used in training, thus completing the training optimization.

[0053] After training, a hierarchical structure is used to visualize the node hierarchy paths and reference links. For example, in the model visualization diagram of a fund transaction scenario, the association path between the order information node and the payment node under the transaction root node, the flow link between the payment node and the account balance node, and the corresponding association weights are clearly marked. This ultimately forms a node relationship model that covers core node information, nested structure, reference relationships, business association verification results, and scenario adaptation parameters. The model is reviewed and confirmed by technical personnel in conjunction with specific business scenarios. For example, after confirming that the association logic of the order number node and transaction amount node identified by the model in the fund transaction scenario conforms to the business flow rules, it serves as the core basis for subsequent splitting and mapping.

[0054] Step 4.2: Based on the node relationship model, the data distribution pattern, and the field statistical characteristics, determine the rules for splitting and mapping XML data nodes. Specifically, this includes: first, determining the core design requirements of the target structured table, such as data storage specifications, field type constraints, and business logic adaptation requirements, to form a target table design reference standard; second, analyzing each XML node one by one in conjunction with the node relationship model: first, determining the node data type, such as text, numeric, or date; then, clarifying the business meaning corresponding to the node in conjunction with the business document; simultaneously, accurately matching the numeric or category identifier field attributes corresponding to the node data by referring to the field statistical characteristics, and establishing a list of corresponding nodes and field attributes; third, determining a reasonable splitting granularity based on the nesting hierarchy in the node relationship model: for data nodes with deep hierarchies and independent business meanings that can be reused independently, such as product detail points nested under order information in XML, splitting them into independent data units according to business logic; for nodes that are closely related and whose splitting would disrupt business integrity, preserving their combination relationship to avoid data fragmentation.

[0055] Based on the data distribution patterns and field statistical characteristics, detailed mapping rules were formulated between nodes and target structured table fields: Numerical node data was strictly mapped to numerical fields in the target table, adhering to the value range, precision requirements, and format standards in the numerical distribution characteristics; Classification identifier node data was mapped to the corresponding fields in the target table, ensuring that the node enumeration values ​​and enumeration distribution characteristics were completely consistent; Date node data was formatted uniformly according to the requirements of the target system; Mapping conflict handling rules were also formulated, clarifying the compliant conversion methods when field types did not match, the rules for distinguishing nodes with the same name but different meanings, and the handling scheme for exceeding data length limits, etc.; The technical team, business team, and target system team jointly reviewed the splitting and mapping rules, focusing on verifying the business logic adaptability of the rules and the compatibility with the target table. The rules were revised based on the issues raised in the review, and reviewed again after revision until the final splitting and mapping rule document was formed. The document included splitting granularity standards, node-field mapping list, conflict handling rules, and a summary of review comments.

[0056] Step 4.3: Based on the determined splitting and mapping rules, a multi-dimensional splitting method is used to perform hierarchical parsing and value extraction on the XML format data, generating atomic data units corresponding to the nodes in the node relationship model. Specifically, this includes: building an execution framework for the multi-dimensional splitting method based on the confirmed splitting and mapping rules, clarifying the splitting dimensions, including node hierarchy dimension, data type dimension, business scenario dimension, and the execution order of each dimension; first, starting the node hierarchy dimension splitting, from the XML root node, parsing nested nodes layer by layer according to the splitting rules, splitting nested nodes that meet the independent splitting conditions from their parent nodes to form multiple intermediate data units, each intermediate data unit is labeled with the corresponding parent node path and splitting basis to ensure that the splitting trajectory is traceable; for related nodes that do not need to be split, their combined structure is retained and marked as a whole as an intermediate data unit; then, data type dimension splitting is performed, classifying and extracting the data in each intermediate data unit, separating different types of data such as numerical, category identifier, and date data to avoid confusion between different types of data, and synchronously recording the node name and data type label corresponding to the data during the extraction process.

[0057] In the data value extraction stage, the extracted node data undergoes preliminary cleaning and verification: invalid characters, blank values, garbled characters, and other redundant information are removed; numerical data undergoes format standardization processing, such as unifying decimal precision and removing redundant symbols; and categorized identification data undergoes coding standardization verification. Each extracted standardized data unit is fully labeled with its corresponding node path, associated key field values, business scenario, and data source sample identifier to ensure the traceability and uniqueness of the atomic data unit. Through the above multi-dimensional decomposition, extraction, and standardization processing, the complex unstructured XML data is decomposed into atomic data units with uniform granularity, unified format, clear logic, and complete traceability information, forming an atomic data unit set. After the set undergoes preliminary quality verification to ensure there are no empty values ​​and the format is compliant, it is used for subsequent steps of reassembly and mapping.

[0058] Step 4.4: Based on the logical correspondence between atomic data units and the node relationship model, the atomic data units are reorganized and mapped to records in the target structured table, generating a record set. Specifically, this includes: confirming the complete structure of the target structured table according to the database design specifications of the target system and the mapping rules in Step 4.2, clarifying details such as the name, data type, length, NOT NULL constraints, and primary / foreign key relationships of each field, and generating a target table structure specification; comparing the node relationship model with the target table structure specification, clarifying the precise logical correspondence between each atomic data unit and the target structured table fields, establishing a mapping table between atomic data units and target table fields, and clarifying the target field positions that each atomic data unit should fill; grouping the atomic data unit set according to the associated key fields (such as order number, user ID), aggregating all atomic data units belonging to the same business object (such as the same order, the same user) into a business object data group, ensuring the complete aggregation of data for the same business object.

[0059] Based on the mapping table and business object data groups, data reorganization is carried out: Atomic data units from the business object data groups are populated into their corresponding target fields one by one, according to the field order of the target table. For target table records involving multiple node associations, such as order table records needing to be associated with user and product table data, atomic data units from different business object data groups are matched using association key fields and integrated into the corresponding fields of the target table. During the reorganization process, the populated data is simultaneously validated in real time using the statistical characteristics of the reference fields to ensure that numerical data conforms to the value range, category identifier data belongs to the enumeration set, and the format fully meets the requirements. After reorganization, a preliminary integrity check is performed on each target table record to verify whether required fields are missing and whether associated field values ​​match. Records that meet the requirements are compiled and summarized to form a structured record set, which is then stored according to business modules.

[0060] Step 4.5 involves performing data integrity verification on the target structured table based on the record set to confirm its logical consistency with the XML source data, ultimately completing the structured data conversion. This includes: first, establishing a multi-dimensional verification rule system based on the business logic of the black-box big data business system, field statistical characteristics, and the data quality requirements of the target system; integrity verification rules covering non-empty checks for each field in the target structured table, missing data checks for required fields, and compliance checks for field value lengths; consistency verification rules including consistency checks of the field mapping between atomic data units and target table records, and checks on associated field values ​​(such as order numbers and user IDs). Cross-table matching consistency verification, data format and field statistical feature consistency verification; accuracy verification rules determine the accuracy judgment criteria for the values ​​of core business fields (such as transaction amount), and specify the sample ratio and sampling method for sampling comparison; conduct comprehensive verification of the record set according to the verification rule system: first, perform batch integrity and consistency verification, count the number and type of abnormal records such as missing fields, format errors, and logical conflicts, and form a preliminary abnormality statistical report; then, extract records according to the preset sampling ratio, compare them field by field with the XML source data of the black box big data business system, verify the accuracy of the core data values, and record the sampling verification results.

[0061] For any anomalies discovered during verification, the causes are analyzed one by one: If the anomaly is due to unreasonable splitting or mapping rules, the process is traced back to step 4.2 to adjust the rules and re-execute the splitting and reorganization process from steps 4.3 to 4.4; if the anomaly is due to incomplete data extraction, the XML source data is retrieved again to supplement the extracted atomic data units and complete the reorganization; if the anomaly is due to errors in data standardization processing, the data format is corrected and the verification is performed again; after the anomaly handling is completed, a comprehensive verification is performed again until the anomaly rate is lower than the set compliance threshold and the sampling accuracy meets the requirements; finally, a final acceptance test is conducted to confirm that the target structured table data is completely consistent with the XML source data logic of the black-box big data business system and that the data quality meets the requirements of the target system. After the acceptance is passed, a data conversion completion report is generated, and the conversion of unstructured XML data to structured data is finally completed.

[0062] In a preferred embodiment of the present invention, step 5 above may include:

[0063] Step 5.1: Based on the confirmed migration scope, determine the set of source business tables to be included in the migration intermediate tables. This includes: initiating preliminary preparations by retrieving the final confirmed black-box big data business system data migration scope list from the project documents. The list contains core information such as table name, table type, business module, and migration status. Based on the list, select all business tables whose migration status is required to be migrated, and systematically review the core information of each table, including its business module, core business function, data volume, and association identifiers with other tables, forming a preliminary candidate list of source tables. Next, conduct a review, focusing on confirming whether all tables in the candidate list are necessary to support the operation of core businesses, whether there are any tables that need to be added or removed due to business adjustments, and verifying whether the table relationships are complete to ensure that no key related tables are omitted. After the review is completed, remove redundant tables that do not need to be included in the intermediate tables, such as temporary auxiliary tables and test tables, and supplement any missing necessary related tables. Finally, form a clear set of source business tables to be included in the migration intermediate tables, with each table's unique identifier, business module, core function, and related table information clearly marked.

[0064] Step 5.2, based on the constructed relationships, perform business logic relationship analysis on the source business table set to define the core business modules and data themes that the migration intermediate tables should support. Specifically, this includes: retrieving the constructed black-box big data business system business table relationship list, which includes table names, key related fields, relationship types, change sequences, etc.; using the source business table set as the analysis object, conduct business logic relationship analysis based on the relationship list: systematically review the business flow path of each source business table, clarify the relationship triggering scenarios between tables, the role of related fields (e.g., order number used to link the order table and payment table), and the flow direction of related data, and draw the relationship path diagram of the source business tables; based on the relationship path diagram and business module division criteria, group closely related business logic modules that belong to the same category... The source business tables of a business process are grouped. For example, user tables, account opening application forms, and qualification review forms related to user account opening are grouped together, while transaction tables, settlement tables, and account tables related to fund transactions are grouped together. Each group corresponds to a core business module. Based on the business attributes and data characteristics of each group, the core data themes under that module are extracted. For example, the data theme of the user account opening module is the entire user account opening process data, and the data theme of the fund transaction module is fund transaction and settlement data. The core data content that each data theme needs to cover is determined, such as transaction amount, transaction time, account balance, etc. Finally, a corresponding list of core business modules and data themes is formed, marking the source business tables, core data themes, and data coverage of each module. After review and confirmation, this list serves as the core guide for the design of intermediate tables.

[0065] Step 5.3: Based on the defined core business modules and data themes, and according to the transformed structured data, while also referring to field statistical characteristics, design the table structure of the migration intermediate table. The migration intermediate table structure covers all fields of the source business tables and meets the business relevance verification requirements. Specifically, this includes: first, determining the core design principles of the migration intermediate table: it must completely cover all fields in the source business table set to ensure no data omissions; the table structure must meet the business relevance verification requirements to facilitate subsequent verification of the consistency of data relationships between tables; field attributes must be compatible with the transformed structured data and also compatible with the target system's data storage specifications; combining the defined core business modules and data themes, designing the intermediate table according to the principle of one theme per intermediate table: each intermediate table corresponds to one core data theme, integrating all fields of the source business tables under that theme into the corresponding intermediate table, removing duplicate fields, such as keeping only one order number field common to multiple tables, and marking it as... Establish primary keys; refer to field statistical characteristics to determine the detailed attributes of each field: field types (e.g., numeric, character, date) refer to the attribute definitions in the field statistical characteristics; field length and precision are determined based on the value range and data length in the statistical characteristics. For example, the amount field is set to decimal type according to the precision requirements in the statistical characteristics, retaining two decimal places; field constraints (e.g., NOT NULL, QUESTION) are set in combination with business logic and the mandatory field analysis results in the statistical characteristics; at the same time, to meet the business relevance verification requirements, a dedicated relevance verification field is designed in the intermediate table, such as the relevance table identifier and data flow sequence identifier, to ensure that the relevance logic between the source business tables can be verified through the intermediate table; after the design is completed, a draft of the migration intermediate table structure is formed, including information such as table name, field name, field type, length, precision, constraints, corresponding source table and data theme; through field integrity verification and relevance adaptation verification, it is ensured that the draft meets the design principles.

[0066] Step 5.4: Based on the table structure design of the target system, establish mapping and splitting rules for the migration intermediate tables to one or more target tables in the target system. The target system's table structure can be a preliminary design version, iteratively adjusted during the migration process. Specifically, this includes: first, obtaining detailed documentation of the target system's table structure, which must include core information such as the target table name, field names, field types, length, precision, constraints, business module, primary and foreign key relationships, and data storage specifications; second, conducting a field-by-field comparison and analysis between the initial draft of the migration intermediate table structure and the target system's table structure documentation: clarifying the correspondence between intermediate table fields and target table fields, defining the mapping type, such as one-to-one mapping (one field in the intermediate table corresponds to one field in the target table); one-to-many mapping (one field in the intermediate table is split and mapped to multiple fields in the target table); and many-to-one mapping (multiple fields in the intermediate table are merged and mapped to one field in the target table); for intermediate tables that need to be split, such as one... Each intermediate table corresponds to multiple target tables. Based on the business module division and data theme of the target tables, detailed splitting rules are formulated: clearly defining which fields in the intermediate tables are split to which target tables, whether the attributes of the fields need to be adjusted after splitting, and how to maintain data integrity and relevance during the splitting process, such as synchronizing the split data through the primary key; simultaneously, mapping conflict handling rules are formulated to address issues such as mismatched field types, inconsistent field lengths, and different value ranges between the intermediate and target tables, determining compliant conversion solutions, such as adjusting the precision of numeric fields and converting the encoding of character fields, and marking the applicable scenarios of the rules; the mapping relationships, splitting rules, and conflict handling rules are compiled into a migration intermediate table to target table mapping and splitting rule document, and then an organizational review is conducted, focusing on verifying the business logic adaptability of the rules, the ability to ensure data integrity, and the compatibility with the target system, revising the rules based on the review comments, until the final version of the rule document is formed.

[0067] Step 5.5: Based on the mapping and splitting rules, verify and optimize the table structure of the designed migration intermediate table to ensure it can meet the requirements of accurate and efficient data migration from the black-box big data business system to the target system. This final step completes the design of the migration intermediate table, specifically including: building a simulated migration verification environment; configuring the environment to simulate a real production environment, including the black-box big data business system's data access specifications, the target system's data reception specifications, network bandwidth, etc.; extracting a small amount of real structured data from the black-box big data business system as verification samples and importing it into the initial draft of the migration intermediate table structure; performing simulated data migration according to the final version of the mapping and splitting rules: mapping and splitting the verification sample data in the intermediate table to the corresponding target table in the target system according to the rules; conducting multi-dimensional verification: data accuracy verification, comparing whether the core field values ​​of the target table data after migration are consistent with the source data; data integrity verification, checking whether the target table completely receives all the data split from the intermediate table, with no missing fields or data omissions; business relevance verification, checking whether the relationships between the target tables after migration are consistent with the intermediate table and whether they conform to business logic; and migration efficiency verification, statistically analyzing the time and data throughput of migrating from the intermediate table to the target table to assess whether it meets the requirements for efficient migration.

[0068] To address issues discovered during the verification process, such as distorted migration data due to unreasonable intermediate table field type design, data omissions due to incomplete splitting rules, and low migration efficiency, optimization plans were developed: if the issue was related to table structure, the types, lengths, or constraints of intermediate table fields were adjusted; if the issue was related to rules, the mapping or splitting rules were revised; if the issue was related to efficiency, the intermediate table index design was optimized and the data storage format was adjusted. After optimization, simulated migration verification was performed again, repeating the verification, optimization, and re-verification process until the accuracy, completeness, and relevance of the data migration met the requirements and the migration efficiency reached the standard. Finally, the optimized migration intermediate table structure, the final version of the mapping and splitting rules, the verification report, and other materials were compiled to form the final design draft of the migration intermediate table. After being signed and confirmed by all relevant teams in the project, the design work of the migration intermediate table was completed.

[0069] In a preferred embodiment of the present invention, step 6 above may include:

[0070] Step 6.1: Based on the final design of the migration intermediate table, identify the source fields from the black-box big data business system and the corresponding target fields in the target system. This includes: retrieving the final design draft of the migration intermediate table, the list of source business tables from the black-box big data business system, and the detailed table structure document of the target system to build a basic data set for field identification; using the fields of the migration intermediate table as the core, trace the origin of each field one by one: by comparing with the list of source business tables, clarify the source table name, source field name, and field attributes of the corresponding field in the black-box big data business system, and mark it as a source field; simultaneously, by comparing with the table structure document of the target system, clarify the target table name, target field name, and field attributes of the corresponding field in the target system, and mark it as a source field. For target fields, for newly added fields in the intermediate migration table (not directly from the source business table), separately label their field type (technical field or business field) and design purpose; during the process, record the correspondence information of each field in detail, including the intermediate table field name; complete information of the source field, including table name, field name, and attributes; complete information of the target field, including table name, field name, and attributes; field correspondence status, such as one-to-one, one-to-many, or no correspondence, and finally form a list of corresponding fields of the intermediate migration table, source fields, and target fields; organize technical personnel and business personnel to jointly review the list, verify the accuracy of field tracing and the completeness of the correspondence, correct tracing errors or omissions of field information, and ensure that the list can clearly and accurately reflect the relationship between the three types of fields.

[0071] Step 6.2: For common fields in the source and target fields that have the same meaning and name, compare and analyze the business meaning and usage rules of the common fields in the black-box big data business system and the target system to obtain the first type of field alignment results. Specifically, this includes: selecting common fields in the source and target fields that have the same meaning and name from the generated corresponding list to form a common field analysis list; for each common field in the list, retrieving relevant documents from the black-box big data business system and the target system respectively: extracting the business meaning description, usage scenarios, value range, field constraints, and data flow rules of the field from the business manual and data dictionary of the black-box big data business system; extracting the same dimension information of the corresponding field from the table structure design specifications and business specification documents of the target system; and forming a special analysis team to analyze the two types of common fields one by one. The system's field information is compared and analyzed: the focus is on verifying whether the business meanings are completely matched, such as whether each order number refers to a unique transaction order identifier; whether the usage scenarios are consistent; whether there is any overlap or difference in the value range; and whether the constraint rules and flow logic are compatible. If the analysis finds that the field information is completely consistent, it is directly recorded as aligned. If there are minor differences, such as slightly different value ranges that do not affect the core business, the reasons for the differences are analyzed, and compatibility handling rules are formulated, such as pruning data according to the value range of the target system. If there are significant differences, such as deviations in business meaning, the analysis is immediately suspended, and the business teams of the two systems are collaborated to reconfirm the field definitions to ensure a consistent understanding before continuing the comparison. Finally, the first type of field alignment results are formed, including a list of common fields, details of the comparative analysis of each field, alignment status, and compatibility handling rules.

[0072] Step 6.3: For unique and necessary technical fields in the target field, design and implement the value generation logic of the technical fields in the migration intermediate table according to the technical rules provided by the target system to obtain the second type of field alignment results. Specifically, this includes: filtering out fields that are unique and necessary technical fields in the target field from the corresponding list, such as primary key identifiers, data insertion timestamps, data source identifiers, etc. required by the target system, to form a technical field processing list; then obtaining the detailed technical rules of these technical fields, including field generation standards (e.g., primary keys must be 32 unique characters, timestamps must be accurate to milliseconds), data format requirements, value constraints, generation timing (e.g., automatically generated when data is inserted), and associated technical logic, such as primary key generation algorithms and timestamp acquisition methods; based on these technical rules, design the corresponding field value generation logic in the migration intermediate table: for automatically generated fields, such as primary keys, timestamps, etc. For intermediate tables, an embedded generation script is designed to ensure that field values ​​are automatically generated according to rules when data is written to the intermediate table. For technical fields that need to be deduced from source data, such as data source identifiers, association mapping logic is designed to extract key information from source fields or other fields in the intermediate table to generate corresponding values. The designed generation logic is deployed to the test environment of the migrated intermediate table, and a small amount of test data from a black-box big data business system is imported to verify the effectiveness of the generation logic: whether the technical field values ​​conform to the technical rules of the target system, whether the generation is stable and without anomalies, and whether it does not affect the data integrity of other fields in the intermediate table. For problems found during verification, such as incorrect generated value formats and generation failures, the generation logic is optimized and adjusted until all technical fields can accurately generate values ​​according to the rules. Finally, a second type of field alignment result is formed, which includes a list of unique technical fields, corresponding technical rules, generation logic design scheme, verification results, and optimization explanations.

[0073] Step 6.4: For unique and necessary business fields within the target field, collaborate with business personnel to supplement data or confirm mapping rules, obtaining the third type of field alignment results. Specifically, this includes: selecting unique and necessary business fields from the corresponding list, such as newly added business classification codes and risk level identifiers in the target system, forming a business field processing list; organizing joint discussions between the target system business team and the black-box big data business system business team to analyze each field in the list: determining the core business meaning of each business field; data requirements, such as value range, precision, and format; corresponding business scenarios and data source possibilities; and developing processing solutions based on the analysis results: if field data can be derived from existing source fields in the black-box big data business system, such as deriving the target field transaction level from the source field transaction amount, collaborating with technical personnel... Personnel confirm the derivation mapping rules, determine the derivation logic, judgment conditions, and value correspondence; if field data cannot be derived from existing source fields, business personnel need to supplement the data, and detailed supplementation specifications are formulated, including the source of supplemented data (such as business ledgers, manual review results), supplementation format requirements, supplementation responsibility, supplementation time nodes, and quality verification standards; for the mapping rule scheme, the accuracy of the rules is verified in the test environment using sample data; for the supplementation scheme, business personnel are organized to conduct small-scale trial supplementation to optimize the supplementation process and specifications; after the scheme is verified, the mapping configuration or data supplementation work is officially executed to ensure that each unique business field can obtain valid data that meets the requirements; finally, a third type of field alignment result is formed, including a list of unique business fields, processing solutions, mapping rules or supplementation specifications, verification results, and data acquisition status.

[0074] Step 6.5: For fields unique to the source fields but missing in the target fields, collaborate with business and development teams to confirm whether the missing fields need to be added to the target system, and formulate a supplementation plan to obtain the fourth type of field alignment results. Specifically, this includes: selecting fields unique to the source fields but missing in the target fields from the corresponding list to form a missing field review list; organizing a special review meeting with the black-box big data business system team, the target system business team, and the development team to assess the business necessity of these missing fields: analyzing the core business functions of each field; the usage scenarios in the black-box big data business system; the data value, such as whether it is necessary for business traceability and statistical analysis; the potential business impact of the missing fields, such as whether it leads to business process disruptions or incomplete data; and simultaneously assessing the feasibility of adding the fields to the target system: including the difficulty of adjusting the target system's table structure and system compatibility. The review process considers factors such as compatibility impact, development cycle, resource investment, and the degree of disruption to existing business. Based on the review results, decisions are made: if a field is business-essential and addition is feasible, a detailed supplementary plan is developed, specifying the tables and fields to be added to the target system (field names, types, constraints), table structure adjustment process, development schedule, data migration synchronization strategy, and testing and verification plan. If a field is not core-essential or addition is not feasible, the reasons for the decision are explained in detail, and the data handling method for that field is determined, such as archiving or not migrating. The review decisions and supplementary plans are compiled into review minutes, which are then signed and confirmed by the participating teams. For fields requiring supplementation, preliminary preparations for adjusting the target system's table structure are initiated simultaneously. For fields not requiring supplementation, data archiving records are maintained. Finally, a fourth type of field alignment result is formed, including a list of missing fields, review details for each field, decision conclusions, supplementary plans, and reasons for not supplementing.

[0075] Step 6.6 integrates the first, second, third, and fourth type of field alignment results to generate a complete field alignment result. This includes: building an integration framework for the field alignment results, layering the framework according to field type (common fields, target-specific technical fields, target-specific business fields, and source-specific missing fields), with each layer containing core elements such as basic field information, alignment processing method, processing result, and verification status; incorporating the first, second, third, and fourth type results into the framework according to their field type to ensure that the alignment information for each field accurately matches the corresponding level; conducting a completeness check on the integrated information: confirming that all fields selected from the list have been included in the integration result without omission; and conducting a... Consistency Verification: Verify whether there are conflicts in the processing rules of different categories of fields, whether the terminology is consistent, and whether the decision-making logic is coherent; for any omissions or conflicts found during the verification, backtrack to the corresponding steps, correct them, and then re-integrate; after integration, generate a summary table of field alignment results between the black-box big data business system and the target system. The summary table clearly presents the complete alignment of all fields, including field correspondence, alignment status, processing rules, decision basis, and verification results; organize the project business team, technical team, and target system development team to conduct a final review, fully confirm the contents of the summary table, propose revisions and improve it, and finally generate a complete field alignment result document, including the summary table, details of each category of results, review minutes, and subsequent execution suggestions, serving as the core guiding document for data migration execution.

[0076] In a preferred embodiment of the present invention, step 7 above may include:

[0077] Step 7.1: Based on the final design of the migration intermediate table and the complete field alignment results, formulate a full-cycle verification strategy encompassing pre-migration, during-migration, and post-migration phases. Specifically, this includes: retrieving the final design draft of the migration intermediate table and the complete field alignment result document; combining the business specifications of the black-box big data business system and the data quality requirements of the target system; and forming a verification strategy formulation team composed of the technical migration team, business team, operations and maintenance team, and the target system team. The core objective of the full-cycle verification is to ensure the accuracy and completeness of the data migration. Based on this, the verification focus is broken down into three phases: pre-migration, during-migration, and post-migration. Pre-migration focuses on the readiness of basic conditions; during-migration focuses on the controllability of the data flow process; and post-migration focuses on the validity of the target data. For each phase, detailed verification strategies are formulated. The verification content includes: before migration, infrastructure status, consistency of system table structure, and validity of migration program version; during migration, comparison of record counts at each stage of data flow and real-time verification of business rules; after migration, verification of total data volume, multi-dimensional summary verification, and accounting logic verification. The verification criteria for each item are clearly defined, namely, a consistency error of less than 0.01%; the execution method, combining manual sampling verification with automated verification; the responsible parties, completion deadlines, and anomaly handling procedures, including rectification requirements and review processes for failed verifications. The verification content, standards, and methods at each stage are integrated to form a draft of the full-cycle verification strategy. A development team conducts multiple rounds of reviews to optimize the rationality of verification items and the coherence of processes. Once approved, a formal full-cycle data migration verification strategy document is generated.

[0078] Step 7.2: According to the full-cycle verification strategy, check the infrastructure involved in the migration, the consistency of table structures between the black-box big data business system and the target system, and the version of the migration program to ensure that the migration start conditions are met and generate pre-migration verification results. Specifically, this includes: initiating pre-migration verification work according to the established full-cycle verification strategy; firstly, conducting infrastructure verification, checking the hardware status of the data source server of the black-box big data business system, the migration intermediate table deployment server, and the target system receiving server, i.e., CPU, memory, and disk space utilization, verifying the stability of the network link and the bandwidth compliance, and checking the read / write permissions and response speed of the data storage nodes to ensure that all infrastructure can meet the resource requirements of the migration process; then, performing table structure consistency verification, based on the complete field alignment results and the final design of the migration intermediate table, checking the source table of the black-box big data business system, the migration intermediate table, and the target table of the target system one by one. The process involves comparing field information, including field name, data type, field length, precision, constraints (e.g., NOT NULL, Unicode), and primary / foreign key relationships), with a focus on verifying whether the explicit mapping relationships in the field alignment results are accurately reflected in the table structure. Inconsistencies are recorded and their causes analyzed. Next, a migration program version check is performed to verify that the current version is a tested and verified stable version. The program release record and test report are verified to confirm that known vulnerabilities have been fixed and that the program is compatible with the interface specifications of the black-box big data business system and the target system. After completing all checks, the results are summarized to determine if the migration initiation conditions are met: no major infrastructure vulnerabilities, table structure consistency, and a valid migration program version. Rectification plans are developed for any issues, and the completion status is tracked. Finally, a pre-migration check result report is generated, including a check item list, details of each check item, rectification records, and the conclusion of the initiation condition determination.

[0079] Step 7.3, based on the complete field alignment results and the design of the migration intermediate table, performs phased record count comparisons and business rule verifications during the data flow from the black-box big data business system to the target system via the migration intermediate table, generating migration check results. Specifically, this includes: establishing a real-time monitoring mechanism for migration checks according to the full-cycle check strategy and complete field alignment results; identifying three core check nodes for data flow: after data is exported from the black-box big data business system to the migration intermediate table; after data processing in the migration intermediate table is completed; and during the import of data from the migration intermediate table into the target system. At each check node, a record count comparison is performed first, counting the output record count of the previous stage with the received record count of the current stage to ensure complete consistency. If a record count discrepancy occurs, data flow is immediately paused, and abnormal logs during data export, transmission, or import are checked to pinpoint the cause of the discrepancy, such as data loss due to network interruption or accidental deletion of data by filtering rules. Once the data flow is repaired and the record count is confirmed to be consistent, the data flow can continue. Simultaneously, business rule verification is performed. Based on the business rule requirements and field statistical characteristics in the field alignment results, the data flow of the current stage is sampled and verified. This includes the compliance of the value range of numeric fields, the consistency of the enumerated values ​​of category identifier fields, and the logical matching of related fields, such as the correspondence between order numbers and user IDs. The sampling ratio is set according to the verification strategy requirements to ensure that the sample covers all core business modules. During the verification process, the verification time, record count comparison results, sampling verification details, and anomaly handling status of each node are recorded in real time, forming a migration verification log. After the entire data flow process is completed, the verification logs of each node are summarized, the frequency and type of anomalies are analyzed, the stability of the data flow process is assessed, and finally, a migration verification result report is generated, including details of each verification node, a summary of record count comparisons, business rule verification results, anomaly handling records, and a process stability assessment.

[0080] Step 7.4: After the data is completely migrated to the target system, based on the complete field alignment results, perform total quantity verification, multi-dimensional summary verification, and accounting verification on the migrated data in the target system to generate post-migration verification results. Specifically, after the data is completely migrated to the target system, based on the full-cycle verification strategy and the complete field alignment results, initiate a comprehensive post-migration verification. First, conduct a total quantity verification, counting the total number of records in the source table of the black-box big data business system, the number of processed records in the migration intermediate table, and the final number of migrated records in the target table of the target system. Compare the total quantities of the three parties to ensure complete consistency. If there are discrepancies in the total quantities, trace the specific range of the discrepancies through the data migration log, analyze the links that caused the discrepancies, i.e., exporting, processing, importing, and completing data supplementation or correction. Next, perform multi-dimensional summary verification, according to business modules, such as user management and fund transactions; time periods, such as daily and monthly; and business types, such as account opening and settlement. The process involves summarizing and statistically analyzing the source data from the black-box big data business system and the migrated data from the target system. This includes comparing the summative values ​​of each dimension, such as total transaction amount, total number of users, and total number of business transactions, to ensure that the deviation in the summative results is within the allowable range of the verification strategy. If the deviation exceeds the threshold, the dimensions are broken down for fine-grained verification to pinpoint the specific deviation fields and data ranges. Finally, accounting reconciliation is performed. For fund-related business modules, the accounting logic of the migrated data in the target system is verified according to the accounting processing rules of the black-box big data business system. This includes checking the loan balance, the accuracy of account balances, and the consistency between transaction records and accounting records, ensuring that the migrated accounting data is free of logical errors. After completing all verification work, the verification results are summarized, unresolved anomalies are identified and rectification suggestions are formulated, and a post-migration verification result report is generated. This report includes details of the total verification, multi-dimensional summary comparison data, accounting reconciliation results, a list of anomalies, and rectification suggestions.

[0081] Step 7.5 integrates the pre-migration, during-migration, and post-migration verification results to generate a comprehensive verification report reflecting the accuracy and completeness of the data migration. This includes: establishing a verification result integration framework, divided into pre-migration, during-migration, and post-migration phases. Each phase includes core elements such as verification items, verification standards, verification results, anomaly handling records, and rectification completion status, ensuring a clear and logically coherent integrated information structure; integrating the pre-migration, during-migration, and post-migration verification results one by one according to the framework dimensions, focusing on linking anomalies at each phase to form a complete record of problem discovery, rectification, and review; and conducting a completeness and consistency check after integration to confirm that all verification items explicitly defined in the verification strategy have been included in the integration results without omission, and verifying the verification results at different stages. To ensure consistency in the description of the same problem and the matching of rectification measures and results, logical contradictions should be avoided. Based on the integrated results, a comprehensive analysis of the accuracy of data migration should be conducted, including the accuracy of field mapping, the adaptability and completeness of business rules (e.g., no missing records and complete field coverage). The overall compliance status of the migration work should be clarified, and risk levels and subsequent processing deadlines should be marked for any incomplete rectification issues. Finally, a full-cycle data migration verification report should be generated, including an overview of the verification work, a summary of verification results at each stage, an assessment conclusion on the accuracy and completeness of the migration, tracking records of abnormal issues and rectification, and suggestions for subsequent data use. The verification report should be jointly reviewed by all relevant project teams to confirm its accuracy. After the review is passed, the verification report will serve as the core basis for the acceptance of the data migration work and will be archived for future reference.

[0082] In a preferred embodiment of the present invention, step 8 above may include:

[0083] Step 8.1: In the application development process of the target system, based on the final design of the migration intermediate table, the complete field alignment results, and the verification report, formulate an integration strategy that aligns with the application development process of the target system. This includes retrieving the final design draft of the migration intermediate table, the complete field alignment result document, and the full-cycle data migration verification report. Based on the established application development process of the target system—requirements analysis, design and development, testing, and deployment / switching—build an integration strategy framework. The core objective of the integration strategy is to achieve seamless collaboration between the migrated data and the target system's application development, ensuring that the migrated data accurately supports the implementation of the target system's functions. Based on this, break down the integration priorities and collaboration nodes for each development stage, and determine the migration-related deliverables required for each stage. For example, field alignment results, key conclusions of the inspection report, and collaborative deliverables such as requirement verification specifications and design alignment records; construct specific collaborative mechanisms, including regular cross-process synchronization nodes such as weekly development progress and data integration synchronization nodes, document sharing and version control rules, and closed-loop processes for issue feedback and rectification; simultaneously set integration verification standards for each stage, such as verifying the compatibility between functional requirements and migration data specifications in the requirements stage, and confirming the consistency between table structure design and field alignment results in the development stage; integrate the inspection content, standards, and methods of each stage to form an initial draft of the integration strategy, optimize the rationality of collaborative nodes and the feasibility of verification standards through multiple rounds of process-oriented reviews, and form a formal integration strategy document for migration data and target system application development after the review is passed.

[0084] Step 8.2, during the requirements analysis phase of the target system, based on the final design of the migration intermediate table and the complete field alignment results, provides input and verification basis for the functional requirements and data model design of the target system. Specifically, this includes: during the requirements analysis phase of the target system, outputting the final design of the migration intermediate table and the complete field alignment results document to the requirements analysis process, synchronously migrating the core information of the data: including the source range of the migration data, field types and constraints, inter-table relationships, data specification requirements such as value range, format standards, and the data quality status confirmed in the verification report; embedding the migration data specifications into the functional requirements design process of the target system, focusing on completing the functional requirements and... Data migration compatibility verification: For example, verifying whether the query function planned by the target system covers the field dimensions of the migrated data, and whether the business process design matches the flow logic of the migrated data, to avoid situations where functional requirements depend on unmigrated data or conflict with the specifications of the migrated data; for issues where the migrated data does not match the functional requirements discovered during the requirements analysis process, such as the requirement to add a field but it is missing in the migrated data, initiating a cross-process analysis mechanism to output solutions, such as supplementing data migration or adjusting the requirement description; finally, generating a data integration verification specification for the target system requirements analysis phase, determining the scope of support of the migrated data for functional requirements, the requirements verification conclusions, and a list of issues that need to be optimized.

[0085] Step 8.3: During the design and development phase of the target system, based on the complete field alignment results and verification report, collaboratively complete the alignment of the target system's table structure, business logic, and migration data specifications. Specifically, this includes: During the design and development phase of the target system, using the complete field alignment results and the full-cycle data migration verification report as the core basis, conduct the alignment process for the table structure and business logic. First, perform table structure alignment: Compare the field alignment results with the designed table structure of the target system and the field specifications of the migration data, confirming that the field names, data types, lengths, precision, constraints, and primary / foreign key relationships are consistent with the migration data requirements. If the table structure needs to be adjusted during the design process, simultaneously verify the impact of the adjustment scheme on the migration data access to avoid disrupting field mapping. Next, align the business logic: In the target system's business logic coding process, verify the compatibility between the business logic processing rules and the migrated data. For example, check whether the data validation logic conforms to the value specifications of the migrated data, and whether the data calculation logic matches the statistical characteristics of the migrated data. At the same time, import the rules for handling data anomalies from the inspection report and build anomaly handling logic that is compatible with the characteristics of the migrated data. Establish phased alignment review nodes. After the design and development of each functional module is completed, trigger the alignment review process to check the consistency between the table structure design, business logic coding, and the migration data specifications. Record the alignment deviations and generate rectification plans to form a data alignment record for the target system design and development phase, ensuring that the target system's table structure and business logic can accurately match the migration data specifications.

[0086] Step 8.4, during the testing phase of the target system, based on the final design and verification report of the migration intermediate table, supports functional testing, data verification, and dual-system comparison testing of the target system. Specifically, this includes: outputting the final design draft of the migration intermediate table and a full-cycle data migration verification report to the testing process during the target system's testing phase; supporting the construction of the testing environment and test cases, ensuring that test cases closely match the actual situation of the migrated data; supporting the functional testing process: providing high-quality migration test data that has been verified and confirmed, ensuring that test cases cover the core fields and business scenarios of the migrated data, and verifying the accuracy of the target system's processing of the migrated data, such as whether data display, query, and statistical functions are normal; and supporting data verification testing: generating data verification test cases based on field alignment results and verification standards, focusing on verifying the target system's data. The system ensures the integrity of migrated data (no missing fields), accuracy (values ​​consistent with source data), and consistency (correct inter-table relationships). In conjunction with dual-system comparison testing, a parallel operating environment is built for the black-box big data business system and the target system. Comparison rules are established, such as selecting core business scenarios and key indicators for comparison. The system outputs migration data flow information, assists in analyzing differences during the comparison process (e.g., inconsistent data display, deviations in business results), determines whether the differences stem from system functional design or data migration issues, and provides data-level troubleshooting basis. During testing, real-time responses are provided to feedback on testing processes, resolving testing challenges related to migration data. Finally, a migration data support report for the target system testing phase is generated, recording test support details, data verification results, and dual-system comparison difference analysis conclusions.

[0087] Step 8.5: During the target system's go-live switchover phase, based on the verification report and aligned migration data, support the target system in completing the business takeover of the migrated data, achieving a smooth switchover between the black-box big data business system and the target system. Specifically, this includes: during the target system's go-live switchover phase, using the full-cycle data migration verification report and aligned migration data as core support, constructing a detailed go-live switchover collaboration mechanism; before the switchover, completing the final readiness verification of the migration data: re-verifying the total amount of migration data in the target system, the integrity of core fields, and the accuracy of key business data to confirm that the migration data status meets business takeover requirements; simultaneously completing the configuration verification of the switchover environment to ensure that the target system can normally read the migration data; during the switchover process, monitoring the business operations of the migrated data according to the switchover order determined by the collaboration mechanism, such as non-core business modules first, then core business modules. Service Acceptance Status: Real-time tracking of the target system's modules' access to the migrated data to confirm that business processes can operate normally based on the migrated data, such as order placement, account inquiry, and bill generation. If data access anomalies or business interruptions occur, an investigation mechanism is immediately triggered, and a pre-set rollback plan is initiated if necessary. After the switchover is completed, post-launch business verification is conducted: Select core business scenarios for practical verification, check the matching of business results with the migrated data, and collect feedback information on the support of the migrated data for business operation. At the same time, monitor the storage performance and access response speed of the migrated data to ensure that the data can stably support the long-term operation of the target system. Finally, generate a business acceptance verification report for the migration data during the target system's launch and switchover phase, recording details of the switchover process, business acceptance verification results, and problem handling records to ensure a smooth switchover between the black-box big data business system and the target system.

[0088] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data migration method for black-box big data business systems, characterized in that, The method includes: Based on the database layer table activity information of the black-box big data business system, the tables in the black-box big data business system are classified and their migration scope is confirmed, including: Query the database system tables of the black-box big data business system to obtain the last data update time of all tables; Based on the last data update time, tables with no data changes within five years are classified as static tables, tables with no data changes within one year are classified as cold tables, and tables with data changes within one year are classified as hot tables. Perform parameter table identification and update checks on static tables, and identify non-parameter tables that are not updated as non-migration tables; Perform offline status analysis on cold meters, remove offline meters, and confirm the necessity of migrating for offline cold meters; The migration of heat meters was confirmed by reviewing each one individually; the migration scope was finally confirmed by combining the confirmation results of static meters, cold meters, and heat meters. Deploy triggers at the database layer of the black-box big data business system to track real-time database changes caused by transactions and obtain tracking results. Based on the tracking results, analyze the relationships between business tables and data distribution patterns in the black-box big data business system, including: In the database layer of the black-box big data business system, triggers are created for the business tables to track data changes; The data operation records on the business table are captured by triggers to form the original tracking log; The original tracking logs are parsed and cleaned to extract structured tracking records containing operation type, table name, key field values ​​and operation timestamps; Based on structured tracking records, multiple business tables that have undergone collaborative changes in business transactions are identified by performing correlation analysis according to time series and transaction identifiers, so as to construct the correlation relationship between the business tables; Analyze the frequency and distribution of field value changes in structured tracking records, and combine the constructed relationships to statistically analyze the data distribution patterns under different business scenarios; Based on the data distribution patterns, statistical analysis was performed on the business table fields of the black-box big data business system to obtain the field statistical characteristics; Based on the aforementioned relationships and data distribution patterns, and with reference to field statistical characteristics, unstructured data in XML format in the black-box big data business system is converted into structured data. Based on the confirmed migration scope, the aforementioned relationships, and the transformed structured data, design a migration intermediate table connecting the black-box big data business system and the target system; By designing an intermediate migration table, the meaning and usage of fields between the black-box big data business system and the target system are aligned to obtain the field alignment results. Based on the design of the intermediate migration table and field alignment results, perform full-cycle data verification from the black-box big data business system to the target system and generate a verification report. In the target system refactoring project, the design of the migration intermediate table, field alignment results, and verification reports will be coordinated with the application development process of the target system to support the functional testing, data verification, and system switchover of the target system.

2. The data migration method for black-box big data business systems according to claim 1, characterized in that, Based on the data distribution patterns, statistical analysis was performed on the business table fields of the black-box big data business system to obtain the field statistical characteristics, including: Based on the data distribution patterns obtained through statistical analysis, the numerical fields and category identifier fields in the business table are identified. For the identified numerical fields, analyze the numerical range, data density, and trend of change of the numerical fields to form numerical distribution characteristics; For the identified classification identifier fields, analyze the set of values, frequency of occurrence and stability of the classification identifier fields to form an enumerated distribution feature; Based on the data distribution patterns in different business scenarios, the numerical distribution characteristics and enumerated distribution characteristics are verified and corrected in a scenario-based manner, and scenario-based verification results are generated. Integrate numerical distribution features, enumerated distribution features, and scenario-based validation results to generate field statistical features.

3. The data migration method for black-box big data business systems according to claim 2, characterized in that, Based on the aforementioned relationships and data distribution patterns, and referring to field statistical characteristics, the unstructured data in XML format in the black-box big data business system is converted into structured data, including: Based on the constructed association, the nesting hierarchy and reference relationships of data nodes in XML format unstructured data are parsed to form a node relationship model; Based on the node relationship model, the data distribution pattern, and the field statistical characteristics, rules for splitting and mapping XML data nodes are determined; Based on the established rules for splitting and mapping, a multidimensional splitting method is used to perform hierarchical parsing and value extraction on XML format data, generating atomic data units corresponding to nodes in the node relationship model; Based on the logical correspondence between atomic data units and node relationship models, atomic data units are reorganized and mapped to records in the target structured table to generate a record set; Based on the record set, the target structured table is subjected to data integrity verification to confirm its logical consistency with the XML source data, and finally the transformation of structured data is completed.

4. The data migration method for black-box big data business systems according to claim 3, characterized in that, Based on the confirmed migration scope, including the static parameter table, and the cold and hot tables confirmed for migration; The structured data relating to the relationships and transformations is used to design an intermediate migration table connecting the black-box big data business system and the target system, including: Based on the confirmed migration scope, determine the set of source business tables that need to be included in the migration intermediate table; Based on the constructed relationships, a business logic relationship analysis is performed on the source business table set to define the core business modules and data themes that the migration intermediate table should support; Based on the defined core business modules and data themes, and according to the transformed structured data, while also referring to the statistical characteristics of the fields, the table structure of the migration intermediate table is designed. The table structure of the migration intermediate table covers all the fields of the source business table and meets the business relevance check requirements. Based on the table structure design of the target system, establish mapping and splitting rules for migrating intermediate tables to one or more target tables in the target system. The table structure of the target system can be a preliminary design version and is iteratively adjusted during the migration process. Based on the mapping and splitting rules, the table structure of the designed migration intermediate table is verified and optimized to ensure that it can meet the requirements of accurate and efficient data migration from the black-box big data business system to the target system, and finally complete the design of the migration intermediate table.

5. The data migration method for a black-box big data business system according to claim 4, characterized in that, By designing an intermediate migration table, the meaning and usage of fields between the black-box big data business system and the target system are aligned to obtain the field alignment results, including: Based on the final design of the migration intermediate table, identify the source fields from the black-box big data business system and the target fields corresponding to the target system in the migration intermediate table; For common fields in the source and target fields that have the same meaning and name, the business meaning and usage rules of the common fields in the black-box big data business system and the target system are compared and analyzed to obtain the first type of field alignment results; For the unique and necessary technical fields in the target fields, the logic for generating the values ​​of the technical fields is designed and implemented in the migration intermediate table according to the technical rules provided by the target system, so as to obtain the second type of field alignment results; For unique and necessary business fields in the target fields, collaborate with business personnel to supplement data or confirm mapping rules to obtain the third type of field alignment results; For fields that are unique to the source fields but missing in the target fields, the business and development teams confirm whether the missing fields need to be added to the target system and formulate a supplementation plan to obtain the fourth type of field alignment results. Integrate the first type of field alignment results, the second type of field alignment results, the third type of field alignment results, and the fourth type of field alignment results to generate a complete field alignment result.

6. The data migration method for a black-box big data business system according to claim 5, characterized in that, Based on the design of the intermediate migration table and field alignment results, perform a full-cycle data check from the black-box big data business system to the target system, and generate a check report, including: Based on the final design of the migration intermediate table and the complete field alignment results, formulate a full-cycle verification strategy that includes pre-migration, during migration, and post-migration. According to the full-cycle verification strategy, check the infrastructure involved in the migration, the consistency of table structure between the black-box big data business system and the target system, and the version of the migration program to ensure that the migration start conditions are met and generate pre-migration verification results. Based on the complete field alignment results and the design of the migration intermediate table, during the process of data flowing from the black-box big data business system to the target system through the migration intermediate table, staged record count comparison and business rule verification are performed to generate migration check results. After the data has been completely migrated into the target system, based on the complete field alignment results, the migrated data in the target system is checked for total amount, multi-dimensional summary and verification, and accounting verification, and the post-migration verification results are generated. Integrate the pre-migration check results, the during-migration check results, and the post-migration check results to generate a check report that comprehensively reflects the accuracy and integrity of the data migration.

7. The data migration method for a black-box big data business system according to claim 6, characterized in that, In the target system refactoring project, the design of the migration intermediate table, field alignment results, and verification reports will be coordinated with the target system's application development process to support functional testing, data verification, and system switchover, including: In the application development process of the target system, based on the design of the final migration intermediate table, the complete field alignment results, and the check report, an integration strategy that is coordinated with the application development process of the target system is formulated. During the requirements analysis phase of the target system, the design of the final migration intermediate table and the complete field alignment results provide input and verification basis for the functional requirements and data model design of the target system. During the design and development phase of the target system, based on the complete field alignment results and verification reports, we collaboratively complete the alignment of the target system's table structure, business logic, and migration data specifications. During the testing phase of the target system, based on the final design and verification report of the migration intermediate table, support the functional testing, data verification, and dual-system comparison testing of the target system. During the go-live switchover phase of the target system, based on the verification report and the aligned migration data, the target system is supported in completing the business takeover of the migration data, thus achieving a smooth switchover between the black-box big data business system and the target system.

Images