Data migration method and system

Abstract

The application discloses a data migration method and system, obtains a data update message in an original database, parses the data update message to obtain to-be-migrated data and an original data table identifier of the to-be-migrated data; a master data synchronization service is used to execute a delegate class method to migrate the to-be-migrated data from the original database to a target database; the delegate class method comprises the following steps: a corresponding target synchronization processing container is matched from a plurality of pre-constructed synchronization processing containers according to the original data table identifier; a subclass executor which is bound with the original data table in the target synchronization processing container is obtained, and the subclass executor is called to process the to-be-migrated data according to preset data processing logic; and the processed to-be-migrated data is written into a target data table of the target database. The application matches a corresponding subclass executor for the to-be-migrated data to perform data processing, does not need to separately develop custom logic for each migration task, realizes automatic data migration between heterogeneous table structures, improves migration efficiency, and reduces development cost.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, specifically to data migration methods and systems. Background Technology

[0002] Data migration typically involves two parts: First, replacing database components. Since the functionalities of mainstream domestic database products are relatively mature and well-defined, this part is relatively easy to implement and can be completed by directly selecting a suitable database product based on specific business needs. Second, refactoring unreasonable table structures. When the original data system is complex, has a large data volume, and unclear storage logic, the difficulty of refactoring the data table structure increases significantly. In particular, if the system does not use an expansion table mechanism for business expansion, a large number of temporary appended fields will be generated, further exacerbating the chaos of the table structure. Targeted optimization is required during refactoring, such as merging multiple vehicle basic information tables into a single data table, or splitting real-name authentication-related status information into a dedicated table to avoid multiple tables repeatedly maintaining the same status data.

[0003] In existing technologies, all data is stored in a single table. As business expands, new fields are continuously added to the existing table to meet new business needs, causing these new fields to affect all historical data. With the continuous growth of data volume, data access paths become increasingly complex, leading to decreased business access performance or even interruptions. To improve business performance issues caused by the unreasonable design of the original table structure, the table structure needs to be redesigned and the data migrated from the original table to the newly constructed table. However, existing data processing servers only support data migration between tables with the same structure. For data migration between different table structures, custom data migration logic needs to be added for each migration task, resulting in low migration efficiency, high development costs, and difficulty in effectively ensuring data integrity and business compatibility. Summary of the Invention

[0004] This invention provides a data migration method and system to solve the problems of related technologies being unable to adapt to different table structures during data migration, resulting in low migration efficiency, high development costs, and difficulty in effectively ensuring data integrity and business compatibility.

[0005] In a first aspect, the present invention provides a data migration method applied to a data migration system, the method comprising: Obtain the data update message from the original database, parse the data update message to obtain the data to be migrated and the original data table identifier of the data to be migrated; The delegated methods are executed through the master data synchronization service to migrate the data to be migrated from the original database to the target database; wherein the delegated methods include: Based on the original data table identifier, a corresponding target synchronous processing container is matched from a plurality of pre-built synchronous processing containers, and each synchronous processing container has a corresponding original data table; Obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to the preset data processing logic; The processed data to be migrated is written into the target data table of the target database.

[0006] This invention introduces a delegated method for master data synchronization service execution. It dynamically matches a pre-built synchronization processing container based on the original data table identifier and retrieves the bound subclass executor from the container to process the data. This decouples the data transformation logic from the main migration process, improving the system's scalability and maintainability. Through the two-tier architecture of containers and executors, adding a new data table only requires adding the corresponding subclass executor, without modifying the core code. Simultaneously, by parsing the original data table identifier from the data update message and quickly matching the corresponding processing container, it achieves efficient migration of real-time incremental data, avoiding the risk of business interruption caused by table structure changes.

[0007] In one optional implementation, obtaining the data update message from the original database includes: By responding to the data update operation in the original database through the cloud server, the data corresponding to the data update operation is serialized into a data object, and the data object is written into the cloud middleware that communicates with the cloud server. The data processing server subscribes to and retrieves the data object from the cloud middleware, and uses the data object as the data update message.

[0008] This invention decouples data producers from consumers by introducing cloud middleware. At the same time, it effectively prevents data loss or duplicate consumption during transmission by leveraging the persistence features and offset mechanism of message queues, providing reliable guarantees for subsequent data integrity verification and breakpoint resume.

[0009] In one optional implementation, parsing the data update message to obtain the data to be migrated and the original data table identifier of the data to be migrated includes: Parse the data update message to obtain the operation type, the target data to be processed, and the original data table name of the target data corresponding to the data update message; The target data is used as the data to be migrated, and the original data table name is used as the original data table identifier of the data to be migrated.

[0010] The data processing server of this invention accurately obtains the operation type, target data to be processed, original data table name, and optional auxiliary information from the original data object, providing complete and accurate input parameters for subsequent synchronous processing container matching, subclass executor invocation, and data writing operations, ensuring the accuracy and reliability of the data migration process.

[0011] In one optional implementation, matching the corresponding target synchronization processing container from a pre-built plurality of synchronization processing containers based on the original data table identifier includes: Access a preset global context variable, which stores multiple key-value pairs. Each key-value pair uses the original data table identifier as the key and the synchronization processing container corresponding to the original data table identifier as the value. Using the parsed original data table identifier as the key, the target synchronization processing container corresponding to the original data table identifier is searched and determined from the global context variable.

[0012] Through the matching mechanism based on global context variables described above, this embodiment achieves fast mapping and efficient lookup between the original data table identifier and the synchronization processing container. The data processing server only needs a hash lookup operation to obtain the target synchronization processing container, without traversing all containers or executing complex matching logic, which greatly improves the efficiency of data migration processing.

[0013] In one alternative implementation, the pre-construction method for the synchronization processing container is as follows: During the initialization of the data migration system, all classes in the data migration system marked with specific annotations are scanned to identify the subclass executors; Obtain the original data table identifier and original table data type corresponding to the subclass executor; Based on the original data table identifier and the original table data type, a synchronization processing container corresponding to the subclass executor is created, and the subclass executor is declared in the synchronization processing container; The created synchronization processing container is stored in the global context variable using the original data table identifier as the key.

[0014] This invention achieves automated identification and registration of subclass executors by automatically scanning classes with specific annotations during system initialization. This method eliminates the need for manual configuration of mapping relationships, reducing system deployment and maintenance costs. By creating a corresponding synchronization processing container for each subclass executor and encapsulating the original table data types, it provides complete type information support for subsequent data migration. Storing the container in a global context variable with the original data table identifier as the key enables rapid dynamic matching between executors and the original table, improving lookup efficiency during data migration. The pre-built mechanism of the synchronization processing container allows for adding annotated subclass executors when adding new data tables, without modifying core code or configuration files, effectively supporting rapid iteration of data table structures in complex business scenarios.

[0015] In one optional implementation, the step of calling the subclass executor to process the data to be migrated according to a preset data processing logic includes: Convert the data type of the data to be migrated to the original table data type that the subclass executor can process; The subclass executor is invoked to perform at least one of the insert, update, or delete operations on the data to be migrated after data type conversion, based on the parsed operation type.

[0016] The type conversion in this embodiment ensures accurate matching of data formats. The polymorphic calls of the subclass executor implement differentiated processing logic for different data tables and different operation types. The complex data conversion and mapping logic is encapsulated inside the subclass executor, making the main process of the data processing server simple and unified, and providing flexible expansion capabilities to cope with various complex table structure reconstruction needs.

[0017] In an optional implementation, the method further includes: When data from multiple original data tables needs to be merged and migrated to the same target data table, each original data table corresponds to a subclass executor. Each subclass executor processes the data in its corresponding original data table and writes the processed data into the same target data table.

[0018] Through the above-mentioned multi-source merging and migration implementation mechanism, this invention provides a flexible data migration mode that can adapt to various complex table structure reconstruction needs. The system can achieve automated migration by configuring the corresponding subclass executors without modifying the core processing flow, which greatly improves the adaptability and scalability of the data migration system.

[0019] In an optional implementation, the method further includes: When data in an original data table needs to be split and migrated to multiple different target data tables, the original data table corresponds to multiple subclass executors, and the multiple subclass executors respectively write the data into the multiple different target data tables.

[0020] Through the above-described single-source split migration implementation mechanism, this invention provides a flexible data migration mode that can adapt to various complex table structure reconstruction needs. It can achieve automated migration by configuring the corresponding subclass executor without modifying the core processing flow, which greatly improves the adaptability and scalability of the data migration system.

[0021] In a second aspect, the present invention provides a data migration system, which applies the data migration method of the first aspect or any corresponding embodiment described above, the system comprising: The original database stores the data to be migrated; The cloud server communicates with the original database to obtain data update operations from the original database and serializes the data corresponding to the data update operations into data objects before writing them into the cloud middleware. The cloud middleware communicates with the cloud server and is used to receive and store the data objects written by the cloud server in a message queue manner. A data processing server is communicatively connected to the cloud middleware, and the data processing server includes a storage module and a master data synchronization service module. The storage module is used to store global context variables, which contain multiple key-value pairs with the original data table identifier as the key and the synchronization processing container as the value. The master data synchronization service module is used to obtain the data object from the cloud middleware, parse the data object to obtain the data to be migrated and the original data table identifier; based on the original data table identifier, match the corresponding target synchronization processing container from a plurality of pre-built synchronization processing containers, each of the synchronization processing containers having a corresponding original data table; obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to a preset data processing logic; and write the processed data to be migrated into the target data table of the target database. The target database is communicatively connected to the data processing server and is used to receive and store the data to be migrated after being processed by the subclass executor.

[0022] In one optional implementation, the data processing server further includes: The initialization scanning module is used to scan all classes annotated with specific annotations in the data migration system during initialization to identify subclass executors; obtain the original data table identifier and original table data type corresponding to the subclass executor; create a synchronization processing container corresponding to the subclass executor based on the original data table identifier and the original table data type, and declare the subclass executor in the synchronization processing container; and store the created synchronization processing container in a global context variable with the original data table identifier as the key. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a data migration method according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a data migration apparatus according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0027] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0028] Data migration typically involves two parts: First, replacing database components. Since the functionalities of mainstream domestic database products are relatively mature and well-defined, this part is relatively easy to implement and can be completed by directly selecting a suitable database product based on specific business needs. Second, refactoring unreasonable table structures. When the original data system is complex, has a large data volume, and unclear storage logic, the difficulty of refactoring the data table structure increases significantly. In particular, if the system does not use an expansion table mechanism for business expansion, a large number of temporary appended fields will be generated, further exacerbating the chaos of the table structure. Targeted optimization is required during refactoring, such as merging multiple vehicle basic information tables into a single data table, or splitting real-name authentication-related status information into a dedicated table to avoid multiple tables repeatedly maintaining the same status data.

[0029] In existing technologies, all data is stored in a single table. As business expands, new fields are continuously added to the existing table to meet new business needs, causing these new fields to affect all historical data. With the continuous growth of data volume, data access paths become increasingly complex, leading to decreased business access performance or even interruptions. To improve business performance issues caused by the unreasonable design of the original table structure, the table structure needs to be redesigned and the data migrated from the original table to the newly constructed table. However, existing data processing servers only support data migration between tables with the same structure. For data migration between different table structures, custom data migration logic needs to be added for each migration task, resulting in low migration efficiency, high development costs, and difficulty in effectively ensuring data integrity and business compatibility.

[0030] Based on this, the present invention provides a data migration method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a data migration method that can be applied to a data migration system. Figure 1 This is a flowchart of a data migration method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the data update message from the original database, parse the data update message, and obtain the data to be migrated and the identifier of the original data table of the data to be migrated.

[0032] Step S102: Execute the delegated method through the master data synchronization service to migrate the data to be migrated from the original database to the target database.

[0033] Step S102 includes the following steps: Step S1021: Based on the original data table identifier, match the corresponding target synchronous processing container from the pre-built multiple synchronous processing containers. Each synchronous processing container has a corresponding original data table.

[0034] Step S1022: Obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to the preset data processing logic.

[0035] Step S1023: Write the processed data to be migrated into the target data table of the target database.

[0036] The data migration method provided in this embodiment obtains data update messages from the original database, parses the data update messages to obtain the data to be migrated and the identifier of the original data table of the data to be migrated; it then executes a delegated method through the master data synchronization service to migrate the data to be migrated from the original database to the target database. The delegated method includes: matching the corresponding target synchronization processing container from multiple pre-built synchronization processing containers based on the original data table identifier; obtaining the subclass executor bound to the original data table in the target synchronization processing container and calling the subclass executor to process the data to be migrated according to preset data processing logic; and writing the processed data to be migrated into the target data table of the target database to complete the migration of the data to be migrated. This embodiment obtains and parses data update messages and matches the corresponding subclass executor for data processing of the data to be migrated, eliminating the need to develop custom logic separately for each migration task. This achieves automated data migration between heterogeneous table structures, significantly improving migration efficiency and reducing development costs. At the same time, the differentiated processing of the subclass executors can ensure the accuracy of data conversion, data integrity, and business compatibility.

[0037] The steps described above will be explained in detail below.

[0038] In step S101, the data update message in the original database is obtained, the data update message is parsed, and the data to be migrated and the original data table identifier of the data to be migrated are obtained.

[0039] In one embodiment, obtaining data update messages from the original database includes: By responding to data update operations in the original database through the cloud server, the data corresponding to the data update operation is serialized into a data object, and the data object is written into the cloud middleware that communicates with the cloud server. The data processing server subscribes to and retrieves data objects from the cloud middleware, and uses these data objects as data update messages.

[0040] In this embodiment, the data migration system includes a cloud server, cloud middleware, a data processing server, and a target database, which are sequentially connected in communication. The cloud server communicates with the original database, forming a data capture link; the cloud middleware is located between the cloud server and the data processing server, forming a message relay link; and the data processing server communicates with the target database, forming a data writing link. Through this layered architecture, the system achieves automated data migration from the original database to the target database.

[0041] Specifically, the original database is the source database for the data to be migrated, used to store the raw data generated during the operation of the business system, such as a MySQL relational database. In actual business scenarios, the original database stores a large amount of historical data and real-time incremental data. Due to business expansion and unreasonable table structure design, the original database tables may have redundant fields, chaotic field definitions, and complex relationships, requiring restructuring and optimization of the table structure through data migration. The cloud server communicates with the original database. When the business system performs data update operations such as insert, update, or delete on the original database, the cloud server listens in real time and obtains the data corresponding to the data update operation. To ensure the integrity and parsability of data during cross-system transmission, the cloud server converts the obtained data into structured data objects according to a preset serialization protocol. Serialization refers to the process of converting the data structure or object state in memory into a byte sequence that can be stored or transmitted. The data object format used in this embodiment should be able to fully express the field names, field types, field values, and operation types in the original data table. After serialization, the cloud server writes the generated data objects as message producers into the cloud middleware.

[0042] Next, the data processing server subscribes to and retrieves data objects from the cloud middleware, using these data objects as data update messages. The cloud middleware, acting as a message relay layer between the cloud server and the data processing server, employs a message queue mechanism to achieve asynchronous data transmission and buffered storage. The cloud middleware can be Kafka message queue middleware, a distributed, partitioned, multi-replica distributed messaging system that operates on a publish-subscribe model, supporting high-throughput data persistence and reliable transmission. Kafka includes topics, partitions, producers, and consumers, where topics are the classification units of messages, partitions are the physical groups of topics, producers are the message publishers, and consumers are the message subscribers. In this embodiment, the cloud server acts as the message producer, publishing the serialized data objects to a specified Kafka topic; the data processing server, as the message consumer, subscribes to this topic in real time through a Kafka listener, pulling data objects from the message queue in a first-in, first-out order. This data object is the data update message, which at least contains the original data table name, operation type, and data to be migrated.

[0043] This embodiment decouples the data production and consumption ends by introducing cloud middleware. At the same time, by leveraging the persistence characteristics and offset mechanism of message queues, it effectively prevents data loss or duplicate consumption during transmission, providing reliable guarantees for subsequent data integrity verification and breakpoint resume transmission.

[0044] In one embodiment, parsing the data update message to obtain the data to be migrated and the identifier of the original data table of the data to be migrated includes: Parse the data update message to obtain the operation type, the target data to be processed, and the name of the original data table of the target data; The target data is used as the data to be migrated, and the original data table name is used as the identifier of the original data table to be migrated.

[0045] Specifically, after the data processing server obtains the data update message from the cloud middleware, it parses and processes the message to obtain the corresponding operation type, the target data to be processed, and the name of the original data table. The data update message is a data object generated by the cloud server after serializing the data update operation in the original database. It encapsulates complete information about the data change and serves as the basic input for subsequent data migration processing. Data update messages can be serialized using JSON or Avro format. Both are lightweight data exchange formats with excellent cross-platform and cross-language capabilities, capable of fully expressing complex data structures.

[0046] During parsing, the data processing server uses deserialization to restore the acquired data update message from a byte sequence back to a data structure in memory. Deserialization is the reverse process of serialization, that is, restoring a byte sequence to its original data structure or object state. After deserialization, the data processing server extracts the operation type corresponding to the data update message, the target data to be processed, and the name of the original data table for the target data from the parsing results. The operation type identifies the specific operation of this data change, including insert, update, and delete; the target data to be processed is the specific data content involved in this data change, including the field names and values ​​of the corresponding records in the original data table; and the original data table name identifies the original data table corresponding to this data change.

[0047] The extraction results are categorized and organized. The target data obtained from the parsing is identified as the data to be migrated, which will serve as the input object for subsequent data processing. The original data table name obtained from the parsing is identified as the original data table identifier of the data to be migrated, which will serve as the lookup key value for the matching synchronization processing container.

[0048] In a preferred embodiment, the data update message may also include timestamp information to record the time when the data change occurred. The data processing server synchronously extracts this timestamp information during parsing and stores it as auxiliary information for data migration, used for subsequent data consistency verification and migration log recording. For example, during incremental data synchronization after a full data migration, the timestamp information can be used to determine the order of incremental data, ensuring that data is written to the target database in the correct time sequence.

[0049] In this embodiment, the data processing server accurately obtains the operation type, target data to be processed, original data table name, and optional auxiliary information from the original data object. This provides complete and accurate input parameters for subsequent synchronous processing container matching, subclass executor invocation, and data writing operations, ensuring the accuracy and reliability of the data migration process.

[0050] In step S102, the delegated method is executed through the master data synchronization service to migrate the data to be migrated from the original database to the target database.

[0051] After parsing and obtaining the data to be migrated and its corresponding original data table identifier, the data processing server needs to match the corresponding sub-executor based on the original data table identifier and then hand over the data to be migrated to that sub-executor for processing. A sub-executor is a processor unit that actually executes the data migration logic. Each sub-executor implements customized data transformation and processing logic for a specific original data table, enabling it to accurately convert the data in the original table structure into the data format required by the target table structure.

[0052] For the delegate class method in step S102, the following steps are performed: Step S1021: Based on the original data table identifier, match the corresponding target synchronous processing container from the pre-built multiple synchronous processing containers. Each synchronous processing container has a corresponding original data table.

[0053] Step S1022: Obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to the preset data processing logic.

[0054] Step S1023: Write the processed data to be migrated into the target data table of the target database.

[0055] In step S1021, based on the original data table identifier, the corresponding target synchronous processing container is matched from the pre-built multiple synchronous processing containers, and each synchronous processing container has a corresponding original data table.

[0056] Specifically, a synchronization processing container is used to encapsulate processing units related to the original data table. Each synchronization processing container contains an original data table identifier and a subclass executor bound to the original data table corresponding to that identifier. The original data table identifier identifies the original data table corresponding to the container; the subclass executor is bound to the original data table and is used to actually perform the migration processing of data in the original data table.

[0057] In one embodiment, the pre-construction method for the synchronous processing container is as follows: During the initialization of the data migration system, all classes in the data migration system marked with specific annotations are scanned to identify the subclass executors; Obtain the original data table identifier and original table data type corresponding to the subclass executor; Based on the original data table identifier and the original table data type, create a synchronization processing container corresponding to the subclass executor, and declare the subclass executor in the synchronization processing container; The created synchronization processing container is stored in the global context variable using the original data table identifier as the key.

[0058] Specifically, during system initialization, the data processing server scans all classes annotated with specific tags to identify all subclass executors. Based on the original table name and data entity type, it creates a corresponding synchronization processing container for each subclass executor and stores these containers in a global context variable. This pre-storage mechanism eliminates the need to repeatedly create synchronization processing containers during runtime; they can be directly retrieved from the global context variable, significantly improving lookup efficiency and system performance. The global context variable is a globally accessible storage space during system runtime. During storage, a key-value mapping relationship is constructed using the original data table identifier as the key and the synchronization processing container as the value. Specifically, when the data processing server needs to match a target synchronization processing container, it uses the parsed original data table identifier as the key to search for and retrieve the corresponding value from the global context variable, which is the target synchronization processing container corresponding to that original data table.

[0059] In one embodiment, matching a corresponding target synchronization processing container from a pre-built plurality of synchronization processing containers based on the original data table identifier includes: Access the preset global context variable, which stores multiple key-value pairs. Each key-value pair uses the original data table identifier as the key and the synchronization processing container corresponding to the original data table identifier as the value. Using the parsed original data table identifier as the key, the target synchronization processing container corresponding to the original data table identifier is searched and determined from the global context variables.

[0060] Global context variables are globally accessible storage spaces during system runtime. Their scope covers the entire data processing server instance, enabling data sharing across different modules and processing flows. After parsing the data update message and obtaining the original data table identifier, the data processing server immediately performs a lookup operation. Specifically, the data processing server uses the parsed original data table identifier as the lookup key, calls the global context variable's retrieval method (e.g., the `get` method of `ConcurrentHashMap`), and passes in the key-value pair. The global context variable performs a hash calculation based on the key-value pair, locates the corresponding storage location, and quickly returns the value matching the key—that is, the synchronization processing container corresponding to the original data table identifier. If the lookup is successful, this container is designated as the target synchronization processing container for subsequent use by subclass executors. If the lookup fails, meaning the global context variable does not contain a key-value pair corresponding to the original data table identifier, it indicates that the system has not yet configured a corresponding synchronization processing container for that original data table. The data processing server can log an exception or trigger an alarm mechanism to prompt developers to check the configuration.

[0061] In a preferred embodiment, the global context variable is initialized upon system startup and remains read-only during system operation to avoid data inconsistency issues caused by concurrent modifications. The initialization process specifically includes: upon system startup, the Spring framework instantiates all Beans; subsequently, a custom post-processor implementing the BeanPostProcessor interface executes scanning logic on all initialized Beans; for identified subclass executors, the original data table identifier configured in their annotations is obtained, a synchronization processing container is created, and the container is stored in the global context variable as key-value pairs. After initialization, the global context variable contains the mapping relationships of all configured original data tables' synchronization processing containers for quick lookup during subsequent data migration processing.

[0062] In another preferred embodiment, when the system receives a configuration update instruction, it can dynamically create a new synchronization processing container and store it in a global context variable, or replace the original container with the updated container. This allows the system to expand its new data migration capabilities without restarting, thereby improving the system's flexibility and availability.

[0063] In another preferred embodiment, the global context variable also stores the original table data type information, used to identify the entity type of the data in the original data table. After obtaining the target synchronization processing container, the data processing server can extract this data type information from the container and convert the data to be migrated into a type that the subclass executor can process. Specifically, if the current type of the data to be migrated is inconsistent with the original table data type specified in the container, the data processing server can call a type converter to perform data conversion, ensuring that the data format matches before being handed over to the subclass executor for processing, thus avoiding data processing anomalies caused by type mismatch.

[0064] Through the matching mechanism based on global context variables described above, this embodiment achieves fast mapping and efficient lookup between the original data table identifier and the synchronization processing container. The data processing server only needs a hash lookup operation to obtain the target synchronization processing container, without traversing all containers or executing complex matching logic, which greatly improves the efficiency of data migration processing.

[0065] In one embodiment, each synchronous processing container corresponds to only one original data table, which is used to encapsulate the processing logic related to that original data table. However, an original data table can correspond to one or more synchronous processing containers depending on business needs, specifically depending on whether the data in the original data table needs to be migrated to one or more target data tables.

[0066] In the first mapping scenario, one source data table corresponds to one synchronization processing container, i.e., a one-to-one mapping. In this case, when data in the source data table only needs to be migrated to a single target data table, the source data table is configured with only one synchronization processing container, which is bound to a subclass executor. This subclass executor is responsible for processing all the data to be migrated from the source data table according to a preset transformation logic and then writing it to the unique target data table. For example, if a business system restructures its user basic information table and only needs to migrate the data to the newly designed user information table, a one-to-one mapping approach is sufficient to meet the requirements.

[0067] In the second mapping scenario, one original data table corresponds to multiple synchronous processing containers, i.e., a one-to-many split migration. In this case, when data in an original data table needs to be split and migrated to multiple different target data tables, the original data table corresponds to multiple synchronous processing containers, each container is bound to a subclass executor, and each subclass executor is responsible for writing the data to different target data tables. For example, the original system has a large user table containing various fields such as basic user information, account information, real-name authentication information, and preference settings. When restructuring the table structure, this table needs to be split into four specialized sub-tables: a basic user information table, a user account table, a user authentication information table, and a user preference table. At this point, four synchronization processing containers are configured for the original user table. The first synchronization processing container is bound to the first sub-executor and is responsible for writing basic information such as user ID, name, gender, and date of birth into the user basic information table. The second synchronization processing container is bound to the second sub-executor and is responsible for writing account information such as user ID, login name, password hash value, and account status into the user account table. The third synchronization processing container is bound to the third sub-executor and is responsible for writing authentication information such as user ID, real-name authentication status, authentication time, and document information into the user authentication information table. The fourth synchronization processing container is bound to the fourth sub-executor and is responsible for writing preference information such as user ID, topic preferences, and notification settings into the user preference table. When the data processing server receives a data update message from the original user table from the cloud middleware, it simultaneously passes the corresponding data to be migrated to the sub-executors bound to the four synchronization processing containers. Each sub-executor extracts the fields it is interested in from the data to be migrated, performs necessary format conversions, and writes them into its respective target data table, achieving a one-to-many data splitting and migration.

[0068] In the third mapping scenario, multiple source data tables correspond to multiple synchronous processing containers, i.e., many-to-one merging and migration. When data from multiple source data tables needs to be merged and migrated to the same target data table, each source data table corresponds to a synchronous processing container. The subclass executor bound to each container writes the processed data into the same target table. For example, in the original system, basic vehicle information is scattered across three tables: a vehicle basic attribute table, a vehicle technical parameter table, and a vehicle usage status table. After reconstruction, these need to be merged into a single complete vehicle information table. In this case, a synchronous processing container is configured for each source table. The first synchronous processing container is bound to the first subclass executor, responsible for processing the vehicle basic attribute table data and extracting information such as license plate number, brand, model, and color. The second synchronous processing container is bound to the second subclass executor, responsible for processing the vehicle technical parameter table data and extracting information such as engine number, displacement, and emission standard. The third synchronous processing container is bound to the third subclass executor, responsible for processing the vehicle usage status table data and extracting information such as current mileage, maintenance records, and insurance information. After processing the data, each sub-executor writes the processed data into the same target vehicle information table, using a unique vehicle identifier such as the VIN as the primary key. If a record corresponding to the primary key does not yet exist in the target table, an insert operation is performed; if it already exists, an update operation is performed. This process gradually assembles vehicle information from different original tables into a complete vehicle record, achieving many-to-one data merging and migration.

[0069] Through the above-mentioned flexible multi-level mapping mechanism, this embodiment can adapt to various complex table structure reconstruction needs. Whether it is splitting a single table into multiple tables or merging multiple tables into a single table, automatic migration can be achieved by configuring the corresponding synchronization processing container and subclass executor without modifying the core processing flow.

[0070] In step S1022, the subclass executor bound to the original data table in the target synchronization processing container is obtained, and the subclass executor is called to process the data to be migrated according to the preset data processing logic.

[0071] After matching the target synchronization processing container, the data processing server extracts the encapsulated subclass executor from that container. The subclass executor is a concrete implementation of the base class executor, which defines general interface methods for data processing, including insertion, update, and deletion operations. Each subclass executor implements the specific logic of these interface methods based on the characteristics of its corresponding original data table and the requirements of the target table structure. For example, for the subclass executor corresponding to the vehicle basic information table, its insertion operation might require merging vehicle information scattered across multiple fields in the original table and then writing it into the corresponding field of the target table; for the subclass executor corresponding to the real-name authentication status table, its update operation might require mapping the status fields in the original table to multiple status identifier fields in the target table. In this embodiment, the subclass executor encapsulates the complex table structure conversion logic internally, providing a unified interface calling method externally.

[0072] After obtaining the subclass executor, the data processing server calls the subclass executor, passing the data to be migrated as input parameters. The subclass executor then processes the data according to its preset data processing logic. Specifically, the data processing server calls the corresponding interface method in the subclass executor based on the parsed operation type. If the operation type is insert, the insert method of the subclass executor is called; if the operation type is update, the update method is called; and if the operation type is delete, the delete method is called. When executing the specific method, the subclass executor performs field mapping, data conversion, and format adjustment on the data to be migrated according to its internal business logic, generating a data object that conforms to the target table structure requirements. The processed data is then returned to the data processing server, which writes it into the target database.

[0073] In a preferred embodiment, the synchronization processing container also contains original table data type information, used to identify the entity type of the data in the original data table. Before invoking the subclass executor, the data processing server first converts the data to be migrated into a type that the subclass executor can process based on this original table data type information, ensuring data type matching and avoiding data processing anomalies caused by type mismatch.

[0074] In another preferred embodiment, the system supports scenarios where multiple sub-executors are configured for the same source data table. For example, when data in a source data table needs to be split and migrated to multiple different target data tables, the synchronization processing container corresponding to the source data table is bound to multiple sub-executors. Each sub-executor is responsible for writing data to different target data tables in the target database. After obtaining the target synchronization processing container, the data processing server iterates through all sub-executors in the container, sequentially calls each sub-executor to process the data to be migrated, and writes the processed data from each sub-executor into the corresponding target data table, thus achieving one-to-many data splitting and migration.

[0075] In this embodiment, the data processing server does not need to concern itself with the specific table structure conversion logic. It only needs to find the corresponding subclass executor based on the original data table identifier and call its interface method to complete the complex data migration process, thereby improving the scalability and maintainability of the system. When it is necessary to add a new data table or adjust the table structure mapping relationship, it is only necessary to add or modify the corresponding subclass executor without changing the main process code of the data processing server.

[0076] In one embodiment, invoking a subclass executor to process the data to be migrated according to preset data processing logic includes: Convert the data type of the data to be migrated to the original table data type that can be processed by the subclass executor; Invoke the subclass executor and, based on the parsed operation type, perform at least one of the following operations on the data to be migrated after data type conversion: insert, update, or delete.

[0077] The data to be migrated refers to the target data parsed from the data update message. Its data type is a serialized general data format, such as a JSON object or an Avro record. The original table data type that the subclass executor can handle refers to the input data type expected by the internal implementation of the subclass executor, which is usually the entity class type corresponding to the original data table structure.

[0078] Specifically, each subclass executor is bound to the data type of its corresponding original data table upon creation. This data type can be a POJO class in the Java programming language, with its attribute fields corresponding one-to-one with the fields of the original data table. Since the data to be migrated undergoes serialization during transmission, its data type may differ from the original table data type expected by the subclass executor, thus requiring type conversion. After the data processing server obtains the original table data type information from the target synchronization container, it calls the type converter to convert the data to be migrated from a general data format to the original table data type. The type converter is a component specifically responsible for data format conversion. It can map the field values ​​of the source data object to the corresponding fields of the target data object one by one according to preset mapping rules, while performing necessary data format adjustments and type casting. For example, if a field in the original data table is an integer type, while the field in the data to be migrated is a string type, the type converter can convert the string to an integer; if a field in the original data table is a date type, while the field in the data to be migrated is a timestamp format, the type converter can convert the timestamp to the specified date format. Through type conversion, the data to be migrated is converted into a format that the subclass executor can directly process.

[0079] After data type conversion, the data processing server uses the converted data to be migrated as input parameters and calls the subclass executor bound in the target synchronization processing container. The subclass executor is a concrete implementation of the base class executor, which defines general interface methods for data processing, including insert, update, and delete methods. Each subclass executor implements the specific logic of these interface methods based on the characteristics of its corresponding original data table and the requirements of the target table structure. The data processing server determines which interface method of the subclass executor to call based on the operation type parsed from the data update message: if the operation type is insert, the insert method is called; if the operation type is update, the update method is called; and if the operation type is delete, the delete method is called. When executing the specific method, the subclass executor processes the incoming data to be migrated according to its internally implemented business logic.

[0080] In one specific embodiment, when the operation type is insert, the insert method of the subclass executor inserts the data to be migrated into the corresponding table in the target database. This insert operation may involve complex data transformation and mapping logic. For example, for the subclass executor corresponding to the vehicle basic information table, its insert method may need to merge vehicle information scattered across multiple fields in the original table. Suppose the original data table has three fields: "Brand," "Model," and "Year of Production," while the target data table requires these three fields to be merged into a single "Vehicle Description" field. The subclass executor's insert method will perform a string concatenation operation to generate description information in the format of "Brand-Model-Year of Production," and then write it into the "Vehicle Description" field of the target data table. As another example, for the subclass executor corresponding to the real-name authentication status table, its insert method may need to map the status fields in the original table to multiple status identifier fields in the target table. Suppose the original data table has an "authentication status" field with the value "authenticated" or "unauthenticated", while the target data table requires an "is_certified" boolean field to represent the authentication status. The insert method of the subclass executor will perform a conditional judgment: if the original field value is "authenticated", then set "is_certified" to true in the corresponding record of the target table; if it is "unauthenticated", then set it to false.

[0081] In another specific embodiment, when the operation type is update, the subclass executor's update method locates the corresponding record in the target database based on the primary key information in the data to be migrated, and updates the other field values ​​in the data to be migrated to that record. Update operations may also involve field mapping and format conversion. For example, the subclass executor needs to map field names in the original data table to field names in the target data table, and convert the data format in the original data table to the format required by the target data table. Before performing the update operation, the subclass executor can also first query whether a corresponding record exists in the target database. If it exists, the update is performed; if it does not exist, it can decide whether to convert it to an insert operation based on the configuration strategy.

[0082] In another specific embodiment, when the operation type is deletion, the deletion method of the subclass executor locates the corresponding record in the target database based on the primary key information in the data to be migrated, and deletes the record from the target data table. The deletion operation also needs to consider data integrity and business compatibility. For example, the subclass executor can be configured to directly remove the record from the table or only update the deletion flag field in the record; the specific strategy can be determined according to business requirements.

[0083] In another embodiment, the subclass executor also supports a batch processing mode to improve data migration efficiency. When the data processing server obtains a batch of data update messages belonging to the same original data table from the cloud middleware, it can aggregate the data to be migrated corresponding to these messages into a batch data set, and then call the batch processing method of the subclass executor all at once. The batch insert, batch update, or batch delete methods of the subclass executor internally adopt database batch operation interfaces, such as JDBC batch update or MyBatis batch insert, merging multiple data operations into a single database interaction, significantly reducing database connection overhead and improving data migration performance.

[0084] The type conversion in this embodiment ensures accurate matching of data formats. The polymorphic calls of the subclass executor implement differentiated processing logic for different data tables and different operation types. The complex data conversion and mapping logic is encapsulated inside the subclass executor, making the main process of the data processing server simple and unified, and providing flexible expansion capabilities to cope with various complex table structure reconstruction needs.

[0085] In step S1023, the processed data to be migrated is written into the target data table of the target database.

[0086] After the data processing server calls the subclass executor to process the data to be migrated, it obtains the processed data returned by the subclass executor and writes this data into the corresponding target data table in the target database. The target database refers to the new database after its table structure has been redesigned according to the business domain, with clear business boundaries and a reasonable data organization method. The target data table is a table in the target database used to store specific business data. Its table structure may differ significantly from the original data table, for example, in the number of fields, field names, field types, or data organization methods. The data processing server obtains a connection to the target database through the database connection pool, calls the database access interface to perform insert, update, or delete operations, and persistently stores the processed data in the target data table. After the data writing is completed, the data migration process ends, and the data changes in the original database have been successfully synchronized to the target database.

[0087] In one embodiment, the method further includes: When data from multiple source tables needs to be merged and migrated to the same target table, each source table corresponds to a subclass executor. Each subclass executor processes the data in its corresponding source table and writes the processed data into the same target table.

[0088] This embodiment also supports multi-source merging migration, which refers to a data migration model that integrates data from multiple tables with different sources into a single target table. This is typically used in data migration scenarios where the original system suffers from complex queries and difficult maintenance due to scattered data storage. Specifically, taking vehicle basic information management as an example, the original system might have scattered vehicle basic information across multiple different tables due to business expansion and legacy issues. For instance, one table might store basic vehicle attributes such as license plate number, brand, model, and color; another table might store technical parameters such as engine number, displacement, and emission standards; and a third table might store vehicle usage status such as current mileage, maintenance records, and insurance information. During table structure reconstruction, this scattered vehicle information needs to be merged into a single complete vehicle information table to simplify query logic and improve access performance.

[0089] In multi-source merging and migration scenarios, the system configures a corresponding sub-executor for each original data table. For example, a first sub-executor is configured for the vehicle basic attribute table, a second sub-executor for the vehicle technical parameter table, and a third sub-executor for the vehicle usage status table. Each sub-executor implements customized data extraction, transformation, and loading logic for its corresponding original data table. When the data processing server obtains data update messages from different original data tables from the cloud middleware, they are handled by the corresponding sub-executors. The first sub-executor extracts fields such as license plate number, brand, model, and color from the data in the vehicle basic attribute table and maps them to the corresponding fields in the target vehicle information table; the second sub-executor extracts fields such as engine number, displacement, and emission standard from the data in the vehicle technical parameter table and writes them to the corresponding fields in the same target vehicle information table; the third sub-executor extracts fields such as current mileage, maintenance records, and insurance information from the data in the vehicle usage status table and also writes them to the same target vehicle information table. Since multiple sub-executors operate on the same target table, it is necessary to ensure the accuracy and consistency of the data writing. Therefore, when processing data, each sub-type actuator uses a unique vehicle identifier, such as the license plate number or vehicle identification number, as the primary key of the target data table to perform insert or update operations. When the target data table does not yet have a record corresponding to this primary key, an insert operation is performed to create a new record; when the target data table already has a record corresponding to this primary key, an update operation is performed to supplement or overwrite the corresponding fields, thereby gradually assembling vehicle information from different original tables into a complete vehicle record.

[0090] In a preferred embodiment, when data from multiple source tables provides different values ​​for the same target field, the system can determine which source table's value to use based on a preset priority rule. For example, vehicle color information may appear in both the basic attribute table and the configuration table. If the basic attribute table has a higher priority than the configuration table, the system can be configured to prioritize the basic attribute table and ignore the color values ​​in the configuration table, ensuring data consistency.

[0091] Through the multi-source merging and migration implementation mechanism described above, this embodiment provides a flexible data migration mode that can adapt to various complex table structure reconstruction needs. The system can achieve automated migration by configuring the corresponding subclass executors without modifying the core processing flow, which greatly improves the adaptability and scalability of the data migration system.

[0092] In one embodiment, the method further includes: When data in an original data table needs to be split and migrated to multiple different target data tables, one original data table corresponds to multiple subclass executors, and the multiple subclass executors write the data into the multiple different target data tables respectively.

[0093] This embodiment also supports single-source split migration, which refers to a data migration pattern that splits data from a source table into multiple different target tables. This is typically used in scenarios where the original system has an excessively large single table, redundant fields, or severe business coupling. Specifically, taking user information management as an example, the original system might centrally store all user-related data in a single, massive user table. This table contains various fields such as basic user information, account information, real-name authentication information, preference settings, and behavior logs, resulting in a bloated table structure, degraded access performance, and difficult maintenance. When restructuring the table, this large table needs to be split into multiple specialized sub-tables according to business domains, such as a user basic information table, a user account table, a user authentication information table, and a user preference table, to achieve specialized data storage and decoupled management.

[0094] In a single-source split migration scenario, the system configures multiple corresponding sub-executors for the same source data table. Each sub-executor is responsible for writing data from the source table to a different target data table. For example, for the source user table, the first sub-executor is responsible for writing to the user basic information table, the second sub-executor is responsible for writing to the user account table, the third sub-executor is responsible for writing to the user authentication information table, and the fourth sub-executor is responsible for writing to the user preference table. When the data processing server obtains a data update message belonging to the source user table from the cloud middleware, it simultaneously transmits the data to be migrated corresponding to that message to all associated sub-executors, or selectively transmits it to some sub-executors based on the data content. Each sub-executor extracts the fields it is interested in from the data to be migrated, performs necessary format conversions, and then writes it to its corresponding target data table. For example, the first sub-executor extracts fields such as user ID, name, gender, date of birth, and contact information from the data to be migrated and writes them into the user basic information table; the second sub-executor extracts fields such as user ID, login name, password hash value, and account status and writes them into the user account table; the third sub-executor extracts fields such as user ID, real-name authentication status, authentication time, document type, and document number and writes them into the user authentication information table; the fourth sub-executor extracts fields such as user ID, theme preference, notification settings, and language preference and writes them into the user preference table.

[0095] This embodiment also supports determining which subclass executors should handle the data based on its content. For example, if the data update message only contains changes to basic user information, the system can call only the first subclass executor to process it, avoiding unnecessary subclass executor calls; if the message contains changes to real-name authentication information, the system will call the third subclass executor to process it. This conditional dispatch strategy can reduce unnecessary processing overhead and improve system performance.

[0096] Through the aforementioned multi-source merging migration and single-source splitting migration mechanisms, this embodiment provides a flexible data migration mode capable of adapting to various complex table structure reconstruction needs. Whether storing dispersed data centrally or centralized data in a distributed manner, automated migration can be achieved by configuring corresponding subclass executors without modifying the core processing flow, greatly enhancing the adaptability and scalability of the data migration system.

[0097] This embodiment also provides a data migration system for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The system includes: The original database stores data that needs to be migrated; The cloud server communicates with the original database to obtain data update operations from the original database, and serializes the data corresponding to the data update operations into data objects before writing them into the cloud middleware. Cloud middleware communicates and connects with cloud servers to receive and store data objects written by cloud servers in the form of message queues. The data processing server communicates with the cloud middleware and includes a storage module and a master data synchronization service module. The storage module is used to store global context variables, which contain multiple key-value pairs with the original data table identifier as the key and the synchronization processing container as the value. The master data synchronization service module is used to obtain data objects from the cloud middleware, parse the data objects to obtain the data to be migrated and the original data table identifier; based on the original data table identifier, it matches the corresponding target synchronization processing container from multiple pre-built synchronization processing containers, each synchronization processing container has a corresponding original data table; it obtains the subclass executor bound to the original data table in the target synchronization processing container, and calls the subclass executor to process the data to be migrated according to the preset data processing logic; and writes the processed data to be migrated into the target data table of the target database. The target database communicates with the data processing server and is used to receive and store the data to be migrated after being processed by the subclass executor.

[0098] The data processing server also includes: The initialization scanning module is used to scan all classes annotated with specific annotations in the data migration system during initialization to identify subclass executors; obtain the original data table identifier and original table data type corresponding to the subclass executor; create a synchronization processing container corresponding to the subclass executor based on the original data table identifier and original table data type, and declare the subclass executor in the synchronization processing container; and store the created synchronization processing container in the global context variable with the original data table identifier as the key.

[0099] The structure of the system will be described in detail below with reference to specific implementation methods.

[0100] The data migration system in this embodiment is used to migrate data from a source database to a target database, and is particularly suitable for scenarios where the structures of the original data table and the target data table are inconsistent. For example... Figure 2 As shown, the system includes: the original database, cloud server, cloud middleware, data processing server, and target database.

[0101] The original database is the source database for the data to be migrated, used to store the raw data generated during the operation of the business system. The business system contains one or more business services, which interact with the original database through CRUD operations to add, query, update, or delete data. Whenever a business service initiates a data update operation, the original database records the data change to trigger the subsequent data migration process. In this embodiment, the original database can be a MySQL relational database, which organizes data using a table structure. The tables store historical data accumulated during the operation of the business system, as well as incremental data generated in real time. Due to historical reasons or business expansion needs, the table structure in the original database may have design flaws, such as storing data from multiple business domains in a single table, or having a large number of temporarily added fields, leading to data redundancy and complex access paths. Therefore, data migration is necessary to restructure and optimize the table structure.

[0102] The cloud server communicates with the original database to monitor and capture data update operations in the original database in real time. Specifically, the cloud server obtains the data content corresponding to data update operations such as inserts, updates, and deletions performed by the original database by parsing its binary logs or through trigger mechanisms. To ensure the integrity and parsability of data during cross-system transmission, the cloud server converts the captured data into structured data objects according to a preset serialization protocol. Serialization refers to the process of converting data structures or object states in memory into a byte sequence that can be stored or transmitted. The data object format used in this embodiment should be able to fully express information such as field names, field types, field values, and operation types in the original data table. After serialization, the cloud server writes the generated data objects as message producers into the cloud middleware.

[0103] The cloud middleware communicates with the cloud server, acting as a message relay layer between the cloud server and the data processing server. It employs a message queue mechanism to achieve asynchronous data transmission and buffered storage. Specifically, in this solution, the cloud middleware can be a Kafka message queue middleware. Kafka is a distributed, partitioned, multi-replica distributed messaging system that operates on a publish-subscribe model, supporting high-throughput data persistence and reliable transmission. The cloud server, acting as the message producer, publishes serialized data objects to a designated topic in Kafka; the data processing server, acting as the message consumer, subscribes to this topic in real time through a Kafka listener, pulling data objects from the message queue in a first-in, first-out order. By introducing the cloud middleware, this embodiment decouples the data producer and consumer. Furthermore, leveraging the persistence features and offset mechanism of message queues, it effectively prevents data loss or duplicate consumption during transmission, providing reliable guarantees for subsequent data integrity verification and breakpoint resumption.

[0104] The data processing server communicates with the cloud middleware to subscribe to and retrieve data objects, parse and process the data objects, and write the processed data to the target database. The data processing server includes a storage module and a master data synchronization service module.

[0105] The storage module stores global context variables, which are globally accessible storage spaces during system runtime. Their scope covers the entire data processing server instance, enabling data sharing between different modules and processing flows. Specifically, each global context variable contains multiple key-value pairs. Each key-value pair uses the original data table identifier as the key and the corresponding synchronization processing container as the value. During system initialization, the data processing server scans all classes annotated with specific tags to identify all subclass executors. For each subclass executor, a corresponding synchronization processing container is created, and these containers are then stored as key-value pairs in the global context variables for quick retrieval during subsequent data migration processing.

[0106] The master data synchronization service module is used to execute delegated methods to implement full-process control of data migration. Specifically, the master data synchronization service module listens for messages on a specified topic in the cloud middleware in real time through a Kafka listener. When a new message is detected, it retrieves a data object from the cloud middleware. After retrieving the data object, the master data synchronization service module parses the data object to obtain the data to be migrated, its corresponding original data table identifier, and information such as the operation type. The data object is a data object generated by the cloud server serializing the data update operation on the original database, which at least contains the original data table name, operation type, and target data to be migrated.

[0107] After obtaining the original data table identifier, the main data synchronization service module matches the corresponding target synchronization processing container from multiple pre-built synchronization processing containers based on this identifier. Specifically, using the parsed original data table identifier as the key, the module searches for and retrieves the value corresponding to that key from the global context variable; that is, the target synchronization processing container corresponding to the original data table identifier. Each synchronization processing container corresponds to one original data table, and each synchronization processing container is bound to a subclass executor corresponding to that original data table. The subclass executor is a concrete implementation subclass of the base class executor, which defines general interface methods for data processing, including operations such as insertion, update, and deletion. Each subclass executor implements the specific logic of these interface methods according to the characteristics of its corresponding original data table and the requirements of the target table structure.

[0108] After acquiring the target synchronization processing container, the main data synchronization service module retrieves the bound subclass executor from the container and hands over the data to be migrated to that subclass executor for processing. Specifically, the main data synchronization service module determines which interface method of the subclass executor to call based on the parsed operation type. If the operation type is insert, the insert method of the subclass executor is called; if the operation type is update, the update method is called; and if the operation type is delete, the delete method is called. When executing the specific method, the subclass executor performs field mapping, data format conversion, and data cleaning on the data to be migrated according to its internal business logic, generating a data object that conforms to the target table structure requirements.

[0109] After the subclass executor completes data processing, it returns the processed data to the main data synchronization service module. The main data synchronization service module obtains a connection to the target database through the database connection pool, calls the database access interface to perform insert, update, or delete operations, and persistently stores the processed data in the corresponding target data table in the target database. Once the data writing is complete, the data migration process ends, and the data changes in the original database have been successfully migrated to the target database.

[0110] In a preferred embodiment, the master data synchronization service module also supports batch processing mode to improve data migration efficiency. When the master data synchronization service module obtains a batch of data update messages belonging to the same original data table from the cloud middleware, it can aggregate the data to be migrated corresponding to these messages into a batch data set, and then call the batch processing method of the subclass executor all at once. The batch insert, batch update, or batch delete methods of the subclass executor internally adopt the database batch operation interface, merging multiple data operations into a single database interaction, significantly reducing database connection overhead and improving data migration performance.

[0111] In another preferred embodiment, when a single data update operation involves writing to multiple target tables, such as in a one-to-many split migration scenario, the master data synchronization service module can wrap multiple write operations within the same database transaction. When the target database supports cross-table transactions, the master data synchronization service module sequentially calls each subclass executor to process the data and executes write operations on each target table within a single database transaction. All writes take effect simultaneously upon transaction commit and are completely undone upon transaction rollback, thereby ensuring data consistency.

[0112] Through the above structural design, the storage module provides fast container lookup service through global context variables, the master data synchronization service module separates process control from business logic through delegate class methods, and the subclass executor is responsible for specific data transformation processing. Each module has a clear division of labor and works together to form an efficient, reliable, and scalable data migration processing mechanism.

[0113] In one embodiment, the data processing server further includes an initialization scanning module, which performs the construction and registration of the synchronization processing container during the initialization phase of the data migration system. Through the initialization scanning module, the system can automatically identify all subclass executors and establish corresponding synchronization processing containers without manual configuration of mapping relationships, significantly improving the system's scalability and ease of use.

[0114] Specifically, during the initialization of the data migration system, the initialization scanning module scans all classes annotated with specific annotations to identify subclass executors. In this embodiment, the specific annotations include `@Component` and `@SyncExecutor`. `@Component` is a general annotation provided by the Spring framework, used to mark a class as a Bean managed by the Spring container; `@SyncExecutor` is a custom annotation used to identify the class as a subclass executor for data migration. Developers need to annotate these two annotations on the class declaration when writing subclass executors.

[0115] When the system starts, after the Spring framework completes the instantiation of all Beans, the initialization scanning module executes scanning logic on all instantiated objects through the post-processor mechanism provided by the Spring framework. Specifically, the initialization scanning module calls a custom post-processor method to traverse all initialized Bean objects and uses the instanceof operator to determine whether the object is an implementation class of the DataSyncExecutor interface. Since each subclass executor is annotated with @Component, meaning it has already been instantiated as a Bean object by the Spring container, it can be scanned by the post-processor.

[0116] Once an object is identified as a subclass of `DataSyncExecutor`, the initialization scanning module further extracts necessary information from that subclass executor. The original data table identifier can be obtained by parsing custom annotation parameters on the subclass executor or through the `getTableName()` method provided in the subclass executor. The original table data type refers to the entity class type corresponding to the original table processed by the subclass executor. This type information can be obtained from the generic parameters or specific methods of the subclass executor using reflection. After obtaining the original data table identifier and the original table data type, the initialization scanning module creates a corresponding synchronization processing container instance for each subclass executor. The synchronization processing container is a custom data structure used to encapsulate the processing units related to the original data table.

[0117] In this embodiment, the synchronization processing container includes at least the original data table identifier, the original table data type, and a subclass executor instance. The original data table identifier identifies the source data table corresponding to the container; the original table data type is the entity type of the data in the original table; and the subclass executor instance is declared using the interface type `DataSyncExecutor`. When creating the synchronization processing container, the initialization scanning module places the acquired subclass executor instance into the container using the interface declaration, and also stores the original table data type into the container. Thus, the synchronization processing container contains both the executor instance and type information, providing complete support for subsequent data processing.

[0118] The created synchronization processing containers are stored in the global context variable. After container creation, the initialization scanning module stores key-value pairs in the global context variable, using the original data table identifier as the key and the synchronization processing container as the value. After storage, the global context variable contains the mapping relationships of all configured synchronization processing containers for the original data tables. After completing the above initialization process, the data migration system enters a ready state. When a data update message is subsequently received, the main data synchronization service module can quickly find and retrieve the corresponding target synchronization processing container directly from the global context variable based on the parsed original data table identifier, eliminating the need for repeated scanning or dynamic creation, thus greatly improving the efficiency of data migration processing.

[0119] In another preferred embodiment, the initialization scanning module supports a dynamic update mechanism. When a new subclass executor is added during system operation, the initialization scanning module can re-execute the scanning logic, identify the new subclass executor, create a corresponding synchronization processing container, and add the new container to the global context variables. Through the dynamic update mechanism, the system in this embodiment can expand its new data migration capabilities without restarting, improving the system's flexibility and availability.

[0120] The target database communicates with the data processing server to receive and store data processed by the server. In this embodiment, the target database can be a PostgreSQL database, whose table structure has been redesigned according to the business domain, exhibiting clear business boundaries and a reasonable data organization method. For example, basic vehicle information scattered across multiple tables in the original database is merged into a single vehicle information table, and real-name authentication-related status information is separated from the main business table into an independent authentication status table, avoiding the duplication of maintaining the same status data across multiple tables. By redesigning the table structure, the target database eliminates data redundancy and field clutter issues from the original database, optimizes data access paths, and improves business query performance.

[0121] In another preferred embodiment, the system supports multiple data migration modes. When data from multiple source tables needs to be merged and migrated to the same target table, each source table corresponds to a sub-executor. Each sub-executor processes the data in its respective source table and writes the processed data into the same target table. When data from a single source table needs to be split and migrated to multiple different target tables, that source table corresponds to multiple sub-executors, each writing the data into its corresponding target table. This flexible mapping mechanism enables the system to adapt to various complex table structure reconstruction needs, achieving efficient migration between heterogeneous data tables.

[0122] Through the above system architecture, this embodiment ensures the reliability of data transmission by introducing cloud middleware, realizes flexible data processing logic through pre-built synchronous processing containers and subclass executors, and achieves fast processor matching through global context variables. It effectively solves the problems of low efficiency, high cost, and difficulty in ensuring data integrity and business compatibility caused by the need to manually develop custom logic for data migration of heterogeneous table structures in the prior art.

[0123] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0124] The following is a detailed reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on a program stored in a read-only memory (i.e., ROM 302) or a program loaded from memory 308 into a random access memory (i.e., RAM 303). The RAM 303 also stores various programs and data required for the operation of the electronic device. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. Input / output (i.e., I / O interface 305) is also connected to the bus 304.

[0125] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0126] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a memory 308, or installed from a ROM 302. When the computer program is executed by the processor 301, it performs the functions defined in the data migration method of the embodiments of the present invention.

[0127] Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0128] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data migration method shown in the above embodiments is implemented.

[0129] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0130] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data migration method, characterized in that, The method, applied to a data migration system, includes: Obtain the data update message from the original database, parse the data update message to obtain the data to be migrated and the original data table identifier of the data to be migrated; The delegated methods are executed through the master data synchronization service to migrate the data to be migrated from the original database to the target database; wherein the delegated methods include: Based on the original data table identifier, a corresponding target synchronous processing container is matched from a plurality of pre-built synchronous processing containers, and each synchronous processing container has a corresponding original data table; Obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to the preset data processing logic; The processed data to be migrated is written into the target data table of the target database.

2. The method according to claim 1, characterized in that, The step of obtaining the data update message from the original database includes: By responding to the data update operation in the original database through the cloud server, the data corresponding to the data update operation is serialized into a data object, and the data object is written into the cloud middleware that communicates with the cloud server. The data processing server subscribes to and retrieves the data object from the cloud middleware, and uses the data object as the data update message.

3. The method according to claim 1, characterized in that, The process of parsing the data update message to obtain the data to be migrated and the original data table identifier of the data to be migrated includes: Parse the data update message to obtain the operation type, the target data to be processed, and the original data table name of the target data corresponding to the data update message; The target data is used as the data to be migrated, and the original data table name is used as the original data table identifier of the data to be migrated.

4. The method according to claim 1, characterized in that, The step of matching the corresponding target synchronization processing container from a pre-built set of multiple synchronization processing containers based on the original data table identifier includes: Access a preset global context variable, which stores multiple key-value pairs. Each key-value pair uses the original data table identifier as the key and the synchronization processing container corresponding to the original data table identifier as the value. Using the parsed original data table identifier as the key, the target synchronization processing container corresponding to the original data table identifier is searched and determined from the global context variable.

5. The method according to claim 1, characterized in that, The method for pre-building the synchronization processing container is as follows: During the initialization of the data migration system, all classes in the data migration system marked with specific annotations are scanned to identify the subclass executors; Obtain the original data table identifier and original table data type corresponding to the subclass executor; Based on the original data table identifier and the original table data type, a synchronization processing container corresponding to the subclass executor is created, and the subclass executor is declared in the synchronization processing container; The created synchronization processing container is stored in the global context variable using the original data table identifier as the key.

6. The method according to claim 3, characterized in that, The process of calling the subclass executor to process the data to be migrated according to the preset data processing logic includes: Convert the data type of the data to be migrated to the original table data type that the subclass executor can process; The subclass executor is invoked to perform at least one of the insert, update, or delete operations on the data to be migrated after data type conversion, based on the parsed operation type.

7. The method according to claim 1, characterized in that, The method further includes: When data from multiple original data tables needs to be merged and migrated to the same target data table, each original data table corresponds to a subclass executor. Each subclass executor processes the data in its corresponding original data table and writes the processed data into the same target data table.

8. The method according to claim 1, characterized in that, The method further includes: When data in an original data table needs to be split and migrated to multiple different target data tables, the original data table corresponds to multiple subclass executors, and the multiple subclass executors respectively write the data into the multiple different target data tables.

9. A data migration system, characterized in that, The system, employing the method of any one of claims 1 to 8, comprises: The original database stores the data to be migrated; The cloud server communicates with the original database to obtain data update operations from the original database and serializes the data corresponding to the data update operations into data objects before writing them into the cloud middleware. The cloud middleware communicates with the cloud server and is used to receive and store the data objects written by the cloud server in a message queue manner. A data processing server is communicatively connected to the cloud middleware, and the data processing server includes a storage module and a master data synchronization service module. The storage module is used to store global context variables, which contain multiple key-value pairs with the original data table identifier as the key and the synchronization processing container as the value. The master data synchronization service module is used to obtain the data object from the cloud middleware, parse the data object to obtain the data to be migrated and the original data table identifier; based on the original data table identifier, match the corresponding target synchronization processing container from a plurality of pre-built synchronization processing containers, each of the synchronization processing containers having a corresponding original data table; obtain the subclass executor bound to the original data table in the target synchronization processing container, and call the subclass executor to process the data to be migrated according to a preset data processing logic; and write the processed data to be migrated into the target data table of the target database. The target database is communicatively connected to the data processing server and is used to receive and store the data to be migrated after being processed by the subclass executor.

10. The system according to claim 9, characterized in that, The data processing server also includes: The initialization scanning module is used to scan all classes annotated with specific annotations in the data migration system during initialization to identify subclass executors; obtain the original data table identifier and original table data type corresponding to the subclass executor; create a synchronization processing container corresponding to the subclass executor based on the original data table identifier and the original table data type, and declare the subclass executor in the synchronization processing container; and store the created synchronization processing container in a global context variable with the original data table identifier as the key.

Images