Table image-based data synchronization task scheduling method, device, system and medium

Abstract

The application discloses a table image-based data synchronization task arrangement method, device and system and a medium, which comprises the following steps: collecting the metadata of all to-be-synchronized data tables in a source end database and performing multi-dimensional quantization processing to generate corresponding table image feature vectors; classifying all to-be-synchronized data tables according to the table image feature vectors and confirming the table types and synchronization task attributes of each to-be-synchronized data table; performing optimal task arrangement solving based on the table types and synchronization task attributes of each to-be-synchronized data table through a preset cost model to obtain optimal task grouping and execution priorities; generating corresponding task parameter configurations for each task group, and executing corresponding synchronization tasks according to the execution priorities and the task parameter configurations when a trigger execution instruction is given. Through multi-dimensional table image-based intelligent classification and optimal task arrangement, the automatic intelligent arrangement and execution of synchronization tasks are realized, the labor cost and the configuration threshold are effectively reduced, and the efficiency and reliability of data synchronization are improved.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method, apparatus, system, and medium for orchestrating data synchronization tasks based on table profiles. Background Technology

[0002] In the field of data synchronization, third-party synchronization tools are typically used to create synchronization tasks to synchronize data from the source database to the target database. The appropriateness of the whitelist configuration for these synchronization tasks directly determines whether the tasks can be completed within the predetermined synchronization window. Therefore, scientifically scheduling synchronization tasks and properly configuring whitelists are crucial for ensuring the timeliness and reliability of data synchronization.

[0003] Currently, the industry standard is for database administrators (DBAs) to intervene early, manually analyzing and configuring whitelists and creating synchronization tasks. This manual approach not only has high costs and complex configuration requirements in actual operation and maintenance, but also makes it difficult to ensure the rationality of synchronization task orchestration, thus affecting the efficiency and reliability of data synchronization. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a data synchronization task orchestration method, apparatus, system and medium based on table profile that can be applied to financial technology or other related fields, aiming to realize the automated and intelligent orchestration of synchronization tasks, effectively reduce labor costs and configuration thresholds, and improve the efficiency and reliability of data synchronization.

[0005] The technical solution of the present invention is as follows: A data synchronization task orchestration method based on table profiles includes: Collect metadata from all tables in the source database that need to be synchronized; The metadata is subjected to multi-dimensional quantization processing to generate a table profile feature vector corresponding to the data table to be synchronized; Based on the feature vector of the table profile, all data tables to be synchronized are classified to confirm the table type and synchronization task attributes of each data table to be synchronized. The optimal task arrangement is solved by using a preset cost model based on the table type and synchronization task attributes of each data table to be synchronized, and the optimal task grouping and execution priority are obtained. For each task group, generate corresponding task parameter configurations, and when the execution instruction is triggered, execute the corresponding synchronous task according to the execution priority and the task parameter configurations.

[0006] In one embodiment, after generating corresponding task parameter configurations for each task group, the method further includes: Based on the optimal task grouping, the task parameter configuration and execution priority are used to output a visual task orchestration view; Receive user's fine-tuning instructions for the task orchestration view, adjust the task parameter configuration and / or execution priority according to the fine-tuning instructions, and obtain the final task orchestration result.

[0007] In one embodiment, the metadata of all data tables to be synchronized in the source database includes: Receive the full whitelist submitted by the user; A data connection is established with the source database based on the full whitelist, and metadata of all data tables to be synchronized in the full whitelist is collected.

[0008] In one embodiment, the step of performing multi-dimensional quantization processing on the metadata to generate a table profile feature vector corresponding to the data table to be synchronized includes: The metadata is extracted according to preset dimensions to obtain basic scale dimension data, structural feature dimension data, unique attribute dimension data and load feature dimension data respectively. Perform corresponding standard quantization calculations on the data of each dimension to generate feature values ​​for the corresponding dimension; The feature values ​​of each dimension of each data table to be synchronized are concatenated and combined to generate a table profile feature vector corresponding to each data table to be synchronized.

[0009] In one embodiment, the step of classifying all data tables to be synchronized based on the table profile feature vector, and confirming the table type and synchronization task attributes of each data table to be synchronized, includes: The system calls a preset, extensible rule base and iterates through the table profile feature vectors of all data tables to be synchronized, matching them with the rule conditions in the extensible rule base. Match the data tables to be synchronized that meet the corresponding rule conditions with the corresponding category labels, and mark the corresponding synchronization mode constraints and resource limit requirements. All data tables to be synchronized are grouped and categorized according to the classification labels, the table type corresponding to each data table is determined, and the corresponding synchronization task attributes are confirmed according to the synchronization label content.

[0010] In one embodiment, the step of solving for optimal task orchestration based on the table type and synchronization task attributes of each data table to be synchronized using a preset cost model to obtain optimal task grouping and execution priority includes: Based on the table type and synchronization task attributes of each data table to be synchronized, each data table to be synchronized is mapped to a scheduling unit with resource overhead attributes, and a single synchronization task is mapped to an execution container with resource capacity constraints, with one execution container corresponding to one task group. The estimated synchronization time and resource consumption of each scheduling unit are calculated using a preset cost model. The optimization algorithm, under the constraint of the upper limit of the resource capacity of a single execution container, aims to find the optimal solution with the shortest total synchronization time and the minimum total capacity of the required execution containers, and outputs the optimal task group and the execution priority of each task group.

[0011] In one embodiment, the optimization algorithm, under the constraint of the resource capacity limit of a single execution container, aims to minimize the total synchronization time and the total required execution container capacity, and outputs the optimal task grouping and the execution priority of each task group, including: Based on the resource overhead attributes, special scheduling units are selected from all scheduling units, and the special scheduling units are assigned to independent execution containers to obtain special task groups. The remaining ordinary scheduling units are sorted in descending order according to the estimated synchronization time and resource consumption. The first-fit bin packing algorithm is used to perform the optimal task partitioning. Under the constraint of the upper limit of the resource capacity of a single execution container, the ordinary task group with the shortest total synchronization time and the smallest total execution container capacity is obtained. Based on the business priority requirements of each task group, the execution order of all task groups is arranged to determine the execution priority of each task group.

[0012] A data synchronization task orchestration device based on table profiles, comprising: The data acquisition module is used to collect metadata from all tables in the source database that need to be synchronized. The profile building module is used to perform multi-dimensional quantification processing on the metadata and generate a table profile feature vector corresponding to the data table to be synchronized. The rule classification module is used to classify all data tables to be synchronized according to the feature vector of the table profile, and to confirm the table type and synchronization task attributes of each data table to be synchronized. The orchestration optimization module is used to solve for the optimal task orchestration based on the table type and synchronization task attributes of each data table to be synchronized using a preset cost model, so as to obtain the optimal task grouping and execution priority. The task generation and execution module is used to generate corresponding task parameter configurations for each task group, and execute the corresponding synchronous tasks according to the execution priority and the task parameter configurations when the execution instruction is triggered.

[0013] A data synchronization task orchestration system based on table profiles, the system comprising at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the above-described data synchronization task orchestration method based on table profiles.

[0014] A non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the above-described data synchronization task orchestration method based on table profiles.

[0015] Beneficial Effects: This invention discloses a data synchronization task orchestration method, apparatus, system, and medium based on table profiles. Compared to existing technologies, this invention collects metadata from all tables to be synchronized in the source database and performs multi-dimensional quantification to generate corresponding table profile feature vectors. Based on these feature vectors, all tables to be synchronized are categorized to determine their table type and synchronization task attributes. An optimal task orchestration solution is obtained using a preset cost model based on the table type and synchronization task attributes of each table, resulting in optimal task grouping and execution priority. Corresponding task parameter configurations are generated for each task group, and the corresponding synchronization tasks are executed according to the execution priority and task parameter configurations when the execution command is triggered. Through intelligent classification and optimal task orchestration using multi-dimensional table profiles, automated intelligent orchestration and execution of synchronization tasks are achieved, effectively reducing manual costs and configuration barriers, and improving the efficiency and reliability of data synchronization. Attached Figure Description

[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 A schematic diagram of an application environment for the data synchronization task orchestration method based on table profile provided in this embodiment of the invention; Figure 2 A flowchart of a data synchronization task orchestration method based on table profiles provided in an embodiment of the present invention; Figure 3 This is a flowchart of step S201 in the data synchronization task orchestration method based on table profile provided in an embodiment of the present invention; Figure 4 This is a flowchart of step S202 in the data synchronization task orchestration method based on table profile provided in an embodiment of the present invention; Figure 5 This is a flowchart of step S203 in the data synchronization task orchestration method based on table profile provided in an embodiment of the present invention; Figure 6 This is a flowchart of step S204 in the data synchronization task orchestration method based on table profile provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the functional modules of the data synchronization task orchestration device based on table profile provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of a data synchronization task orchestration system based on table profiles provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0018] The data synchronization task orchestration method based on table profiles provided in this invention can be applied to, for example... Figure 1 In the application environment, the data acquisition module in the intelligent task orchestration unit obtains the input information required by the intelligent analysis engine from the source database based on the full whitelist (i.e., the tables to be synchronized) provided by the user. The intelligent analysis engine then uses the extracted table information as input to construct a table profile from multiple dimensions, covering dimensions such as basic size, structural characteristics, special attributes, and load characteristics. Based on the table profile and combined with the current status of synchronization task capabilities, it outputs task orchestration strategies and suggestions through a built-in rule engine and optimization algorithms. The built-in rule engine is a set of whitelist analysis and classification rules built into the synchronization tool, including, for example, distinguishing between ordinary and special tables, whether the number of tables reaches a built-in threshold, table width, whether it contains large field types, whether the rowmovement / nologging attribute is enabled, and whether it is a hot table. The optimization algorithm comprehensively considers factors such as the capabilities of the source and target databases, server resources, and synchronization window period to intelligently solve for the relatively optimal task grouping and priority. After the intelligent analysis engine provides task orchestration strategies and suggestions, the system provides a visual task observation entry point, allowing users to fine-tune tasks to meet minor customization needs. After user confirmation, the corresponding data synchronization task is launched and executed, synchronizing the data from the source database to the target database through the data synchronization units.

[0019] like Figure 2 As shown, the data synchronization task orchestration method based on table profiles provided in this embodiment specifically includes the following steps: S201. Collect metadata of all data tables to be synchronized in the source database.

[0020] In this embodiment, before executing the synchronization task orchestration, the full attribute information of the data table to be synchronized is obtained from the source database as input for intelligent analysis. The source database can be various relational databases such as Oracle and MySQL, while the data table to be synchronized is the business data table that the user needs to synchronize from the source database to the target database. Metadata is statistical information describing the structure, size, attributes, and operational characteristics of the data table itself; it forms the data foundation for subsequent table profiling, rule matching, and optimization. Traditional methods rely on DBAs to manually perform statistical analysis table by table, which is extremely inefficient and prone to errors when the tables are large. This embodiment uses an automated collection method to batch retrieve metadata from all tables, which can significantly reduce the cost of manual investigation while ensuring the accuracy and integrity of the data.

[0021] S202. Perform multi-dimensional quantization processing on the metadata to generate a table profile feature vector corresponding to the data table to be synchronized.

[0022] In this embodiment, since the collected raw metadata consists of scattered heterogeneous indicators containing various data types such as numerical, Boolean, and enumeration, it cannot be directly used for rule determination and algorithm calculation. Therefore, the metadata of all data tables to be synchronized is subjected to multi-dimensional quantization processing, transforming various attributes of each data table into a standardized, computable feature set, thereby generating a table profile feature vector corresponding to the data tables to be synchronized.

[0023] For example, a detailed profile of all tables to be synchronized is constructed from multiple dimensions, including table size, data volume, table structure characteristics (such as non-primary key tables, index-organized tables, whether rowmovement / nologing is enabled, etc.), update characteristics (incremental ratio, change frequency), load characteristics (I / O overhead, read / write hotspots), and business relevance. This generates a quantified feature vector for each table, thereby uniformly quantifying and expressing the differentiated characteristics of different tables, providing standardized input for subsequent intelligent analysis and orchestration. Through this multi-dimensional, detailed profiling, the synchronization difficulty, resource consumption, and potential risks of each table can be accurately identified, ensuring the global optimization capability of intelligent orchestration for synchronization tasks.

[0024] S203. Classify all data tables to be synchronized according to the feature vector of the table profile, and confirm the table type and synchronization task attributes of each data table to be synchronized.

[0025] In this embodiment, after obtaining the feature vectors of all tables, all data tables to be synchronized are first preliminarily classified and labeled using a rule-driven approach. This involves pre-establishing executable judgment rules based on DBA synchronization configuration experience, operational standards, and the capability boundaries of synchronization tools, identifying different types of data tables and classifying and labeling them, clarifying the type of each table, the applicable synchronization mode, and resource limitations, among other synchronization task attributes.

[0026] Preliminary classification and labeling not only improves synchronization efficiency by grouping similar tables into the same task, but also prioritizes the identification of special data tables that pose synchronization compatibility risks, severely impact synchronization stability, or disrupt production operations, thus isolating high-risk tables in advance. Taking the rules corresponding to the capabilities of synchronization tools as an example, in cases where the source database is an Oracle database, if the table profile extracted from the Oracle database identifies index-organized tables with LOB fields, or tables with table names or column names exceeding 30 characters, these tables cannot support real-time incremental synchronization and can only undergo full data migration; otherwise, the synchronization task will directly fail. Classifying and identifying these tables and employing special synchronization modes can improve the success rate of synchronization tasks. Therefore, this embodiment, through pre-classification processing, facilitates targeted processing during intelligent analysis, avoids compatibility conflicts during subsequent algorithm orchestration, and ensures the feasibility and stability of the final generated synchronization task.

[0027] S204. Based on the table type and synchronization task attributes of each data table to be synchronized, the optimal task arrangement is solved by using a preset cost model to obtain the optimal task grouping and execution priority.

[0028] In this embodiment, after classifying and labeling the data tables, global optimization of task orchestration is achieved through mathematical modeling and optimization algorithms, thereby replacing the traditional manual allocation mode. Specifically, this embodiment transforms the synchronization task orchestration problem into a constrained combinatorial optimization problem. Based on the table type and synchronization task attributes of each data table to be synchronized, a preset cost model quantifies the time overhead and resource consumption of single-table synchronization. It comprehensively considers constraints such as source-side extraction pressure, target-side write capability, synchronization window limitations, and business priorities, aiming for the shortest global time consumption and highest efficiency, and automatically plans the optimal task grouping and execution order.

[0029] This embodiment maximizes resource utilization and shortens overall synchronization time by using global algorithm optimization, ensuring that tasks are completed within the predetermined synchronization window. It effectively solves the problems of lack of global coordination and reliance on personal experience in manual orchestration, thereby realizing automated and intelligent orchestration of synchronization tasks and improving the efficiency and intelligence of synchronization task orchestration.

[0030] S205. Generate corresponding task parameter configurations for each task group, and execute the corresponding synchronous tasks according to the execution priority and the task parameter configurations when the execution instruction is triggered.

[0031] In this embodiment, after determining the task group and execution priority, each task group generates an independent synchronization task. The group contains multiple data tables to be synchronized. The system automatically matches and binds the corresponding task parameter configurations for each synchronization task, eliminating the need for manual configuration for each task.

[0032] Different types of task groups correspond to differentiated parameter strategies, thereby specifically adapting to the synchronization characteristics of all data tables within the group. The parameters apply to the entire synchronization task corresponding to the group. For example, the LARGE_LOB task group marked as a large field type is automatically configured with batchSize=500, fetchSize=1MB, and compression algorithm=ZSTD to reduce network transmission overhead; the MUST_THROTTLE task group marked as a hotspot rate limiting type is automatically configured with concurrent thread count=1, synchronization rate limit=10MB / s, and collection window=off-peak to avoid business peaks; the ordinary large table type task group is automatically configured with concurrent thread count=8, batchSize=2000, and no special rate limit to maximize synchronization efficiency.

[0033] Each generated synchronization task contains a complete execution configuration. This configuration includes a whitelist of all data tables within the group, the synchronization mode (full / incremental / full + incremental), the source and target database types and instance information, task execution parameters, component resource specifications, and execution priority. When an execution command is triggered, the corresponding synchronization task is executed according to the execution priority and task parameter configuration. Synchronization task execution can be triggered in two modes: immediate execution upon receipt (starting execution immediately after task generation and user confirmation) and scheduled execution within a pre-defined synchronization window. Both modes automate the entire process from metadata analysis to task execution, significantly improving the intelligent orchestration and execution efficiency of synchronization tasks.

[0034] In one embodiment, after generating corresponding task parameter configurations for each task group, the method further includes: Based on the optimal task grouping, the task parameter configuration and execution priority are used to output a visual task orchestration view; Receive user's fine-tuning instructions for the task orchestration view, adjust the task parameter configuration and / or execution priority according to the fine-tuning instructions, and obtain the final task orchestration result.

[0035] In this embodiment, after automatically analyzing and generating a task orchestration scheme, a visual task orchestration view is output based on the optimal task group's task parameter configuration and execution priority. That is, all task information is displayed to the user in the form of a visual view, including the list of tasks included in each task group, synchronization mode, running parameters, estimated time, resource specifications, and execution order. The visual view adopts a white-box display method, which makes it easy for users to quickly and intuitively understand the task orchestration scheme and logic.

[0036] Building upon the visual display, users can flexibly fine-tune the task orchestration scheme within the visual view, taking into account actual business needs, real-time resource status, and special operational requirements. This includes adjusting the splitting and merging of task groups, modifying operational parameters such as concurrency and rate limits, adjusting task execution order, and adjusting component resource specifications. Upon receiving fine-tuning instructions, the orchestration scheme is updated in real time, resulting in a task orchestration result confirmed by the user.

[0037] This embodiment combines the efficiency advantages of automation with the actual execution requirements through a collaborative mode of automatic orchestration and manual confirmation and fine-tuning. It reduces labor costs while ensuring that the solution meets actual business and operation and maintenance requirements.

[0038] In one embodiment, such as Figure 3 As shown, step S201 includes: S301, Receive the full whitelist submitted by the user; S302. Establish a data connection with the source database based on the full whitelist, and collect metadata of all data tables to be synchronized in the full whitelist.

[0039] In this embodiment, the full whitelist is a list of all data tables specified by the user that need to be synchronized, serving as the basis for the scope of metadata collection. After receiving the whitelist, a connection is established with the source database via standard database connection protocols such as JDBC, and the corresponding metadata is retrieved one by one based on the table names in the whitelist, avoiding the additional performance overhead caused by a full database scan.

[0040] The specific metadata collected includes table data volume, number of columns, average row length, whether the table name and column name exceed 30 characters, whether it contains large fields (such as LOB, TEXT, XMLType, GIS, etc.), whether it is an index-organized table, whether it is a table without a primary key / unique index, whether the row migration attribute is enabled, whether the NOLOGING attribute is enabled, etc.

[0041] This embodiment uses a whitelist for precise targeted data collection, which can accurately obtain the required data while minimizing the impact on the source production business.

[0042] In one embodiment, such as Figure 4As shown, step S202 includes: S401. Extract the metadata according to preset dimensions to obtain basic scale dimension data, structural feature dimension data, unique attribute dimension data and load feature dimension data respectively. S402. Perform corresponding standard quantization calculations on the data of each dimension to generate feature values ​​for the corresponding dimension; S403. Concatenate and combine the feature values ​​of each dimension of each data table to be synchronized to generate the table profile feature vector corresponding to each data table to be synchronized.

[0043] In this embodiment, when constructing the table profile, metadata is extracted from multiple preset dimensions, namely, basic scale dimension data, structural feature dimension data, unique attribute dimension data, and load feature dimension data. The basic scale dimension data includes data volume (estimated row count based on statistics or sampling), table size (occupied storage space), and average row length (estimated row length based on sampling). The structural feature dimension data includes the number of columns (the number of fields in the table), table characteristics (whether it lacks a primary key or unique index; tables without a primary key severely impact synchronization breakpoint resumption, data consistency, and verification performance), column characteristics (whether it contains large fields such as LOB, TEXT, XMLType, and GIS; these fields severely slow down synchronization performance, and related tables need to be handled separately), row migration attributes (whether row movement is enabled; for Oracle databases, enabling it may change the physical address of the data, and if the synchronization tool uses rowid as the consistency key, it will severely damage data consistency), and the NOLOGING attribute (when enabled, table changes will not be written to the archive log, causing CDC to be unable to capture incremental changes).

[0044] Next, standard quantification calculations are performed on the data of each dimension. Specific quantification methods are used for different types of indicators. For example, numerical indicators such as data volume and number of columns are normalized and mapped to standard scores in the 0-1 range; boolean attributes such as whether there is no primary key or whether it contains large fields are weighted and converted into corresponding feature scores, with higher risk resulting in higher weights; and load-related indicators such as update frequency are converted into hot / cold scores through hierarchical mapping. Finally, the feature values ​​of all dimensions are concatenated and combined in a fixed order to form a unique table profile feature vector for each table to be synchronized.

[0045] This embodiment uses standardized feature vector representation to eliminate the dimensional differences between different indicators, providing a unified input format for subsequent rule matching and algorithm solving, and ensuring the objectivity and consistency of the analysis results.

[0046] In one embodiment, such as Figure 5 As shown, step S203 includes: S501. Call the preset expandable rule base, traverse the table profile feature vectors of all data tables to be synchronized, and match them with the rule conditions in the expandable rule base; S502. Match the data tables to be synchronized that meet the corresponding rule conditions with the corresponding category labels, and mark the corresponding synchronization mode constraints and resource limit requirements. S503. Group and classify all data tables to be synchronized according to the classification labels, determine the table type corresponding to each data table, and confirm the corresponding synchronization task attributes according to the synchronization label content.

[0047] In this embodiment, the extensible rule base is a set of configurable and expandable judgment rules, with multiple built-in core classification rules, and the ability to flexibly add rules based on the capabilities of synchronization tools, database types, and business needs. Specific core rules include: 1. Resource protection rules: Match data tables whose update frequency reaches the hotspot threshold with the rate limiting tag MUST_THROTTLE, and mark them with the recommended low concurrency and rate limiting resource restrictions to avoid synchronous tasks occupying production database resources and affecting online business operations; 2. Special table isolation rules: Tables with no primary key and no unique index ("three-no tables"), tables with row movement / nologging enabled, index-organized tables, and tables with table name and column name length exceeding the limit are matched with the special pattern class label SPECIAL_LOGIC, and are marked to indicate that special synchronization modes such as full migration and temporary primary key columns are required to ensure that the synchronization task can be executed normally. 3. Large Field Slowdown Rule: Match data tables containing large fields and those with average row length exceeding the limit with the slowdown category label LARGE_LOB, and suggest reducing the batchSize and network buffer parameter requirements of their channels to avoid large field data slowing down the performance of the entire synchronization channel.

[0048] During rule matching, a traversal approach is adopted, iterating through the table profile feature vectors of all data tables to be synchronized and matching them with each rule condition. Data tables to be synchronized that meet the corresponding rule conditions are matched with the corresponding category labels. After completing the label matching, the data tables are classified according to the label type, and divided into table types such as hotspot rate limiting, special synchronization, large field speed reduction, and ordinary. At the same time, the constraints and requirements corresponding to the labels are used as the synchronization task attributes of the table.

[0049] This embodiment uses a rule engine for pre-classification, which can solidify human experience, operation and maintenance specifications, and the capability boundaries of synchronization tools into automated judgment logic. It groups and classifies all data tables to be synchronized and confirms the corresponding synchronization task attributes, thereby helping to isolate risky tables in advance, avoid synchronization failures and production impacts caused by subsequent orchestration, and improve the stability of the synchronization task orchestration scheme.

[0050] In one embodiment, such as Figure 6 As shown, step S204 includes: S601. Based on the table type and synchronization task attributes of each data table to be synchronized, map each data table to be synchronized to a scheduling unit with resource overhead attributes, and map a single synchronization task to an execution container with resource capacity constraints, where one execution container corresponds to one task group. S602. Calculate the estimated synchronization time and resource consumption of each scheduling unit using a preset cost model. S603. Under the constraint of the upper limit of the resource capacity of a single execution container, the optimization algorithm is used to find the optimal solution with the shortest total synchronization time and the minimum total capacity of the required execution container as the optimization objectives, and outputs the optimal task group and the execution priority of each task group.

[0051] In this embodiment, during intelligent task orchestration, the data synchronization task orchestration problem is abstracted into a constrained bin packing optimization problem, which is quantitatively solved through mathematical modeling. Each data table serves as the smallest unit of scheduling, mapped to a scheduling unit with resource overhead attributes, including data volume, load consumption, and category labels. A single synchronization task serves as the smallest unit of execution, mapped to an execution container with a resource capacity limit. An execution container can hold multiple scheduling units, corresponding to a task group containing multiple data tables to be synchronized. The resource capacity limit of the execution container is determined by the capability limit of the synchronization tool, server hardware resources, and database load thresholds, including maximum network bandwidth, memory, disk I / O, and the number of CPU cores. This capacity limit constraint ensures that a single synchronization task will not exceed the resource carrying capacity limit.

[0052] The synchronization overhead of a single scheduling unit is quantified by a preset cost model. The estimated synchronization time of a single table satisfies the formula: Cost(Table)=f(data volume, index complexity, large field impact factor, rate limiting coefficient). That is, by comprehensively considering the data volume of the table, the number and complexity of indexes, the performance degradation coefficient brought by large fields, and the rate limiting coefficient brought by rate limiting rules, the estimated synchronization time of a single table and the corresponding resource consumption are calculated.

[0053] Ultimately, an optimization algorithm is used to find the optimal solution under constraints, aiming to minimize the total synchronization time and the total required execution container capacity. This results in the optimal task grouping and execution priority of each task group. The goal is to use the fewest / optimal synchronization tasks while satisfying all constraints, achieving the shortest total synchronization completion time and highest efficiency. Minimizing the total synchronization time ensures tasks can be completed within the predetermined synchronization window, meeting business timeliness requirements; minimizing the total required execution container capacity maximizes resource utilization, avoiding unnecessary task splitting and resource waste. This embodiment achieves optimal orchestration from a global perspective through modeling and solving, overcoming the limitations of manual experience-based orchestration, thereby improving synchronization efficiency and the rationality of resource utilization.

[0054] In one embodiment, step S603 includes: Based on the resource overhead attributes, special scheduling units are selected from all scheduling units, and the special scheduling units are assigned to independent execution containers to obtain special task groups. The remaining ordinary scheduling units are sorted in descending order according to the estimated synchronization time and resource consumption. The first-fit bin packing algorithm is used to perform the optimal task partitioning. Under the constraint of the upper limit of the resource capacity of a single execution container, the ordinary task group with the shortest total synchronization time and the smallest total execution container capacity is obtained. Based on the business priority requirements of each task group, the execution order of all task groups is arranged to determine the execution priority of each task group.

[0055] In this embodiment, the resource overhead attributes of each scheduling unit include the data volume, load consumption, and classification tags of the data table to be synchronized. Based on the resource overhead attributes, special scheduling units are selected from all scheduling units. These special scheduling units refer to data tables with significantly higher resource overhead than ordinary tables, which have synchronization compatibility constraints or will have a significant impact on the synchronization channel or production database. Examples include tables with synchronization compatibility risks that are not synchronized, tables with no row migration enabled or no log attributes, index-organized tables, tables with table name and column name lengths exceeding the limit; wide tables containing multiple large fields and with average row lengths exceeding the limit; extremely large tables with extremely large data volumes; and hot tables with extremely high read and write activity, etc.

[0056] Based on the selected special scheduling units, each or every type of special scheduling unit with the same attributes is assigned an independent execution container to form a special task group. By isolating special scheduling units in advance, high-overhead and high-risk tables can be avoided from slowing down the entire synchronization channel or causing data consistency problems. This ensures the stability and compatibility of synchronization tasks from the task orchestration level, and also eliminates the interference of special constraints on the subsequent bin packing solution of ordinary units.

[0057] After the isolation and allocation of special scheduling units are completed, the remaining ordinary scheduling units all meet the compatibility requirements of the ordinary synchronization mode and their resource overhead is within the normal range. Therefore, the first-fit bin packing algorithm is used to achieve optimal batch grouping. First, the ordinary scheduling units are sorted in descending order according to resource overhead indicators, including estimated synchronization time and resource occupancy, so that scheduling units with higher resource overhead can be processed first. Then, the first-fit bin packing algorithm is used to traverse the scheduling units one by one, and the current scheduling unit is placed into the first execution container that can accommodate it and does not exceed the capacity limit. If all existing execution containers cannot accommodate it, a new execution container is opened. Under the constraint of the upper limit of the resource capacity of a single execution container, the ordinary task group with the shortest total synchronization time and the smallest total execution container capacity is obtained, ensuring that the synchronization tasks corresponding to each generated ordinary task group will not exceed the resource carrying capacity limit.

[0058] The resulting ordinary task groups, together with the aforementioned special task groups, constitute a complete task grouping scheme. Under the premise of satisfying all constraints, this scheme achieves the dual objectives of minimizing the total capacity of the execution container and optimizing the overall synchronization time. Then, based on the business priority requirements of the data tables within each task group, all task groups are arranged in a unified execution order. Task groups with higher business priority are executed earlier, ensuring that core business data is synchronized first, meeting the timeliness requirements of the business side.

[0059] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0060] Another embodiment of the present invention provides a data synchronization task orchestration device based on table profiles, such as... Figure 7 As shown, device 1 includes: Data acquisition module 11 is used to collect metadata of all data tables to be synchronized in the source database; The profile building module 12 is used to perform multi-dimensional quantification processing on the metadata and generate a table profile feature vector corresponding to the data table to be synchronized. The rule classification module 13 is used to classify all data tables to be synchronized according to the table profile feature vector, and to confirm the table type and synchronization task attributes of each data table to be synchronized. The orchestration optimization module 14 is used to solve the optimal task orchestration based on the table type and synchronization task attributes of each data table to be synchronized using a preset cost model, so as to obtain the optimal task grouping and execution priority. The task generation and execution module 15 is used to generate corresponding task parameter configurations for each task group, and execute the corresponding synchronous tasks according to the execution priority and the task parameter configurations when the execution instruction is triggered.

[0061] The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the execution process of data synchronization task orchestration based on table profiles. For specific implementation methods of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0062] Another embodiment of the present invention provides a data synchronization task orchestration system based on table profiles, such as... Figure 8 As shown, system 10 includes: One or more processors 110 and memory 120, Figure 8 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0063] Processor 110 is used to perform various control logics of system 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0064] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the data synchronization task orchestration method based on table profiles in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the data synchronization task orchestration method based on table profiles in the above method embodiment.

[0065] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the system 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps: Collect metadata from all tables in the source database that need to be synchronized; The metadata is subjected to multi-dimensional quantization processing to generate a table profile feature vector corresponding to the data table to be synchronized; Based on the feature vector of the table profile, all data tables to be synchronized are classified to confirm the table type and synchronization task attributes of each data table to be synchronized. The optimal task arrangement is solved by using a preset cost model based on the table type and synchronization task attributes of each data table to be synchronized, and the optimal task grouping and execution priority are obtained. For each task group, generate corresponding task parameter configurations, and when the execution instruction is triggered, execute the corresponding synchronous task according to the execution priority and the task parameter configurations.

[0067] In one embodiment, after generating corresponding task parameter configurations for each task group, the method further includes: Based on the optimal task grouping, the task parameter configuration and execution priority are used to output a visual task orchestration view; Receive user's fine-tuning instructions for the task orchestration view, adjust the task parameter configuration and / or execution priority according to the fine-tuning instructions, and obtain the final task orchestration result.

[0068] In one embodiment, the metadata of all data tables to be synchronized in the source database includes: Receive the full whitelist submitted by the user; A data connection is established with the source database based on the full whitelist, and metadata of all data tables to be synchronized in the full whitelist is collected.

[0069] In one embodiment, the step of performing multi-dimensional quantization processing on the metadata to generate a table profile feature vector corresponding to the data table to be synchronized includes: The metadata is extracted according to preset dimensions to obtain basic scale dimension data, structural feature dimension data, unique attribute dimension data and load feature dimension data respectively. Perform corresponding standard quantization calculations on the data of each dimension to generate feature values ​​for the corresponding dimension; The feature values ​​of each dimension of each data table to be synchronized are concatenated and combined to generate a table profile feature vector corresponding to each data table to be synchronized.

[0070] In one embodiment, the step of classifying all data tables to be synchronized based on the table profile feature vector, and confirming the table type and synchronization task attributes of each data table to be synchronized, includes: The system calls a preset, extensible rule base and iterates through the table profile feature vectors of all data tables to be synchronized, matching them with the rule conditions in the extensible rule base. Match the data tables to be synchronized that meet the corresponding rule conditions with the corresponding category labels, and mark the corresponding synchronization mode constraints and resource limit requirements. All data tables to be synchronized are grouped and categorized according to the classification labels, the table type corresponding to each data table is determined, and the corresponding synchronization task attributes are confirmed according to the synchronization label content.

[0071] In one embodiment, the step of solving for optimal task orchestration based on the table type and synchronization task attributes of each data table to be synchronized using a preset cost model to obtain optimal task grouping and execution priority includes: Based on the table type and synchronization task attributes of each data table to be synchronized, each data table to be synchronized is mapped to a scheduling unit with resource overhead attributes, and a single synchronization task is mapped to an execution container with resource capacity constraints, with one execution container corresponding to one task group. The estimated synchronization time and resource consumption of each scheduling unit are calculated using a preset cost model. The optimization algorithm, under the constraint of the upper limit of the resource capacity of a single execution container, aims to find the optimal solution with the shortest total synchronization time and the minimum total capacity of the required execution containers, and outputs the optimal task group and the execution priority of each task group.

[0072] In one embodiment, the optimization algorithm, under the constraint of the resource capacity limit of a single execution container, aims to minimize the total synchronization time and the total required execution container capacity, and outputs the optimal task grouping and the execution priority of each task group, including: Based on the resource overhead attributes, special scheduling units are selected from all scheduling units, and the special scheduling units are assigned to independent execution containers to obtain special task groups. The remaining ordinary scheduling units are sorted in descending order according to the estimated synchronization time and resource consumption. The first-fit bin packing algorithm is used to perform the optimal task partitioning. Under the constraint of the upper limit of the resource capacity of a single execution container, the ordinary task group with the shortest total synchronization time and the smallest total execution container capacity is obtained. Based on the business priority requirements of each task group, the execution order of all task groups is arranged to determine the execution priority of each task group.

[0073] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the operations described above. Figure 2 The method steps S201 to S205.

[0074] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0075] In summary, the data synchronization task orchestration method, apparatus, system, and medium based on table profiles disclosed in this invention involve: The method collects metadata from all tables to be synchronized in the source database and performs multi-dimensional quantification to generate corresponding table profile feature vectors; All tables to be synchronized are categorized based on these feature vectors to determine the table type and synchronization task attributes of each table; Optimal task orchestration is solved using a preset cost model based on the table type and synchronization task attributes of each table to obtain optimal task grouping and execution priority; Corresponding task parameter configurations are generated for each task group, and the corresponding synchronization tasks are executed according to the execution priority and task parameter configurations when the execution command is triggered. Through intelligent classification and optimal task orchestration using multi-dimensional table profiles, automated and intelligent orchestration of synchronization tasks is achieved, effectively reducing manual costs and configuration barriers, and improving the efficiency and reliability of data synchronization.

[0076] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0077] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A data synchronization task orchestration method based on table-based profiles, characterized in that, include: Collect metadata from all tables in the source database that need to be synchronized; The metadata is subjected to multi-dimensional quantization processing to generate a table profile feature vector corresponding to the data table to be synchronized; Based on the feature vector of the table profile, all data tables to be synchronized are classified to confirm the table type and synchronization task attributes of each data table to be synchronized. The optimal task arrangement is solved by using a preset cost model based on the table type and synchronization task attributes of each data table to be synchronized, and the optimal task grouping and execution priority are obtained. For each task group, generate corresponding task parameter configurations, and when the execution instruction is triggered, execute the corresponding synchronous task according to the execution priority and the task parameter configurations.

2. The data synchronization task orchestration method based on table profiles according to claim 1, characterized in that, After generating corresponding task parameter configurations for each task group, the method further includes: Based on the optimal task grouping, the task parameter configuration and execution priority are used to output a visual task orchestration view; Receive user's fine-tuning instructions for the task orchestration view, adjust the task parameter configuration and / or execution priority according to the fine-tuning instructions, and obtain the final task orchestration result.

3. The data synchronization task orchestration method based on table-based profiles according to claim 1, characterized in that, The metadata of all tables to be synchronized in the source database is collected, including: Receive the full whitelist submitted by the user; A data connection is established with the source database based on the full whitelist, and metadata of all data tables to be synchronized in the full whitelist is collected.

4. The data synchronization task orchestration method based on table-based profiles according to claim 1, characterized in that, The step of performing multi-dimensional quantization processing on the metadata to generate a table profile feature vector corresponding to the data table to be synchronized includes: The metadata is extracted according to preset dimensions to obtain basic scale dimension data, structural feature dimension data, unique attribute dimension data and load feature dimension data respectively. Perform corresponding standard quantization calculations on the data of each dimension to generate feature values ​​for the corresponding dimension; The feature values ​​of each dimension of each data table to be synchronized are concatenated and combined to generate a table profile feature vector corresponding to each data table to be synchronized.

5. The data synchronization task orchestration method based on table-based profiles according to claim 1, characterized in that, The step involves classifying all data tables to be synchronized based on the table profile feature vector, confirming the table type and synchronization task attributes of each data table to be synchronized, including: The system calls a preset, extensible rule base and iterates through the table profile feature vectors of all data tables to be synchronized, matching them with the rule conditions in the extensible rule base. Match the data tables to be synchronized that meet the corresponding rule conditions with the corresponding category labels, and mark the corresponding synchronization mode constraints and resource limit requirements. All data tables to be synchronized are grouped and categorized according to the classification labels, the table type corresponding to each data table is determined, and the corresponding synchronization task attributes are confirmed according to the synchronization label content.

6. The data synchronization task orchestration method based on table profiles according to claim 1, characterized in that, The optimal task orchestration is solved by using a preset cost model based on the table type and synchronization task attributes of each data table to be synchronized, to obtain the optimal task grouping and execution priority, including: Based on the table type and synchronization task attributes of each data table to be synchronized, each data table to be synchronized is mapped to a scheduling unit with resource overhead attributes, and a single synchronization task is mapped to an execution container with resource capacity constraints, with one execution container corresponding to one task group. The estimated synchronization time and resource consumption of each scheduling unit are calculated using a preset cost model. The optimization algorithm, under the constraint of the upper limit of the resource capacity of a single execution container, aims to find the optimal solution with the shortest total synchronization time and the minimum total capacity of the required execution containers, and outputs the optimal task group and the execution priority of each task group.

7. The data synchronization task orchestration method based on table-based profiles according to claim 6, characterized in that, The optimization algorithm, under the constraint of the resource capacity limit of a single execution container, aims to minimize the total synchronization time and the total required execution container capacity, and outputs the optimal task grouping and the execution priority of each task group, including: Based on the resource overhead attributes, special scheduling units are selected from all scheduling units, and the special scheduling units are assigned to independent execution containers to obtain special task groups. The remaining ordinary scheduling units are sorted in descending order according to the estimated synchronization time and resource consumption. The first-fit bin packing algorithm is used to perform the optimal task partitioning. Under the constraint of the upper limit of the resource capacity of a single execution container, the ordinary task group with the shortest total synchronization time and the smallest total execution container capacity is obtained. Based on the business priority requirements of each task group, the execution order of all task groups is arranged to determine the execution priority of each task group.

8. A data synchronization task orchestration device based on table-based profiles, characterized in that, include: The data acquisition module is used to collect metadata from all tables in the source database that need to be synchronized. The profile building module is used to perform multi-dimensional quantification processing on the metadata and generate a table profile feature vector corresponding to the data table to be synchronized. The rule classification module is used to classify all data tables to be synchronized according to the feature vector of the table profile, and to confirm the table type and synchronization task attributes of each data table to be synchronized. The orchestration optimization module is used to solve for the optimal task orchestration based on the table type and synchronization task attributes of each data table to be synchronized using a preset cost model, so as to obtain the optimal task grouping and execution priority. The task generation and execution module is used to generate corresponding task parameter configurations for each task group, and execute the corresponding synchronous tasks according to the execution priority and the task parameter configurations when the execution instruction is triggered.

9. A data synchronization task orchestration system based on table-based profiles, characterized in that, The system includes at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data synchronization task orchestration method based on table profiles as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the data synchronization task orchestration method based on table profiles as described in any one of claims 1-7.

Images