Data synchronization method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data synchronization method, device and equipment and a computer readable storage medium, which are applied to the technical field of data processing, and the method comprises the following steps: converting a data definition language event and a data manipulation language event into a data definition structured statement and a data manipulation structured statement; determining whether the data manipulation structured statement depends on the data definition structured statement or not by utilizing a dependency analysis model; and when the data manipulation structured statement does not depend on the data definition structured statement, determining to synchronously execute the data definition language event and the data manipulation language event. Compared with the condition that one data definition language event blocks all data manipulation language events during current data synchronization, the method and the device have the advantages that the dependency relationship between the data definition language event and the data manipulation language event can be determined through the dependency analysis model; therefore, the data definition language event and the data manipulation language event can be synchronized at the same time in the target library, and the data synchronization efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Most existing real-time synchronization tools rely on CDC (Change Data Capture) components to capture change logs from the source database. While traditional CDC technology can efficiently capture DML (Data Manipulation Language) changes in the source database, when DDL (Data Definition Language) changes occur (such as ALTERTABLE ADD COLUMN, DROP INDEX, RENAME TABLE), existing CDC solutions are prone to order errors if DML and DDL operations are executed concurrently, leading to data inconsistencies or write failures. Furthermore, a single DDL operation can block all other DML operations.

[0003] It is evident that improving the efficiency of data synchronization is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data synchronization method, apparatus, device and computer-readable storage medium, which solves the technical problem of low data synchronization efficiency in the prior art.

[0005] To address the aforementioned technical problems, this invention provides a data synchronization method, comprising:

[0006] Convert Data Definition Language events and Data Manipulation Language events into Data Definition Language structured statements and Data Manipulation Language structured statements;

[0007] Dependency analysis model is used to determine whether the data manipulation structured statement depends on the data definition structured statement;

[0008] When the data manipulation structured statement does not depend on the data definition structured statement, the data definition language event and the data manipulation language event are executed synchronously.

[0009] Optionally, a dependency analysis model is used to determine whether the data manipulation structured statement depends on the data definition structured statement, including:

[0010] Get dependency analysis suggestions;

[0011] Based on the dependency analysis prompts, the dependency analysis model is used to determine whether the data manipulation structured statement depends on the data definition structured statement; wherein, the dependency analysis model is a large language model.

[0012] Optionally, a dependency analysis model is used to determine whether the data manipulation structured statement depends on the data definition structured statement, including:

[0013] Determine the dependency type; wherein, the dependency type includes state dependency and temporal dependency; the state dependency means that the execution of a Data Manipulation Language (DML) event logically depends on the table structure changes caused by the Data Definition Language (DDL) event taking effect; the temporal dependency means that a DML event must be executed after the execution of the DDL event it depends on has been completed;

[0014] The dependency analysis model and the dependency type are used to determine whether the data manipulation structured statement depends on the data definition structured statement.

[0015] Optionally, after determining whether the data manipulation structured statement depends on the data definition structured statement using a dependency analysis model, the method further includes:

[0016] When it is determined that the data manipulation structured statement depends on the data definition structured statement, the target data definition structured statement that the data manipulation structured statement depends on is determined.

[0017] Semantic parsing is performed on the target data definition language events corresponding to the target data definition structured statements to obtain semantic parsing results;

[0018] The semantic parsing results are converted into target format data definition language events required when the target library is executed.

[0019] Data synchronization is achieved by executing the target format data definition language event in the target library.

[0020] When the target data definition language event is completed, the data manipulation language event is synchronized.

[0021] Optionally, after converting the semantic parsing result into the target format data definition language event required for the target library execution, the method further includes:

[0022] The target format data definition language event is subjected to syntax validation to obtain the syntax-validated event;

[0023] The target library is schema-modified based on the events that have undergone syntax validation.

[0024] Optionally, semantic parsing is performed on the target data definition language events corresponding to the target data definition structured statements to obtain semantic parsing results, including:

[0025] Determine semantic parsing prompts based on the source and target databases used during data synchronization;

[0026] Based on the semantic parsing prompts, a large language model is used to perform semantic parsing on the target data definition language events corresponding to the target data definition structured statements, and the semantic parsing results are obtained.

[0027] Optionally, when the data manipulation structured statement does not depend on the data definition structured statement, after determining that the data definition language event and the data manipulation language event are executed synchronously, the method further includes:

[0028] Obtain the manually corrected dependency information corresponding to the currently stated data manipulation structured statement;

[0029] The dependency analysis model is optimized based on the manually corrected dependency information to obtain an optimized dependency analysis model.

[0030] This invention also provides a data synchronization device, comprising:

[0031] The conversion module is used to convert Data Definition Language events and Data Manipulation Language events into Data Definition Language structured statements and Data Manipulation Language structured statements.

[0032] The dependency analysis module is used to determine, using a dependency analysis model, whether the data manipulation structured statement depends on the data definition structured statement;

[0033] The data synchronization module is used to determine the synchronous execution of the data definition language event and the data manipulation language event when the data manipulation structured statement does not depend on the data definition structured statement.

[0034] This invention also provides a data synchronization device, comprising:

[0035] Memory, used to store computer programs;

[0036] A processor for executing the computer program to implement the steps of the data synchronization method described above.

[0037] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data synchronization method.

[0038] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described data synchronization method.

[0039] As can be seen, this invention converts Data Definition Language (DDL) events and Data Manipulation Language (DML) events into Data Definition Structured Statements (DMLs) and DMLs, respectively. It then uses a dependency analysis model to determine whether DMLs depend on DDLs. When a DML does not depend on a DDL, it determines to synchronize DDL events and DML events. Compared to the current situation where a single DDL blockes all DML operations during data synchronization, this invention, through dependency analysis, can determine the dependencies between DDL events and DML events. This allows for simultaneous synchronization of events that can be synchronized in the target database at the same time. Since there is no need to wait for the DDL to complete before executing DML, this improves the efficiency of data synchronization.

[0040] In addition, the present invention also provides a data synchronization device, apparatus, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart for format conversion of Data Definition Language events provided in an embodiment of the present invention;

[0044] Figure 3 A flowchart illustrating a data synchronization method provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Some terms that appear in the description of the embodiments of this application are subject to the following interpretation:

[0049] Real-time Data Synchronization Task: Create a task on the platform to support the continuous and automatic replication of data changes from the source system to the target system within a second-level latency.

[0050] CDC (Change Data Capture) tools: technologies that identify and record data changes in databases and deliver such changes to different target systems in real time.

[0051] Message Queues: As asynchronous buffers, they decouple producers and consumers, ensuring reliable data transmission, such as Kafka and Pulsar.

[0052] Data execution agent: As a data consumer, it processes data streams in real time, supporting the conversion, filtering, merging, and replication of data in various formats to the target system.

[0053] A Large Language Model (LLM) is a deep learning model trained on a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text.

[0054] Please refer to Figure 1 , Figure 1 A flowchart illustrating a data synchronization method provided in an embodiment of the present invention. The method may include:

[0055] S101 converts Data Definition Language events and Data Manipulation Language events into Data Definition Structured Statements and Data Manipulation Structured Statements.

[0056] The steps in this embodiment can be executed by a designated electronic device, which may specifically be a server, a portable terminal, or other forms. This embodiment does not limit the specific form in which Data Definition Language (DDL) and Data Manipulation Language (DML) events are converted into Data Definition Structured Statements and Data Manipulation Structured Statements. For example, this embodiment can convert DDL and DML into SQL (Data Structure Language) statements, and also into XML (Structured Text) format or JSON format (JavaScript Object Notation).

[0057] S102, use the dependency analysis model to determine whether the data manipulation structured statement depends on the data definition structured statement.

[0058] This embodiment does not limit the specific dependency analysis model. For example, this embodiment can be based on an AST-based analysis model: parsing DML and DDL statements into an Abstract Syntax Tree (AST). It analyzes the tables, columns, and other objects referenced in the DML statements and matches them with the table structure modified by the DDL statements. For example, if the DDL statement adds a column, and the DML statement references this column, then a dependency exists. Alternatively, the dependency analysis model in this embodiment can also be a large language model. Or a graph dependency graph model can be used, constructing a graph where nodes represent database objects (tables, columns, etc.) and edges represent dependencies. When a DDL statement modifies an object, all DML statements that depend on that object will be marked as having a dependency.

[0059] It should be further noted that, based on any of the above embodiments, the determination of whether a data manipulation structured statement depends on the data definition structured statement using a dependency analysis model may include:

[0060] S1021, Obtain dependency analysis prompts;

[0061] S1022, Based on dependency analysis prompts, use a dependency analysis model to determine whether a data manipulation structured statement depends on a data definition structured statement; wherein, the dependency analysis model is a large language model.

[0062] This embodiment does not limit the specific dependency analysis prompt words. For example, the dependency analysis prompt words could be: "You are a database expert." Please determine whether the following SQL operations have an execution order dependency, with the following requirements:

[0063] (1) SQL to be judged: {SQL statement, supports batch processing};

[0064] (2) The result of the judgment is returned in JSON format. The return format is as follows: {"depends":false,"reason":"DML does not involve any new fields, no need to wait for DDL to complete"}, where the depends field represents the execution order dependency and the reason field represents the detailed dependency description.

[0065] This embodiment transforms DDL and DML events into structured SQL statements, submits the structured SQL to a large language model for semantic recognition, and then analyzes dependencies. This embodiment is not limited to a specific large language model. For example, the large language model in this embodiment could be GPT, Wenxin Yiyan, or Tongyi Qianwen. Because it does not rely on a predefined rule base, it can generalize to handle unknown DDL change events.

[0066] It should be further explained that, based on any of the above embodiments, the determination of whether a data manipulation structured statement depends on a data definition structured statement using a dependency analysis model may include: determining the dependency type; wherein, the dependency type includes state dependency and temporal dependency; state dependency means that the successful execution of a data manipulation language event logically depends on the table structure change caused by the data definition language event taking effect; temporal dependency means that a data manipulation language event must be executed after the execution of its dependent data definition language event is completed; and the determination of whether a data manipulation structured statement depends on a data definition structured statement using a dependency analysis model and dependency type. This embodiment provides multiple dependency types, thereby enabling more accurate dependency analysis and improving the accuracy of dependency analysis.

[0067] S103, when the data manipulation structured statement does not depend on the data definition structured statement, determine to execute the data definition language event and the data manipulation language event synchronously.

[0068] In this embodiment, when it is determined that the data manipulation structured statement does not depend on the data definition structured statement, it can be executed synchronously without execution errors leading to data inconsistency. When it is determined that the data manipulation structured statement depends on the data definition structured statement, the data definition language event can be executed first, followed by the data manipulation language event. This embodiment leverages the powerful semantic understanding capabilities of the large model to achieve intelligent translation of DDL statements during the execution of data semantic language events. This effectively solves the translation challenges of accurate DDL semantic conversion and the difficulty in exhaustively enumerating rules during heterogeneous database adaptation, caused by complex data type mappings, significant differences in syntax structures, and incompatibility between constraints and indexing mechanisms. It also addresses the issue of traditional real-time synchronization tools directly ignoring DDL changes, providing a data foundation for subsequent automatic DDL processing.

[0069] Given that existing real-time synchronization tools in the industry have limited support for fault tolerance in handling DDL changes, and some even ignore them altogether, manual intervention is required to perform equivalent operations on the target database, which greatly reduces the timeliness of data synchronization. DDL changes often involve the evolution of table structures, while downstream heterogeneous databases have significant differences in data types, syntax specifications, and constraint mechanisms, making it impossible to directly reuse the DDL statements from the source database.

[0070] It should be further noted that, based on any of the above embodiments, after determining whether a data manipulation structured statement depends on a data definition structured statement using a dependency analysis model, the following may also be included:

[0071] S201: When it is determined that a data manipulation structured statement depends on a data definition structured statement, determine the target data definition structured statement that the data manipulation structured statement depends on.

[0072] S202: Perform semantic parsing on the target data definition language events corresponding to the target data definition structured statements to obtain the semantic parsing results;

[0073] S203: Convert the semantic parsing results into target format data definition language events required by the target library during execution;

[0074] S204: Execute the Target Format Data Definition Language event in the target library to achieve data synchronization;

[0075] S205: When the target data definition language event has finished executing, synchronize the data manipulation language event.

[0076] Please refer to the above process. Figure 2 , Figure 2This is a flowchart illustrating a format conversion process for Data Definition Language (DDL) events, provided as an embodiment of the present invention. This embodiment does not limit the specific semantic parsing method; for example, the semantic parsing method in this embodiment can be a machine learning-based semantic parsing model; or the semantic parsing model in this embodiment can be a large language model. This embodiment utilizes the CDC tool to parse the original DDL statement change logs from different data sources into a unified format, including metadata such as operation type, object name, field definition, and constraint information, which are sequentially written to a message queue. The data execution agent consumes DDL changes from the message queue and invokes a large language model for semantic parsing. Relying on the powerful cross-language understanding and code generation capabilities of large language models (such as Qwen and ChatGLM), semantic-level translation of DDL statements is achieved (translating the consumed unified DDL format into the form required by the target database), rather than simple string replacement. The following code snippet demonstrates how Python can be used to call a Large Language Model (LLM) to generate a target Data Definition Language (DDL) event: `response = llm.generate(prompt); translated_ddl = "ALTER TABLE "users" ADD COLUMN "profile" CLOB CHECK (ISJSON("profile") = 1); COMMENT ON COLUMN "users". "profile" IS 'User Profile';"`. The first line of code indicates that the LLM is invoked to generate a response, and the prompt may contain instructions to convert the source database's (e.g., MySQL) DDL to Oracle DDL. The second line of code shows the expected conversion result: a DDL statement conforming to Oracle syntax. After the generated DDL event is returned to the Data Execution Agent, the agent automatically executes it directly on the target database via JDBC, thus automating the DDL statement modification process. Compared to industry practices that largely rely on predefined rules and database processors for automatic DDL parsing, this embodiment leverages the powerful semantic understanding capabilities of a large model to achieve intelligent translation of DDL statements. This effectively solves the translation challenges of accurate DDL semantic conversion and the difficulty in exhaustively enumerating rules during heterogeneous database adaptation, caused by complex data type mappings, significant differences in syntax structures, and incompatibility between constraints and indexing mechanisms. It also addresses the issue of traditional real-time synchronization tools directly ignoring DDL changes, providing a data foundation for subsequent automatic DDL processing.

[0077] It should be further explained that, based on any of the above embodiments, after converting the semantic parsing result into target format data definition language events required for execution of the target library, the process may further include: performing syntax verification on the target format data definition language events to obtain the syntax-verified events; and performing schema change on the target library based on the syntax-verified events. The syntax verification in this embodiment includes one or more of the following: keyword validity check, data type compatibility verification, database constraint syntax check, and version feature support verification. The schema change operation in this embodiment includes acquiring a database lock, verifying schema version consistency, executing data definition language statements, and updating the schema registry. This embodiment ensures that the converted target library DDL statements are syntactically valid through syntax verification, avoiding execution failures and synchronization interruptions due to syntax errors. This embodiment uses this verified DDL statement to perform schema change on the target library (i.e., executes the DDL statement on the target library). The entire process is: Syntax verification: Checking whether the syntax of the DDL statement is correct. Schema change: Executing the DDL statement on the target library. This avoids the step of internally updating metadata, or the step of internally updating metadata may be included in the schema change (for example, after the schema change is successful, the system updates the metadata). The schema change includes changes to the table structure and changes to the internal metadata (i.e., field mapping relationships).

[0078] It should be further explained that, based on any of the above embodiments, semantic parsing of the target data definition language events corresponding to the above target data definition structured statements, to obtain semantic parsing results, may include: determining semantic parsing prompt words constructed based on the source database and target database corresponding to the data synchronization; and using a large language model based on the semantic parsing prompt words to perform semantic parsing of the target data definition language events corresponding to the target data definition structured statements, to obtain semantic parsing results. The intelligent semantic parsing implementation in this embodiment is as follows: 1. Prompt word design: You are a database expert. Please convert the following MySQL DDL statement into semantically equivalent DM database syntax: ALTER TABLE users ADD COLUMNprofile JSON COMMENT 'User Profile'; 2. Requirements: Use CLOB type to store JSON and add ISJSON validation constraints; retain field comments; follow DM naming conventions (e.g., identifiers enclosed in double quotes); output only contains SQL statements, no interpretation required. This embodiment utilizes a large model to understand the context and handle complex semantics, requiring only a "style description" of the target database to quickly support new database types.

[0079] It should be further explained that, based on any of the above embodiments, when the data manipulation structured statement does not depend on the data definition structured statement, after determining that the data definition language event and the data manipulation language event are executed synchronously, the method may further include: obtaining manually corrected dependency information corresponding to the current data manipulation structured statement; optimizing the dependency analysis model based on the manually corrected dependency information to obtain an optimized dependency analysis model. In this embodiment, obtaining the manually corrected dependency information corresponding to the current data manipulation structured statement means that when the time dependency between the data manipulation structured statement and the data definition language is abnormal, manual correction will be performed to obtain manually corrected dependency information. The manually corrected data manipulation structured statement is the current data manipulation structured statement, i.e., the manually corrected dependency information corresponding to historical data manipulation language events and data definition language events. The dependency analysis model provided in this embodiment supports dynamic learning, feeding back the transformation results of manual correction to the model to continuously optimize accuracy.

[0080] This invention provides a data synchronization method that may include: S101, converting Data Definition Language (DDL) events and Data Manipulation Language (DML) events into Data Definition Structured Statements (DML) and DML statements; S102, using a dependency analysis model to determine whether a DML statement depends on a DDL statement; and S103, when a DML statement does not depend on a DML statement, determining to synchronize the execution of the DDL events and DML events. Compared to the current situation where a single DDL blockes all DML operations during data synchronization, this invention can determine the dependency relationship between DDL events and DML events through a dependency analysis model. This allows for simultaneous synchronization of events that can be synchronized in the target database at the same time. Since there is no need to wait for the DDL to complete before executing the DML, the efficiency of data synchronization can be improved.

[0081] For a clearer understanding of this invention, please refer to the following details. Figure 3 , Figure 3 This flowchart illustrates a data synchronization method provided in this embodiment of the invention, describing the complete process of real-time data synchronization from a source database (such as MySQL) to a target database (such as Oracle). The core innovation lies in the introduction of an intelligent scheduling layer (driven by a large model), which can accurately analyze the dependencies between DML and DDL operations, avoiding common problems in traditional CDC tools such as disordered ordering, data inconsistency, or blocking.

[0082] Specifically, this may include:

[0083] Step 1: The data execution agent consumes events from the message queue and takes different paths depending on the event type.

[0084] Specifically, traditional databases, acting as data sources, generate data changes (DML operations) and structural changes (DDL operations). The CDC tool parses the database's binary logs (such as binlog) in real time, categorizing change events into DML and DDL events. The message queue CDC writes the parsed events into a queue (such as Kafka, a distributed stream processing platform) in an orderly manner, achieving decoupling and buffering to ensure high throughput. The data execution broker consumes events from the message queue and takes different paths based on the event type: DML event processing path: First, it checks if there are any pending DDL events in the cache. If no relevant DDL exists, the DML is executed directly. If a DDL exists, a dependency analysis is performed on the large model: determining whether the DML depends on an incomplete DDL (e.g., whether the DML operation involves adding columns in the DDL). If dependent, the DML enters the waiting queue until the DDL is completed. If not dependent, the DML is executed immediately, avoiding unnecessary blocking.

[0085] Step 2: Call the large model (large language model) to perform DDL transformation, converting the DDL syntax of the source database (such as MySQL) into the syntax of the target database (such as Oracle).

[0086] Step 3: Perform syntax validation on the converted DDL to ensure correctness.

[0087] Step 4: Execute the DDL and, upon success, update the schema change state (update metadata, i.e., the field mapping relationship between the source table and the target table) (clear the DDL flag in the cache) and release the DML events in the waiting queue that depend on the DDL.

[0088] Step 5: Execution failures (DML or DDL) are placed in a waiting queue and automatically retried after a delay to ensure the system recovers from temporary errors.

[0089] In this embodiment, all changes were correctly applied to the target database.

[0090] The beneficial effects of this invention include: the large language model accurately determines the dependencies (such as state dependencies and temporal dependencies) between DML and DDL through semantic understanding, avoiding data inconsistencies. Only DML operations that truly depend on DDL will be blocked, while other DML operations can continue to execute, greatly improving synchronization efficiency. The large model's translation capabilities enable automatic syntax conversion across databases (such as from MySQL to Oracle). Syntax validation and retry mechanisms ensure the stability of the synchronization process.

[0091] The data synchronization device provided in the embodiments of the present invention will be described below. The data synchronization device described below and the data synchronization method described above can be referred to each other.

[0092] Please refer to the details. Figure 4 , Figure 4 A schematic diagram of a data synchronization device provided in an embodiment of the present invention may include:

[0093] The conversion module 100 is used to convert Data Definition Language events and Data Manipulation Language events into Data Definition Structured Statements and Data Manipulation Structured Statements;

[0094] Dependency analysis module 200 is used to determine, using a dependency analysis model, whether the data manipulation structured statement depends on the data definition structured statement;

[0095] The data synchronization module 300 is used to determine the synchronous execution of the data definition language event and the data manipulation language event when the data manipulation structured statement does not depend on the data definition structured statement.

[0096] Furthermore, based on any of the above embodiments, the dependency analysis module 200 may include:

[0097] The prompt word acquisition unit is used to acquire dependency analysis prompt words;

[0098] A dependency analysis unit is used to determine, based on the dependency analysis prompts and the dependency analysis model, whether the data manipulation structured statement depends on the data definition structured statement; wherein, the dependency analysis model is a large language model.

[0099] Furthermore, based on any of the above embodiments, the dependency analysis module 200 may include:

[0100] A dependency type determination unit is used to determine the dependency type; wherein, the dependency type includes state dependency and temporal dependency; the state dependency means that the execution of a Data Manipulation Language (DML) event logically depends on the table structure change caused by the Data Definition Language (DDL) event taking effect; the temporal dependency means that a DML event must be executed after the execution of the DDL event it depends on has been completed.

[0101] The dependency type analysis unit is used to determine whether the data manipulation structured statement depends on the data definition structured statement by utilizing the dependency analysis model and the dependency type.

[0102] Furthermore, based on any of the above embodiments, the data synchronization device may further include:

[0103] The dependent data definition language event determination module is used to determine the target data definition language that the data manipulation structured statement depends on when it is determined that the data manipulation structured statement depends on the data definition structured statement.

[0104] The semantic parsing module is used to perform semantic parsing on the target data definition language events corresponding to the target data definition structured statements, and obtain the semantic parsing results;

[0105] The conversion module is used to convert the semantic parsing results into target format Data Definition Language events required by the target library during execution;

[0106] The execution data synchronization module is used to execute the target format data definition language event in the target library to achieve data synchronization.

[0107] The data manipulation semantic event synchronization module is used to synchronize the data manipulation language event when the target data definition language event has been executed.

[0108] Furthermore, based on any of the above embodiments, the data synchronization device may further include:

[0109] The syntax verification module is used to perform syntax verification on the target format data definition language events to obtain the events after syntax verification.

[0110] The schema change module is used to change the schema of the target library based on the events that have been syntax-validated.

[0111] Furthermore, based on any of the above embodiments, the semantic parsing module may include:

[0112] The semantic parsing prompt word construction unit is used to determine the semantic parsing prompt words constructed based on the source database and target database corresponding to data synchronization.

[0113] The semantic parsing unit based on prompt words is used to perform semantic parsing on the target data definition language events corresponding to the target data definition structured statements based on the semantic parsing prompt words using a large language model, and obtain the semantic parsing results.

[0114] Furthermore, based on any of the above embodiments, the data synchronization device may further include:

[0115] The correction information acquisition module is used to acquire the manual correction dependency information corresponding to the current data manipulation structured statement;

[0116] The update module is used to optimize the dependency analysis model based on the manually corrected dependency information to obtain an optimized dependency analysis model.

[0117] It should be noted that the order of the modules and units in the above data synchronization device can be changed without affecting the logic.

[0118] This invention provides a data synchronization device, which may include: a conversion module 100, used to convert Data Definition Language (DDL) events and Data Manipulation Language (DML) events into Data Definition Structured Statements (DMS) and DMS, respectively; a dependency analysis module 200, used to determine whether the DMS depends on the DDL using a dependency analysis model; and a data synchronization module 300, used to determine that the DDL events and DML events should be executed synchronously when the DMS does not depend on the DML. Compared to the current situation where a single DDL blockes all DML operations during data synchronization, this invention can determine the dependency relationship between DDL events and DML events through a dependency analysis model, thereby synchronizing events that can be synchronized simultaneously in the target database. Since there is no need to wait for the DDL to finish executing before executing the DML, the efficiency of data synchronization can be improved.

[0119] The following describes a data synchronization device provided by an embodiment of the present invention. The data synchronization device described below and the data synchronization method described above can be referred to each other.

[0120] Please refer to Figure 5 , Figure 5 A schematic diagram of a data synchronization device provided in an embodiment of the present invention may include:

[0121] Memory 10 is used to store computer programs;

[0122] Processor 20 is used to execute computer programs to implement the data synchronization method described above.

[0123] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0124] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0125] Convert Data Definition Language events and Data Manipulation Language events into Data Definition Language structured statements and Data Manipulation Language structured statements;

[0126] Dependency analysis models are used to determine whether data manipulation structured statements depend on data definition structured statements.

[0127] When data manipulation structured statements do not depend on data definition structured statements, data definition language events and data manipulation language events are executed synchronously.

[0128] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0129] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0130] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0131] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0132] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the data synchronization device in the embodiments of the present invention. In practical applications, the data synchronization device may include more than Figure 5 More or fewer components as shown, or combinations of certain components.

[0133] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below and the data synchronization method described above can be referred to in correspondence.

[0134] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data synchronization method.

[0135] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0137] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0138] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0139] The above provides a detailed description of a data synchronization method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data synchronization method, characterized by, The method comprises the following steps: Converting data definition language events and data manipulation language events into data definition structured statements and data manipulation structured statements; Determining whether the data manipulation structured statements depend on the data definition structured statements by using a dependency analysis model; When the data manipulation structured statements do not depend on the data definition structured statements, determining to execute the data definition language events and the data manipulation language events synchronously.

2. The data synchronization method of claim 1, wherein, Determining whether the data manipulation structured statements depend on the data definition structured statements by using a dependency analysis model comprises: Obtaining a dependency analysis prompt word; Determining whether the data manipulation structured statements depend on the data definition structured statements by using the dependency analysis model based on the dependency analysis prompt word; wherein the dependency analysis model is a large language model.

3. The data synchronization method of claim 1, wherein, Determining whether the data manipulation structured statements depend on the data definition structured statements by using a dependency analysis model comprises: Determining a dependency type; wherein the dependency type comprises state dependency and timing dependency; the state dependency refers to that execution of a data manipulation language event logically depends on that a table structure change caused by a data definition language event has taken effect; the timing dependency refers to that a data manipulation language event must be executed after execution of a data definition language event on which the data manipulation language event depends; Determining whether the data manipulation structured statements depend on the data definition structured statements by using the dependency analysis model and the dependency type.

4. The data synchronization method according to any one of claims 1 to 3, characterized in that, After determining whether the data manipulation structured statements depend on the data definition structured statements by using a dependency analysis model, the method further comprises the following steps: When it is determined that the data manipulation structured statements depend on the data definition structured statements, determining a target data definition structured statement on which the data manipulation structured statements depend; Performing semantic analysis on a target data definition language event corresponding to the target data definition structured statement to obtain a semantic analysis result; Converting the semantic analysis result into a target format data definition language event required when the target library is executed; Executing the target format data definition language event in the target library to realize data synchronization; When the target data definition language event is executed, performing synchronization on the data manipulation language event.

5. The data synchronization method of claim 4, wherein, After converting the semantic analysis result into a target format data definition language event required when the target library is executed, the method further comprises the following steps: Performing syntax checking on the target format data definition language event to obtain an event after syntax checking; Performing mode change on the target library based on the event after syntax checking.

6. The data synchronization method of claim 4, wherein, Performing semantic analysis on a target data definition language event corresponding to the target data definition structured statement to obtain a semantic analysis result comprises the following steps: Determining a semantic analysis prompt word constructed based on a source database and a target database corresponding to data synchronization; Performing semantic analysis on the target data definition language event corresponding to the target data definition structured statement by using a large language model based on the semantic analysis prompt word to obtain the semantic analysis result.

7. The data synchronization method of claim 1, wherein, determining to synchronously execute the data definition language event and the data manipulation language event when the data manipulation structured statement does not depend on the data definition structured statement, further comprising: obtaining artificial correction dependency information corresponding to the data manipulation structured statement; optimizing the dependency analysis model based on the artificial correction dependency information to obtain an optimized dependency analysis model.

8. A data synchronization apparatus, characterized by comprising: comprising: a conversion module, configured to convert a data definition language event and a data manipulation language event into a data definition structured statement and a data manipulation structured statement; a dependency analysis module, configured to determine whether the data manipulation structured statement depends on the data definition structured statement by using a dependency analysis model; a data synchronization module, configured to determine to synchronously execute the data definition language event and the data manipulation language event when the data manipulation structured statement does not depend on the data definition structured statement.

9. A data synchronization device, characterized by comprising: a memory, configured to store a computer program; a processor, configured to execute the computer program to implement the steps of the data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, a computer readable storage medium, having stored thereon a computer program, wherein the computer program is executable by a processor to implement the steps of the data synchronization method according to any one of claims 1 to 7.

Images