Real-time data synchronization method, device, equipment, and storage medium for property management system

Abstract

The present invention proposes a real-time data synchronization method, device, equipment, and storage medium for a property management system. The method includes: configuring a preset Flink CDC task for each business database, starting the target Flink CDC task corresponding to the source database, obtaining multiple source table names from a mapping table, and determining the source data table in the corresponding source database; fully synchronizing the corresponding source data table from the source database to the first ODS layer of the Doris database; after all target Flink CDC tasks are polled, monitoring the binlog log of the source database in real time based on the source table name, and incrementally synchronizing the source data table updated in the source database from the source database to the first ODS layer. The source table name of the source data table whose data needs to be synchronized can be recorded in an editable mapping table to achieve flexible adjustment of the synchronized data; full data synchronization and incremental update of the entire business database can be achieved through a single Flink CDC task, thereby improving data synchronization efficiency and data management efficiency.

Description

Technical Field

[0001] The present invention relates to the field of database synchronization technology, and in particular to a real-time data synchronization method, device, equipment, and storage medium for a property system. Background Art

[0002] Currently, property management companies often involve multiple property entities. For example, a single company manages multiple residential communities, office buildings, and commercial properties. To achieve intelligent management, these companies typically deploy a universal online property management system that integrates multiple management functions. Because each property entity typically has independent management needs, each property entity is configured with a separate business system and business database within the property management system. Each business system stores the management data generated by the application management functions in its corresponding business database, which is then synchronized with the property management system's data warehouse for unified management.

[0003] Business databases typically have high requirements for ease of use and reliability, so they are mostly based on MySQL. Data warehouses, on the other hand, have high requirements for massive data volumes and computing power, so they are mostly based on Doris. Furthermore, Flink Change Data Capture (Flink CDC) technology is also introduced. Flink CDC tasks detect data updates in the MySQL database in real time and synchronize the data with the operational data store (ODS) layer of the Doris database.

[0004] When using Flink CDC technology for real-time data synchronization, existing technologies typically create a Flink CDC task for each table in a business database. However, there are many business databases, each with a large number of tables requiring data synchronization. Initial configuration is time-consuming, and whenever updates to table structures occur, maintenance personnel must reconfigure the corresponding Flink CDC task. This is not only error-prone, but also inefficient and inflexible in data synchronization configuration, impacting data management efficiency. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a real-time data synchronization method, device, equipment, and storage medium for a property management system, which can improve the efficiency and flexibility of data synchronization.

[0006] In a first aspect, an embodiment of the present invention provides a real-time data synchronization method for a property system, which is applied to a Flink CDC component of the property system. The property system also includes a Doris database and multiple business databases, and the business database is a MySQL database. The real-time data synchronization method for the property system includes:

[0007] Selecting at least one target Flink CDC task from a plurality of preset Flink CDC tasks, and determining the business database associated with the target Flink CDC task as a source database, wherein each business database is configured with one preset Flink CDC task;

[0008] Obtaining multiple source table names corresponding to each target Flink CDC task from a preset mapping table, and determining a source data table in the corresponding source database based on the source table names, wherein each preset Flink CDC task is associated with multiple source table names in the mapping table, and the source table names belong to the business database associated with the preset Flink CDC task;

[0009] Each time a target Flink CDC task is polled, a corresponding first ODS layer is created in the Doris database, and the corresponding source data table is fully synchronized from the source database to the first ODS layer;

[0010] After all the target Flink CDC tasks are polled, the binlog log of the source database is monitored in real time based on the source table name, and the source data table updated in the source database is incrementally synchronized from the source database to the first ODS layer.

[0011] According to some embodiments of the present invention, each of the business databases is preset with a database identifier. Before selecting at least one target Flink CDC task from a plurality of preset Flink CDC tasks, the method further includes:

[0012] Building a task configuration interface, displaying a first list, a second list, and a third list on the task configuration interface, wherein the first list includes the business database that is not configured with the preset Flink CDC task, the initial content of the second list is empty, and the third list includes the configured preset Flink CDC task;

[0013] In response to a selection operation by the user in the first list, determining the selected business database as a database to be configured, and filling all optional database tables of the database to be configured in the second list;

[0014] In response to a user selection operation in the second list, determining the name of the selected optional database table as the source table name, creating the preset Flink CDC task corresponding to the database to be configured, and binding the database identifier of the database to be configured to the corresponding preset Flink CDC task;

[0015] Write the newly created preset Flink CDC task and the corresponding source table name into the mapping table;

[0016] In response to a user selection operation in the third list, determining the selected preset Flink CDC task as the Flink CDC task to be configured, filling all the optional database tables of the source database associated with the Flink CDC task to be configured into the second list, and marking the source table names;

[0017] In response to a user's selection operation in the second list, a new source table name is added to the mapping table, or at least one source table name is deleted.

[0018] According to some embodiments of the present invention, creating a corresponding first ODS layer in the Doris database includes:

[0019] Determine the database identifier bound to each target Flink CDC task as the target identifier;

[0020] Creating the corresponding first ODS layer in the Doris database based on each target identifier, wherein the name of the first ODS layer is the corresponding target identifier;

[0021] A second ODS layer is created in the Doris database, wherein the second ODS layer is used to write the addition, deletion and modification records of each of the first ODS layers.

[0022] According to some embodiments of the present invention, synchronizing the corresponding source data table from the source database to the first ODS layer in full includes:

[0023] Acquire the source table structure and source table data of each source data table from the source database, generate a first Doris statement based on the source table structure, and generate a second Doris statement based on the source table data, wherein the first Doris statement is used to create a data table, and the second Doris statement is used to write data;

[0024] Based on the first Doris statement, a table creation operation is performed on the corresponding first ODS layer to obtain multiple target data tables, and the table creation operation is recorded on the second ODS layer;

[0025] A data write operation is performed in the target data table based on the second Doris statement to obtain target table data, and the data write operation is recorded in the second ODS layer.

[0026] According to some embodiments of the present invention, monitoring the binlog log of the source database in real time based on the source table name and incrementally synchronizing the source data table updated in the source database from the source database to the first ODS layer includes:

[0027] When any of the source table names is recorded in the newly added operation log of the binlog log, the corresponding operation log is determined as the target log, and the target operation field and target operation information are determined based on the target log;

[0028] A third Doris statement is generated based on the target operation information and the target operation field, and the third Doris statement is executed in the first ODS layer associated with the source data table corresponding to the target operation field.

[0029] According to some embodiments of the present invention, synchronizing the corresponding source data table from the source database to the first ODS layer in full includes:

[0030] When the number of the target Flink CDC task is one, all the source table names are arranged into a synchronization name queue based on the order in which the source table names are recorded in the mapping table;

[0031] Alternatively, when there are multiple target Flink CDC tasks, all source table names of the same target Flink CDC task are arranged into a task name queue based on the order in which the source table names are recorded in the mapping table. Multiple task name queues are then concatenated into a synchronized name queue based on the selection order of the target Flink CDC tasks.

[0032] Select at least one target name from the synchronization name queue in sequence, and synchronize the source data table corresponding to the target name from the source database to the corresponding first ODS layer in full;

[0033] When the source data table corresponding to the target name is synchronized, a new target name is selected from the synchronization name queue for data synchronization;

[0034] When synchronization of the source data table corresponding to the target name fails, the corresponding target name is marked as an abnormal name, and the corresponding target Flink CDC task is determined to be an abnormal Flink CDC task. When the abnormal Flink CDC task is restarted, at least one target name is selected from the synchronization name queue with the abnormal name as the starting point for data synchronization.

[0035] According to some embodiments of the present invention, after monitoring the binlog log of the source database in real time based on the source table name, the method further includes:

[0036] Select at least one new Flink CDC task from the preset Flink CDC tasks that have not been started;

[0037] When at least one of the target Flink CDC tasks has not completed full synchronization, the newly added Flink CDC task is added to the tail of the task queue of the target Flink CDC task;

[0038] When all the target Flink CDC tasks have completed full synchronization, each of the target Flink CDC tasks in the running state is suspended, and a new first ODS layer is created in the Doris database for the newly added Flink CDC task. Based on the newly added Flink CDC task and the mapping table, the source data table corresponding to the newly added Flink CDC task is fully synchronized from the source database to the newly created first ODS layer;

[0039] After the newly added Flink CDC task completes full synchronization, restart the suspended target Flink CDC tasks.

[0040] In the second aspect, an embodiment of the present invention provides a real-time data synchronization device for a property system, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the real-time data synchronization method for the property system as described in the first aspect above.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device comprising the real-time data synchronization device of the property system as described in the second aspect above.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the real-time data synchronization method of the property system as described in the first aspect above.

[0043] According to the real-time data synchronization method of the property system according to an embodiment of the present invention, there are at least the following beneficial effects: selecting at least one target Flink CDC task from multiple preset Flink CDC tasks, determining the business database associated with the target Flink CDC task as the source database, wherein each business database is configured with one preset Flink CDC task; obtaining multiple source table names corresponding to each target Flink CDC task from a preset mapping table, and determining the source data table in the corresponding source database based on the source table name, wherein each preset Flink CDC task is associated with multiple source table names in the mapping table, and the source table names belong to the business database associated with the preset Flink CDC task; each time a target Flink CDC task is polled, a corresponding first ODS layer is created in the Doris database, and the corresponding source data table is fully synchronized from the source database to the first ODS layer; when all the target Flink CDC tasks are polled, the binlog log of the source database is monitored in real time based on the source table name, and the source data table updated in the source database is incrementally synchronized from the source database to the first ODS layer. According to the technical solution of the embodiments of the present invention, a preset Flink CDC task can be built for each business database. The source table names of the source data tables whose data need to be synchronized are recorded in an editable mapping table, enabling flexible adjustment of synchronized data. Based on the source table names recorded in the mapping table, full data synchronization of the entire business database can be achieved through a single Flink CDC task. Data updates are synchronized based on binlog logs, improving data synchronization efficiency and data management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a property management system provided by one embodiment of the present invention;

[0045] Figure 2 is a flow chart of a real-time data synchronization method for a property system provided by another embodiment of the present invention;

[0046] Figure 3 This is a complete flow chart of a real-time data synchronization method for a property system provided by another embodiment of the present invention;

[0047] Figure 4 It is a structural diagram of a real-time data synchronization device for a property system provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0049] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0050] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0051] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0052] An embodiment of the present invention provides a real-time data synchronization method, apparatus, device, and storage medium for a property system, wherein the real-time data synchronization method for a property system includes: selecting at least one target Flink CDC task from a plurality of preset Flink CDC tasks, determining the business database associated with the target Flink CDC task as the source database, wherein each business database is configured with one preset Flink CDC task; obtaining multiple source table names corresponding to each target Flink CDC task from a preset mapping table, and determining a source data table in the corresponding source database based on the source table name, wherein each preset Flink CDC task is associated with multiple source table names in the mapping table, and the source table names belong to the business database associated with the preset Flink CDC task; each time a target Flink CDC task is polled, a corresponding first ODS layer is created in the Doris database, and the corresponding source data table is fully synchronized from the source database to the first ODS layer; when all the target Flink After the CDC task polling is completed, the binlog log of the source database is monitored in real time based on the source table name, and the source data table updated in the source database is incrementally synchronized from the source database to the first ODS layer. According to the technical solution of the embodiment of the present invention, a preset Flink CDC task can be constructed for each business database, and the source table name of the source data table that needs to synchronize data is recorded in an editable mapping table to achieve flexible adjustment of synchronized data; based on the source table name recorded in the mapping table, the full data synchronization of the entire business database can be achieved through a Flink CDC task, and data updates are synchronized based on the binlog log, which improves the efficiency of data synchronization and improves the efficiency of data management.

[0053] First, refer to Figure 1 , Figure 1 This is a schematic diagram of a property management system provided in an embodiment of the present invention. The property management system in this embodiment includes a Flink CDC component 20, a Doris database 30, and multiple business databases, where the business database is a MySQL database.

[0054] It should be noted that each business database corresponds to a property entity, and the business databases are not shared with each other, for example Figure 1 As shown, the first business database 11 corresponds to community A, the second business database corresponds to office building A, and the third business database corresponds to community B.

[0055] The following is based on the Figure 1 The property system shown further illustrates the technical solution of the embodiment of the present invention.

[0056] Reference Figure 2 , Figure 2 This is a flowchart of a real-time data synchronization method for a property system provided by an embodiment of the present invention. The real-time data synchronization method for a property system includes but is not limited to the following steps:

[0057] S10: selecting at least one target Flink CDC task from multiple preset Flink CDC tasks, and determining a business database associated with the target Flink CDC task as a source database, wherein each business database is configured with one preset Flink CDC task;

[0058] S20: Obtain multiple source table names corresponding to each target Flink CDC task from a preset mapping table, and determine a source data table in the corresponding source database based on the source table names. Each preset Flink CDC task is associated with multiple source table names in the mapping table, and the source table names belong to the business database associated with the preset Flink CDC task.

[0059] S30: Every time a target Flink CDC task is polled, the corresponding first ODS layer is created in the Doris database, and the corresponding source data table is fully synchronized from the source database to the first ODS layer;

[0060] S40: After all target Flink CDC tasks are polled, the binlog logs of the source database are monitored in real time based on the source table name, and the updated source data tables in the source database are incrementally synchronized from the source database to the first ODS layer.

[0061] It should be noted that in this embodiment, each business database corresponds to a property entity and stores the management data of the corresponding property entity. The data in each business database is unrelated. To achieve unified data management, this embodiment configures a preset Flink CDC task for each business database in the Flink CDC component, leveraging Flink CDC's real-time data detection mechanism to achieve data synchronization. Of course, after configuring the preset Flink CDC task, administrators can start or pause the preset Flink CDC task through the visual interface based on actual needs. The started preset Flink CDC task becomes the target Flink CDC task in this embodiment, and the corresponding business database is determined as the source database.

[0062] For example, Figure 1As shown, the first business database 11 corresponds to Flink CDC task 1, the second business database 12 corresponds to Flink CDC task 2, and the third business database 13 corresponds to Flink CDC task 3. Flink CDC task 1 and Flink CDC task 3 are started and determined as target Flink CDC tasks. Flink CDC task 2 is not started and is used as a preset Flink CDC task in the FlinkCDC component.

[0063] It should be noted that the preset Flink CDC task corresponds to a business database, and the data that needs to be synchronized in each business database may not be the same. In this embodiment, a mapping table is preset in the Flink CDC component. The mapping table records the Flink CDC tasks and the corresponding source table names. The maintenance personnel only need to write the names of the database tables that need to be synchronized in the business database into the mapping table. Each preset Flink CDC task can correspond to multiple source table names. After the task is started, the maintenance personnel can add or delete source table names in the mapping table according to actual needs, realizing flexible adjustment of data synchronization without the need for additional configuration of the Flink CDC task.

[0064] It should be noted that after starting the target Flink CDC task, the Flink CDC component reads the corresponding source table name set from the mapping table, and determines the source data table from the corresponding source database based on the source table name, thereby obtaining the synchronized data source. Of course, since the preset Flink CDC task uniquely corresponds to a business database, and the Doris database has computing capabilities, maintenance personnel can directly calculate and obtain output data in the Doris database, so the source table name must belong to the corresponding business database, for example Figure 1 As shown, table name 1-1 and table name 1-2 correspond to the names of two data tables in the first business database 11, but cannot be the data tables of the second business database 12 and the third business database 13, to avoid synchronizing data to other business databases in the first ODS layer corresponding to the business entity, which may cause errors in subsequent data applications.

[0065] For example, the Flink CDC component 20 can access various business databases and therefore synchronize data from any business database. If the source table name corresponding to Flink CDC task 1 in the mapping table contains table name 2-1, the data of the second business database 12 will be synchronized to the first ODS layer (1) corresponding to Flink CDC task 1. This will cause the data of another property entity to be introduced when the relevant data of the property entity corresponding to the first business database 11 is subsequently calculated, resulting in an error. Therefore, this embodiment ensures that in the mapping table, each pair of Flink CDC task and source table name must correspond to the same business database.

[0066] It should be noted that any number of preset Flink CDC tasks can be started in the Flink CDC component. After startup, the target Flink CDC tasks are executed one by one. After the first startup, the corresponding first ODS layer is created in the Doris database, and then the data is synchronized from the source database to the first ODS layer according to the corresponding source table name.

[0067] For example, Figure 1 As shown, after Flink CDC task 1 and Flink CDC task 3 are started, Flink CDC task 1 is executed first, and the first ODS layer (1) is created in the Doris database 30. After multiple source data tables are determined in the first business database 11 according to the source table names corresponding to Flink CDC task 1, such as table name 1-1 and table name 1-2, the source data tables are fully synchronized one by one to the first ODS layer (1); after the data synchronization of the first business database 11 is completed, the first ODS layer (3) is created in the Doris database 30. After multiple source data tables are determined in the third business database 13 according to the source table names corresponding to Flink CDC task 3, such as table name 3-1 and table name 3-2, the source data tables are fully synchronized one by one to the first ODS layer (3), and so on.

[0068] It should be noted that the Flink CDC task is a permanent service in the Flink CDC component. That is, after the target Flink CDC task is started, it will continue to run and continue to capture data updates. In this embodiment, the target Flink CDC task monitors the binlog log of the corresponding source database to determine data updates, and synchronizes the updated data to the corresponding first ODS layer in an incremental synchronization manner to achieve real-time data updates.

[0069] For example, continuing the full synchronization example of Flink CDC task 1 and Flink CDC task 3, after completing the full synchronization of data, the binlog log of the first business database 11 is monitored by Flink CDC task 1, and the binlog log of the third business database 13 is monitored by Flink CDC task 3. When Flink CDC task 1 detects that new data has been added to the source data table corresponding to table name 1-1, the newly added data increment is synchronized to the first ODS layer (1).

[0070] In addition, in one embodiment, each business database is preset with a database identifier. Before executing step S10, the method further includes but is not limited to the following steps:

[0071] S01: Build a task configuration interface, and display a first list, a second list, and a third list on the task configuration interface. The first list includes business databases that are not configured with preset Flink CDC tasks, the second list is initially empty, and the third list includes configured preset Flink CDC tasks.

[0072] S02, in response to the user's selection operation in the first list, determining the selected business database as the database to be configured, and filling the second list with all optional database tables of the database to be configured;

[0073] S03: In response to the user's selection operation in the second list, the name of the selected optional database table is determined as the source table name, a preset Flink CDC task corresponding to the database to be configured is created, and the database identifier of the database to be configured is bound to the corresponding preset Flink CDC task;

[0074] S04: Write the newly created preset Flink CDC task and the corresponding source table name into the mapping table;

[0075] S05: In response to the user's selection operation in the third list, the selected preset Flink CDC task is determined as the Flink CDC task to be configured, and all optional database tables of the source database associated with the Flink CDC task to be configured are added to the second list, and the source table names are marked;

[0076] S06: In response to the user's selection operation in the second list, a new source table name is added to the mapping table, or at least one source table name is deleted.

[0077] It should be noted that the property system includes multiple business databases, each of which corresponds to a property entity. To improve the convenience of task configuration, this embodiment sets a visual task configuration interface in the Flink CDC component. The task configuration interface displays a first list and a second list. The first list includes business databases that are not configured with preset Flink CDC tasks, and the third list includes configured preset Flink CDC tasks. The second list is initially empty and is flexibly filled based on the selection in the first list or the third list.

[0078] It should be noted that after selecting an unconfigured business database in the first list, the business database is determined as the database to be configured, and all database tables of the database to be configured are determined as optional database tables and displayed in the second list. Users can make multiple selections in the second list to configure the database tables that need to be synchronized. After the selection is completed, the names of the selected optional database tables are determined as the source table names, the database identifier of the database to be configured is bound to the preset Flink CDC task, and a new set of preset Flink CDC tasks and source table names are inserted into the mapping table.

[0079] It should be noted that the database identifier can be the name of the business database. For example, if the name of the business database is Cell A, you can bind "Cell A" to the preset Flink CDC task as task information, or you can directly change the task name of the preset Flink CDC task to "Cell A". You can bind the database identifier to the preset Flink CDC task.

[0080] It should be noted that in this embodiment, you can also select a preset Flink CDC task that has been configured from the third list. Only one preset Flink CDC task can be selected at a time as the Flink CDC task to be configured. After selection, all optional database tables of the corresponding source database are filled in the second list, and the source table names recorded in the mapping table are marked. Users can select in the second list to adjust the database tables that need to be synchronized. For example, refer to Figure 1 After selecting Flink CDC task 1, all database tables of the first business database 11 are filled in the second list, and table names 1-1 and 1-2 are highlighted. The user clicks table name 1-2 to remove it from the mapping table. Clicking unselected table name 1-3 writes table name 1-3 to the mapping table, enabling flexible configuration of the source table names recorded in the mapping table and, therefore, flexible configuration of the synchronized data tables.

[0081] In addition, in one embodiment, in step S30, a corresponding first ODS layer is created in the Doris database, specifically including but not limited to the following steps:

[0082] S311: Determine the database identifier bound to each target Flink CDC task as the target identifier;

[0083] S312, creating a corresponding first ODS layer in the Doris database based on each target identifier, wherein the name of the first ODS layer is the corresponding target identifier;

[0084] S313: Create a second ODS layer in the Doris database, where the second ODS layer is used to write the addition, deletion, and modification records of each first ODS layer.

[0085] It should be noted that this embodiment creates a unique first ODS layer for each target Flink CDC task in the Doris database to ensure that the data of each source database is synchronized to a mutually isolated storage space, thereby ensuring the isolation of data storage.

[0086] It should be noted that this embodiment uses the database identifier bound to the target Flink CDC task as the target identifier, and uses the target identifier as the name of the first ODS layer, for example Figure 1 As shown, the database identifier of the first business database 11 is (1), and the name of the first ODS layer created for the Flink CDC task 1 in the Doris database is (1). The database identifier is used as the matching basis for the source database, Flink CDC task and ODS layer to ensure the accuracy of data synchronization.

[0087] It should be noted that this embodiment also creates a second ODS layer in the Doris database, that is, the Doris database includes multiple first ODS layers and one second ODS layer. Each first ODS layer corresponds to a target Flink CDC task, and the second ODS layer is used to record the operation records of the Doris database. Therefore, there is no need to distinguish between different business databases or target Flink CDC tasks. The data operations of each first ODS layer are recorded in the same second ODS layer. When an exception occurs, the relevant exception location can be performed according to the records of the second ODS layer, and the current data operations of the Doris database can be tracked in real time. The specific application of the operation records is not limited here.

[0088] In addition, in one embodiment, in step S30, the corresponding source data table is fully synchronized from the source database to the first ODS layer, specifically including but not limited to the following steps:

[0089] S321, obtaining the source table structure and source table data of each source data table from the source database, generating a first Doris statement based on the source table structure, and generating a second Doris statement based on the source table data, wherein the first Doris statement is used to create the data table, and the second Doris statement is used to write data;

[0090] S322, based on the first Doris statement, a table creation operation is performed on the corresponding first ODS layer to obtain multiple target data tables, and the table creation operation is recorded on the second ODS layer;

[0091] S323: Based on the second Doris statement, a data write operation is performed in the target data table to obtain target table data, and the data write operation is recorded in the second ODS layer.

[0092] It should be noted that this embodiment is a synchronization process for one target Flink CDC task. In the case of multiple target Flink CDC tasks, the steps of this embodiment can be executed one by one, and will not be repeated later.

[0093] It should be noted that after determining the source table name, each source data table is determined in the source database according to the source table name. Since the source database is a MySQL database, the source table structure can be obtained through the "DESCRIBE" or "SHOW CREATE TABLE" statement, and then the corresponding source table data can be obtained through the "SELECT" statement. The specific instructions are not described in detail here. Those skilled in the art are familiar with how to export the source table structure and source table data when the source data table is determined.

[0094] It should be noted that this embodiment generates a first Doris statement based on the source table structure, creates a table in the first ODS layer using the first Doris statement to obtain multiple target data tables, generates a second Doris statement based on the source table data, inserts data into the established target data table, ensures that the table structure and table data of the source data table can be fully synchronized to the Doris database, and records each of the above steps in the second ODS layer. Those skilled in the art are familiar with how to convert between MySQL statements and Doris statements, and will not be limited here.

[0095] In addition, in one embodiment, step S40 specifically includes but is not limited to the following steps:

[0096] S41: When any source table name is recorded in the newly added operation log of the binlog log, the corresponding operation log is determined as the target log, and the target operation field and target operation information are determined based on the target log;

[0097] S42: Generate a third Doris statement based on the target operation information and the target operation field, and execute the third Doris statement in the first ODS layer associated with the source data table corresponding to the target operation field.

[0098] It should be noted that the binlog log records the operations of each database table in the business database, including table structure modifications, data updates, etc. The Flink CDC task can detect data updates in real time. When the source table name is matched in the binlog log, the table structure or table data of the source data table synchronized to the Doris database has been updated. This embodiment synchronizes the relevant updates to the Doris database in real time, thereby ensuring the real-time performance of data synchronization.

[0099] It should be noted that this embodiment extracts the target operation fields and target operation information related to the relevant operations from the target log. The target operation information can include adding a target operation field, deleting a target operation field, modifying the content within the target operation field, etc. When the target operation field and target operation information are determined, a corresponding third Doris statement can be generated. Data synchronization is achieved by executing the third Doris statement at the first ODS layer. Of course, the first ODS layer that executes the third Doris statement and the corresponding source table name that has changed belong to the same source database, and no further details will be given here.

[0100] In addition, in one embodiment, in step S30, the corresponding source data table is fully synchronized from the source database to the first ODS layer, specifically including but not limited to the following steps:

[0101] S331: When the number of target Flink CDC tasks is one, all source table names are arranged into a synchronization name queue based on the order in which the source table names are recorded in the mapping table.

[0102] S332: When there are multiple target Flink CDC tasks, all source table names of the same target Flink CDC task are arranged into a task name queue based on the order in which the source table names are recorded in the mapping table. Multiple task name queues are then concatenated into a synchronized name queue based on the selection order of the target Flink CDC tasks.

[0103] S333: Select at least one target name from the synchronization name queue in sequence, and synchronize the source data table corresponding to the target name from the source database to the corresponding first ODS layer in full.

[0104] S334: When the source data table corresponding to the target name is synchronized, a new target name is selected from the synchronization name queue for data synchronization;

[0105] S335: When the synchronization of the source data table corresponding to the target name fails, the corresponding target name is marked as an abnormal name, and the corresponding target Flink CDC task is determined as an abnormal Flink CDC task. When the abnormal Flink CDC task is restarted, at least one target name is selected from the synchronization name queue with the abnormal name as the starting point for data synchronization.

[0106] It should be noted that this embodiment may include multiple target Flink CDC tasks. To ensure the continuity of data transmission, this embodiment introduces a breakpoint-resume mechanism and constructs a synchronization name queue based on the number of target Flink CDC tasks. The synchronization name queue records all source table names in order. When there is only one target Flink CDC task, the source table names recorded in the mapping table can be placed in the synchronization name queue. When there are multiple target Flink CDC tasks, a task name queue is generated for each target Flink CDC task, and the source table names recorded in the mapping table are placed in the task name queue. The task name queues are then concatenated in the order of the target Flink CDC tasks to obtain a synchronization name queue. The synchronization name queue represents the synchronization order of the source data table and provides an index basis for breakpoint-resume transmission.

[0107] It should be noted that after determining the synchronization name queue, you can select multiple target names each time to achieve parallel full synchronization, and synchronize the table structure and table data of the source data table to the corresponding first ODS layer. After completing the full synchronization, select the next group of target names to perform the same operation to achieve sequential synchronization.

[0108] It should be noted that when a synchronization failure is detected, the target name where the synchronization failure occurred is marked as an exception name, the corresponding target Flink CDC task is identified as an abnormal Flink CDC task, and data synchronization is suspended. After the maintenance personnel resolve the root cause of the exception, the abnormal Flink CDC task is restarted and the above steps are re-executed starting from the abnormal name. The name of the source table that has been synchronized will not be synchronized again, thus achieving breakpoint resumption and ensuring the continuity of data synchronization.

[0109] In addition, in one embodiment, after step S40 is performed, the following steps are also included but not limited to:

[0110] S51, selecting at least one new Flink CDC task from the preset Flink CDC tasks that have not been started;

[0111] S52: When at least one of the target Flink CDC tasks has not completed full synchronization, add the newly added Flink CDC task to the tail of the task queue of the target Flink CDC task;

[0112] S53: When all the target Flink CDC tasks have completed full synchronization, each of the target Flink CDC tasks in the running state is suspended, and a new first ODS layer is created in the Doris database for the newly added Flink CDC task. Based on the newly added Flink CDC task and the mapping table, the source data table corresponding to the newly added Flink CDC task is fully synchronized from the source database to the newly created first ODS layer;

[0113] S54: After the newly added Flink CDC task completes full synchronization, restart the suspended target Flink CDC tasks.

[0114] It should be noted that after starting multiple target Flink CDC tasks, you can further add new Flink CDC tasks to achieve flexible control of multiple business databases. For example Figure 1 As shown, after starting Flink CDC task 1 and Flink CDC task 3, the user can further start Flink CDC task 2 in Flink CDC component 20.

[0115] It should be noted that, according to the description of the above embodiment, data synchronization in this implementation includes full synchronization and incremental synchronization. Full synchronization synchronizes data from the MySQL database to the empty Doris database, while incremental synchronization is used to update data in real time after the full synchronization is completed. Therefore, after determining to add a new Flink CDC task, this embodiment determines how to insert the new Flink CDC task based on the synchronization progress of the previously started target Flink CDC task.

[0116] It should be noted that if at least one target Flink CDC task has not completed full synchronization, that is, the Flink CDC component is in the full synchronization phase, the newly added Flink CDC task will be placed at the end of the task queue. After the last target Flink CDC task completes full synchronization, the full synchronization of the newly added Flink CDC task will continue. For example, in the above example of Flink CDC Task 1 and Flink CDC Task 3, when Flink CDC Task 1 completes full synchronization and Flink CDC Task 3 is in the full synchronization state, Flink CDC Task 2 is added and Flink CDC Task 2 is queued after Flink CDC Task 3. After Flink CDC Task 3 completes full synchronization, the full synchronization of Flink CDC Task 2 will continue.

[0117] It should be noted that when all target Flink CDC tasks have completed full synchronization and are in the stage of monitoring binlog logs for real-time incremental synchronization, this embodiment suspends the incremental synchronization of all target Flink CDC tasks, executes the full synchronization of the newly added Flink CDC tasks, and then restarts all incremental synchronizations. At this time, since the newly added FlinkCDC task has completed full synchronization, it also starts incremental synchronization based on the solution of the above embodiment. It is worth noting that since incremental synchronization is performed based on the binlog log, a pause node can be inserted in the binlog log when monitoring is paused. After restarting monitoring, real-time monitoring of the binlog log is resumed and incremental synchronization is performed according to the pause node. Therefore, pausing the target Flink CDC task will not cause data to be confirmed. Incremental synchronization after restarting can ensure data synchronization.

[0118] For example, Figure 1 As shown in the figure, after Flink CDC Task 1 and Flink CDC Task 3 complete full synchronization, Flink CDC Task 2 is started, and Flink CDC Task 1 and Flink CDC Task 3's monitoring of the binlog logs is suspended. After performing full synchronization of Flink CDC Task 2, Flink CDC Task 1 and Flink CDC Task 3's monitoring of the binlog logs are restarted and incremental synchronization is performed. At the same time, binlog log monitoring is performed based on Flink CDC Task 2.

[0119] like Figure 4 As shown, Figure 4 This is a structural diagram of a real-time data synchronization device for a property management system provided by an embodiment of the present invention. The present invention also provides a real-time data synchronization device for a property management system, comprising:

[0120] The processor 401 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0121] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called by the processor 401 to execute the real-time data synchronization method of the property management system in the embodiments of this application.

[0122] Input / output interface 403, used to implement information input and output;

[0123] Communication interface 404, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0124] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0125] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via a bus 405 .

[0126] An embodiment of the present application also provides an electronic device, including the real-time data synchronization device of the property system as described above.

[0127] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the real-time data synchronization method of the above-mentioned property system is implemented.

[0128] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned networks include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0130] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the shared conditions that do not violate the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A real-time data synchronization method for a property system, characterized in that: The Flink CDC component is applied to the property management system. The property management system also includes a Doris database and multiple business databases. The business database is a MySQL database. The real-time data synchronization method of the property management system includes: Selecting at least one target Flink CDC task from a plurality of preset Flink CDC tasks, and determining the business database associated with the target Flink CDC task as a source database, wherein each business database is configured with one preset Flink CDC task; Obtaining multiple source table names corresponding to each target Flink CDC task from a preset mapping table, and determining a source data table in the corresponding source database based on the source table names, wherein each preset Flink CDC task is associated with multiple source table names in the mapping table, and the source table names belong to the business database associated with the preset Flink CDC task; Each time a target Flink CDC task is polled, a corresponding first ODS layer is created in the Doris database, and the corresponding source data table is fully synchronized from the source database to the first ODS layer; After all the target Flink CDC tasks are polled, the binlog log of the source database is monitored in real time based on the source table name, and the source data table updated in the source database is incrementally synchronized from the source database to the first ODS layer.

2. The real-time data synchronization method of the property system according to claim 1, characterized in that: Each of the business databases is preset with a database identifier. Before selecting at least one target Flink CDC task from a plurality of preset Flink CDC tasks, the method further includes: Building a task configuration interface, displaying a first list, a second list, and a third list on the task configuration interface, wherein the first list includes the business database that is not configured with the preset Flink CDC task, the initial content of the second list is empty, and the third list includes the configured preset Flink CDC task; In response to a selection operation by the user in the first list, determining the selected business database as a database to be configured, and filling all optional database tables of the database to be configured in the second list; In response to a user selection operation in the second list, determining the name of the selected optional database table as the source table name, creating the preset Flink CDC task corresponding to the database to be configured, and binding the database identifier of the database to be configured to the corresponding preset Flink CDC task; Write the newly created preset Flink CDC task and the corresponding source table name into the mapping table; In response to a user selection operation in the third list, determining the selected preset Flink CDC task as the Flink CDC task to be configured, filling all the optional database tables of the source database associated with the Flink CDC task to be configured into the second list, and marking the source table names; In response to a user's selection operation in the second list, a new source table name is added to the mapping table, or at least one source table name is deleted.

3. The real-time data synchronization method of the property system according to claim 2, characterized in that: Create the corresponding first ODS layer in the Doris database, including: Determine the database identifier bound to each target Flink CDC task as the target identifier; Creating the corresponding first ODS layer in the Doris database based on each target identifier, wherein the name of the first ODS layer is the corresponding target identifier; A second ODS layer is created in the Doris database, wherein the second ODS layer is used to write the addition, deletion and modification records of each of the first ODS layers.

4. The real-time data synchronization method of the property system according to claim 3, characterized in that: Synchronizing the corresponding source data table from the source database to the first ODS layer in full, including: Acquire the source table structure and source table data of each source data table from the source database, generate a first Doris statement based on the source table structure, and generate a second Doris statement based on the source table data, wherein the first Doris statement is used to create a data table, and the second Doris statement is used to write data; Based on the first Doris statement, a table creation operation is performed on the corresponding first ODS layer to obtain multiple target data tables, and the table creation operation is recorded on the second ODS layer; A data write operation is performed in the target data table based on the second Doris statement to obtain target table data, and the data write operation is recorded in the second ODS layer.

5. The real-time data synchronization method of the property system according to claim 3, characterized in that: Monitoring the binlog of the source database in real time based on the source table name, and incrementally synchronizing the source data table updated in the source database from the source database to the first ODS layer includes: When any of the source table names is recorded in the newly added operation log of the binlog log, the corresponding operation log is determined as the target log, and the target operation field and target operation information are determined based on the target log; A third Doris statement is generated based on the target operation information and the target operation field, and the third Doris statement is executed in the first ODS layer associated with the source data table corresponding to the target operation field.

6. The real-time data synchronization method of the property system according to claim 1, characterized in that: Synchronizing the corresponding source data table from the source database to the first ODS layer in full, including: When the number of the target Flink CDC task is one, all the source table names are arranged into a synchronization name queue based on the order in which the source table names are recorded in the mapping table; Alternatively, when there are multiple target Flink CDC tasks, all source table names of the same target Flink CDC task are arranged into a task name queue based on the order in which the source table names are recorded in the mapping table. Multiple task name queues are then concatenated into a synchronized name queue based on the selection order of the target Flink CDC tasks. Select at least one target name from the synchronization name queue in sequence, and synchronize the source data table corresponding to the target name from the source database to the corresponding first ODS layer in full; When the source data table corresponding to the target name is synchronized, a new target name is selected from the synchronization name queue for data synchronization; When synchronization of the source data table corresponding to the target name fails, the corresponding target name is marked as an abnormal name, and the corresponding target Flink CDC task is determined to be an abnormal Flink CDC task. When the abnormal Flink CDC task is restarted, at least one target name is selected from the synchronization name queue with the abnormal name as the starting point for data synchronization.

7. The real-time data synchronization method of the property system according to claim 1, characterized in that: After monitoring the binlog log of the source database in real time based on the source table name, the method further includes: Select at least one new Flink CDC task from the preset Flink CDC tasks that have not been started; When at least one of the target Flink CDC tasks has not completed full synchronization, the newly added Flink CDC task is added to the tail of the task queue of the target Flink CDC task; When all the target Flink CDC tasks have completed full synchronization, each of the target Flink CDC tasks in the running state is suspended, and a new first ODS layer is created in the Doris database for the newly added Flink CDC task. Based on the newly added Flink CDC task and the mapping table, the source data table corresponding to the newly added Flink CDC task is fully synchronized from the source database to the newly created first ODS layer; After the newly added Flink CDC task completes full synchronization, restart the suspended target Flink CDC tasks.

8. A real-time data synchronization device for a property system, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the real-time data synchronization method of the property system as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: Including the real-time data synchronization device of the property system as described in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the real-time data synchronization method for a property system as described in any one of claims 1 to 7.

Images