Remote database synchronization method, device, equipment and medium

Abstract

A remote database synchronization method is applied to a remote deployment system, the remote deployment system includes at least one data management system and multiple remote subsystems, including: executing synchronization tasks at a preset frequency and according to preset synchronization conditions through a collection service deployed on each remote subsystem to generate a data information file, the synchronization task includes collecting and encapsulating data, and the data information file is an SQL file; packaging the synchronization conditions and the data information file into a first data file package, and sending it to the data management system, the file name of the data information file package includes the time mark and task number of this data collection; when the data management system receives the first data file package, calling the synchronization service deployed in the data management system to parse the first data file package, and after determining that the time mark and the task number meet the preset conditions, updating the database according to the synchronization conditions and the data information file. The synchronization frequency and content can be configured remotely by the data management system.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data synchronization, and in particular to a remote database synchronization method, device, electronic device and medium. Background Art

[0002] Due to business needs, many systems are deployed remotely, and the headquarters needs to obtain business data from remote subsystems in near real time to monitor system status. However, in actual applications, the distance between remote subsystems and headquarters is far, and their data transmission status is limited by external conditions such as network instability and bandwidth.

[0003] Currently, most data synchronization technologies use commercial software, such as DSG RealSync or Quest Shareplex. Commercial software is expensive and has the following disadvantages: it requires the use of the same source database or only supports some databases; it has requirements for network bandwidth and cannot be used when the network bandwidth is limited or the network is unstable; it needs to insert some intermediate tables into the original system, affecting reliability. Summary of the invention

[0004] In view of the above problems, the present invention provides a remote database synchronization method, device, electronic device and medium to solve the above technical problems.

[0005] One aspect of the present disclosure provides a remote database synchronization method, which is applied to a remote deployment system, wherein the remote deployment system includes at least one data management system and multiple remote subsystems, including: regularly starting a collection service deployed on each of the remote subsystems, executing a synchronization task according to a preset synchronization condition, and generating a data information file, wherein the synchronization task includes collecting and encapsulating data, and the data information file is an SQL file; packaging the synchronization condition and the data information file into a first data file package, and sending the package to the data management system, wherein the file name of the first data file package includes a time stamp and a task order number for this data collection; when the data management system receives the first data file package, starting the synchronization service deployed in the data management system to parse the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, updating the database according to the synchronization condition and the data information file.

[0006] Optionally, when the data management system receives the first data file package, the synchronization service deployed in the data management system is called to parse the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, the database is updated according to the synchronization conditions and the data information file, including: when the data management system receives the first data file package, the synchronization service is called to parse the time stamp and the task order number from the file name of the first data file package; according to the task order number, the latest time stamp of the synchronization service is queried to determine whether a data file package with the same task order number and a time stamp earlier than the current time stamp has been received; if there is a data file package with the same task order number and a time stamp earlier than the current time stamp, the first data file package is parsed to obtain the synchronization conditions and data information file to update the database; if there is a data file package with the same task order number and a time stamp later than the current time stamp, the first data file package is abandoned.

[0007] Optionally, the synchronization conditions include a database flag, a database table name and a data collection condition, the data collection condition includes a deletion condition, and updating the database according to the synchronization conditions and the data information file includes: searching for database connection information according to the database flag to connect to the corresponding database; passing the database table name and the data collection condition to a preset deletion script to delete the data related to the deletion condition in the corresponding database table; after completing the data deletion, inserting the data included in the data information file into the database table to complete the data synchronization.

[0008] Optionally, the collection service deployed on each of the remote subsystems performs synchronization tasks at a preset frequency and according to preset synchronization conditions to generate a data information file, including: calling the condition generation plug-in of the collection service at a preset frequency, generating an information description file according to the preset synchronization conditions, the information description file including a database flag, a database table name and a data collection condition, wherein the condition generation plug-in is packaged according to the type of database served; starting the data collection plug-in of the collection service, searching for database connection information according to the data flag, so as to connect to the corresponding database according to the database connection information, and collecting data in the database table corresponding to the database table name according to the data collection conditions to generate the data information file.

[0009] Optionally, the collection service and the synchronization service adopt a plug-in design, and the startup method of the collection service and the synchronization service includes: periodically scanning the XML configuration files of the collection service and / or the synchronization service, starting the plug-in that needs to be loaded for the collection service, and calculating the MD5 value of the plug-in; comparing the current MD5 value of the plug-in with the previous MD5 value, and if the two are different, restarting the plug-in; before restarting the plug-in, confirming that there is no synchronization task being executed in the asynchronous subsystem; if there is a synchronization task being executed, restarting the plug-in after the synchronization task is completed.

[0010] Optionally, the method further includes: issuing a synchronization task to each of the remote subsystems through the overall data management system, the synchronization task including the synchronization condition and data synchronization frequency; issuing an update plug-in for the collection service to each of the remote subsystems through the overall data management system.

[0011] Optionally, the method further includes: establishing communication between the overall data management system and each of the remote subsystems and between each of the remote subsystems; when there is a communication failure between one of the remote subsystems and the overall data management system, controlling the remote subsystem to forward the data file package through other remote subsystems.

[0012] The second aspect of the present disclosure provides a remote database synchronization device, including: a data collection module, used to perform synchronization tasks at a preset frequency and in accordance with preset synchronization conditions through a collection service deployed on each of the remote subsystems to generate a data information file, wherein the synchronization task includes collecting and encapsulating data, and the data information file is an SQL file; a data forwarding module, used to package the synchronization conditions and the data information file into a first data file package, and send it to the data management system, wherein the file name of the first data file package includes a time stamp and a task order number for this data collection; a data synchronization module, used to start the synchronization service deployed in the data management system to parse the first data file package when the data management system receives the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, update the database according to the synchronization conditions and the data information file.

[0013] The third aspect of the present disclosure provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the remote database synchronization method described in any one of the first aspects is implemented.

[0014] A fourth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements each step of any one of the remote database synchronization methods in the first aspect.

[0015] At least one of the above technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0016] The remote database synchronization method provided by the embodiment of the present disclosure can adapt to the situation of limited network bandwidth, and can adapt to various databases and databases from different sources. At the same time, it realizes the configuration management of the remote subsystem by the headquarters and the dynamic loading of the remote subsystem plug-in, reducing the difficulty of the headquarters' management of the remote subsystem. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a more complete understanding of the present disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A schematic diagram of a remote database synchronization method provided by an embodiment of the present disclosure is schematically shown;

[0019] Figure 2 A schematic diagram of an application scenario of a remote database synchronization method provided by an embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A schematic diagram of an application scenario of a collection service provided by an embodiment of the present disclosure is schematically shown;

[0021] Figure 4 A schematic diagram of an application scenario of a synchronization service provided by an embodiment of the present disclosure is schematically shown;

[0022] Figure 5 A schematic diagram of a remote database network architecture provided by an embodiment of the present disclosure is schematically shown;

[0023] Figure 6 A schematic diagram of interaction between a remote subsystem and a headquarters provided by an embodiment of the present disclosure is shown schematically;

[0024] Figure 7 A schematic diagram of a structure of a remote database synchronization device provided by an embodiment of the present disclosure is shown;

[0025] Figure 8 The structural block diagram of an electronic device provided by an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0029] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.

[0030] Therefore, the technology of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the technology of the present disclosure can take the form of a computer program product on a computer-readable medium storing instructions, which can be used by an instruction execution system or in combination with an instruction execution system. In the context of the present disclosure, a computer-readable medium can be any medium that can contain, store, transmit, propagate, or transmit instructions. For example, a computer-readable medium can include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, device, or propagation medium. Specific examples of computer-readable media include: magnetic storage devices, such as magnetic tapes or hard disks (HDDs); optical storage devices, such as compact disks (CD-ROMs); memories, such as random access memories (RAMs) or flash memory; and / or wired / wireless communication links.

[0031] The disclosed embodiment provides a remote database synchronization method, which is suitable for data synchronization in situations where the distance is long and the network is unstable. At the same time, it can realize the configuration management of the remote subsystem by the headquarters and the dynamic loading of the remote subsystem plug-in, which is convenient for the headquarters to manage the remote subsystem.

[0032] Figure 1 A schematic diagram of a remote database synchronization method provided by an embodiment of the present disclosure is schematically shown.

[0033] like Figure 1 As shown, a remote database synchronization method provided by an embodiment of the present disclosure includes operations S110 to S130.

[0034] In operation S110, the collection service deployed on each remote subsystem executes the synchronization task at a preset frequency and according to the preset synchronization conditions to generate a data information file. The synchronization task includes collecting and packaging data according to the synchronization conditions. The data information file is an SQL file.

[0035] In this embodiment, the synchronization condition may include database information, database identifier, database table name and collection condition, etc., which are used for the collection service to collect data, and when the synchronization service obtains the data collected by the collection service, the database synchronization operation is performed according to the synchronization data. In order to adapt to multi-source heterogeneous databases, the data information file adopts SQL format.

[0036] In operation S120, the synchronization conditions and the data information file are packaged into a first data file package and sent to the data management system. The file name of the first data file package includes a time stamp and a task order number of this data collection.

[0037] In this embodiment, in order to prevent the synchronized database information files from arriving at the headquarters out of order due to network factors, the time stamp and task order number can be used as marks of data sequence synchronization to ensure that the first data file package is processed in the correct order when the network is unstable.

[0038] In operation S130, when the data management system receives the first data file package, it calls the synchronization service deployed in the data management system to parse the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, updates the database according to the synchronization conditions and the data information file.

[0039] In this embodiment, in order to save network bandwidth, the first data file package is transmitted in the form of a compressed package zip. If the order of the first data file package is incorrect based on the time stamp and the task order number on the name of the first data file package, the compressed package is not decompressed, and the parsing of the first data file package is abandoned to avoid wasting system computing resources. If the order of the first data file package is correct based on the time stamp and the task order number on the name of the first data file package, the parsing of the first data file package is performed again.

[0040] Figure 2 A schematic diagram of an application scenario of a remote database synchronization method provided by an embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the remote database management system consists of a collection service and a synchronization service, wherein the collection service is deployed in the remote subsystem and the synchronization service is deployed in the data management system (i.e., the headquarters). The collection service is responsible for collecting data information from the remote subsystem, packaging the data information with the synchronization conditions, hiding the time stamp in the name of the first data file package, and then sending the first data file package to the headquarters. Because relational databases can directly run SQL files, the collection service at the headquarters can directly generate an SQL file based on the data information in the first data file package after receiving the first data file package, and the synchronization service can run the SQL file generated by the collection service, read the data information, and inject it into the headquarters database for management.

[0042] Optionally, the collection service and synchronization service adopt a plug-in design, and the plug-in to be integrated is specified through XML configuration. The loaded plug-in can be a service or a functional module. The service plug-in is marked with Service, and the startup command of the plug-in is written. The collection service and synchronization service start the plug-in according to the startup command in the configuration file, and the loaded functional module is marked with Module. The XML configuration interface is shown in the following table.

[0043] Table 1 Dynamic loading interface

[0044]

[0045]

[0046] The following table shows a sample configuration of the collection plug-in.

[0047] Table 2

[0048]

[0049] In the embodiment of the present disclosure, the collection service and the synchronization service are designed in a plug-in manner, and the startup method of the collection service and the synchronization service includes operations S210 to S230.

[0050] In operation S210, the XML configuration files of the collection service and / or the synchronization service are scanned periodically, the plug-ins to be loaded are started, and the MD5 values ​​of the plug-ins are calculated. Specifically, the remote subsystem periodically scans the local XML configuration files, persistently stores the contents of the XML configuration files, parses the contents of the XML configuration files, obtains the plug-ins to be loaded, starts the plug-ins to be loaded one by one according to the startup mode configured in XML, and calculates the MD5 values ​​of the plug-ins to be loaded one by one and stores them persistently.

[0051] In operation S220, the current MD5 value of the plug-in is compared with the last MD5 value. If the two are different, the plug-in is restarted. When the XML configuration file is scanned again, the MD5 value of the plug-in is calculated. If the MD5 value of the current plug-in is different from the last MD5 value, it proves that the plug-in has changed and needs to be restarted.

[0052] In operation S230, before restarting the plug-in, confirm that there is no synchronization task being executed in the asynchronous subsystem. If not, restart directly; if there is a synchronization task being executed, restart the plug-in after the synchronization task is completed. In addition, if the plug-in parsed this time has not been loaded before, start it directly according to the configured command.

[0053] Optionally, the data management system can be used to send updated plug-ins for collection services to each remote subsystem. If the headquarters needs to update the plug-in of a remote subsystem, the plug-in is sent directly to the remote subsystem. After the remote subsystem receives the new plug-in, it automatically restarts the plug-in. If the headquarters needs to add a new plug-in, the plug-in and XML configuration are sent to the remote subsystem at the same time. After the remote subsystem receives the plug-in, it starts the plug-in according to the configuration.

[0054] Figure 3 A schematic diagram of an application scenario of a collection service provided by an embodiment of the present disclosure is schematically shown.

[0055] like Figure 3 As shown in the figure, the collection service plug-in is divided into two categories, namely condition generation plug-in and information collection plug-in. The condition generation plug-in changes according to the database table structure and business requirements in the actual business and needs to be customized according to the actual business. The information collection plug-in is stable and unchanged. Different types of relational databases are encapsulated as different plug-ins. Currently, MySQL, Oracle, postgre and other plug-ins have been encapsulated. The interface between the condition generation plug-in and the information collection plug-in is fixed. The condition generation plug-in generates a condition information file. The condition information file exists in XML format and includes database identifier, database table name, and collected condition fields. The information collection plug-in scans the condition information file, parses the conditions, generates an SQL file, encapsulates it into a data information file, and sends it to the headquarters. The collection process is described in detail below.

[0056] According to business needs, formulate the collection content, and write the condition generation plug-in according to the formulated collection content. For example, the collection content is to collect the tb_order table information under the KJ125 instance in the MySQL database every day, with receivingStartTime = today as the collection condition, and collect once every 10 minutes. Then the condition generation plug-in generates a condition information file every 10 minutes. The generated conditions exist in XML form. The generated condition content includes database identifier, table name, and collection conditions. The generated condition information file is placed in the corresponding directory according to the database type. For example, the condition information file of the MySQL database is placed under / MySQL. The condition information file sample is shown in the following table.

[0057] Table 3 Condition information file interface

[0058]

[0059] An example of the condition information file interface is shown in Table 4.

[0060] Table 4

[0061]

[0062] In order to simplify database information and facilitate management, and also for the security of the database, in the above conditional information file, the database information is marked with DBName, and the specific data connection information, including database type, database IP, user name, password, instance name and other information, is stored in the database information lookup table. The plug-in that needs to use the database connection information will query the database information lookup table according to DBName, so that there is no need to write complicated database connection information in the interface, which also increases the security of the database.

[0063] The condition information files of different types of databases are stored in different directories. The information collection plug-in continuously scans the storage directory of the condition information files, parses the DBName and synchronization conditions when the condition information files are scanned, and searches the database information lookup table for database connection information based on the DBName. Connect to the database based on the obtained database connection information. Use the table name and condition in the synchronization condition as parameters, execute the collection action, and generate an SQL file. Take MySQL as an example, and execute the command:

[0064] mysqldump-u$DBUser-h$DBhost-p$DBPass--lock-tables=false-t $DB$table-w″$condition″>$sqlFile

[0065] In the above command, characters with $ are variables, $DBUser is the database user name, $DBhost is the database IP address, $DBPass is the database password, $DB is the database instance name, $table is the database table name, $conditon is the synchronization condition, and $sqlFile is the location of the generated SQL file.

[0066] The generated SQL file is called a data information file, the information collection plug-in generates an information description file according to the synchronization conditions, and the information collection plug-in packages the data information file and the information description file together and sends them to the headquarters. The file package sent to the headquarters is called the first data file package.

[0067] In this embodiment, in order to adapt to the unstable network situation, the remote subsystem and the headquarters interact through files. The remote subsystem sends the first data file package to the headquarters at a set frequency. The content of the first data file package includes: data information file and information description file. In order to adapt to multi-source heterogeneous databases, the data information file adopts SQL format, and the condition information file is in XML format. The content of the information description file includes synchronization conditions and time stamp information. The information description file interface is shown in the following table.

[0068] Table 5 Information description file interface

[0069] Node Name content Remark Collect Root Node Source Collect's child nodes Remote subsystem DBName Collect's child nodes Database ID Table Collect's child nodes Synchronized database table name Condition Collect's child nodes Synchronization conditions Time Collect's child nodes Synchronous time, time stamp taskSN Collect's child nodes Synchronize task number

[0070] To save network bandwidth, the first data file package is transmitted in the form of a compressed package zip, and the naming rule is: CollectSql_$Source_$taskSN_$Time.zip. $Time is the time stamp of the current first data file package, and the time stamp is in the form of yyyyMMddHHmmss. yyyy represents year, MM represents month, dd represents day, HH represents hour, mm represents minute, and ss represents second.

[0071] According to the above interface definition, in this embodiment, according to operation S110, the collection service deployed on each remote subsystem performs synchronization tasks at a preset frequency and according to preset synchronization conditions to generate a data information file. Operation S110 may specifically include operations S111 to S112.

[0072] In operation S111, a condition generation plug-in of a collection service is called at a preset frequency, and an information description file is generated according to a preset synchronization condition. The information description file includes a database flag, a database table name, and a data collection condition.

[0073] In operation S112, the data collection plug-in of the collection service is called to search for database connection information according to the data mark, to connect to the corresponding database according to the database connection information, and to collect data in the database table corresponding to the database table name according to the data collection condition to generate a data information file.

[0074] Figure 4 A schematic diagram of an application scenario of a synchronization service provided by an embodiment of the present disclosure is schematically shown.

[0075] like Figure 4 As shown, the synchronization service of the disclosed embodiment includes two categories: deletion plug-in and injection plug-in. The deletion plug-in passes the synchronization conditions and the name of the synchronized database table as parameters to delete the data that meets the conditions. The injection plug-in runs the SQL file to inject the information into the database. Due to different types of relational databases, the commands for running SQL are slightly different, and different plug-ins need to be packaged according to different types of databases. Taking the MySQL database as an example, the source command is used, while the @ command is used in Oracle.

[0076] The workflow of the synchronization service includes the following operations.

[0077] In operation S410, after receiving the first data file package, the data management system calls the synchronization service to parse the time stamp and the task order number from the file name of the first data file package.

[0078] In operation S420, according to the task order number, the latest time stamp of the synchronization service is queried to determine whether a data file package with the same task order number and a time stamp earlier than the current time stamp has been received. If there is a data file package with the same task order number and a time stamp earlier than the current time stamp, the first data file package is parsed to obtain the synchronization condition and the data information file to update the database. If there is a data file package with the same task order number and a time stamp later than the current time stamp, the first data file package is abandoned.

[0079] The above design ensures that the first data file packets are processed in the correct order when the network is unstable.

[0080] Specifically, the data synchronization method includes the following steps.

[0081] In operation S421, database connection information is searched according to the database identifier to connect to the corresponding database.

[0082] In operation S422, the database table name and the data collection condition are passed to a preset deletion script to delete the data related to the deletion condition in the corresponding database table in the database.

[0083] In operation S423, after the data deletion is completed, the data included in the data information file is inserted into the database table to complete the data synchronization.

[0084] Among them, in the database information lookup table of the headquarters, each database identifier and the corresponding data connection information are recorded, including IP and user name, password, database instance name, and database type. The database connection information of the same database identifier may be different in the headquarters and remote subsystems. For example, the database marked as A is a MySQL database in the remote subsystem, and the database marked as A in the headquarters is an Oracle database. This design ensures that the system can be compatible with databases from different sources.

[0085] Figure 5 A schematic diagram of a remote database network architecture provided by an embodiment of the present disclosure is schematically shown.

[0086] like Figure 5 As shown, in the disclosed embodiment, the networks between each remote subsystem and the headquarters are connected, and data communication is established between them. Under normal circumstances, the remote subsystem sends data directly to the headquarters. Under abnormal circumstances, due to some reasons, when the communication between one of the remote subsystems and the data management system fails, in order to send data to the headquarters as much as possible, the remote subsystem is controlled to forward the data file package subsystem through other remote subsystems to try to send data to other subsystems connected to it. After receiving the data, if the network with the headquarters is normal, the other subsystems will forward the data to the headquarters. If the network between other subsystems and the headquarters also fails, the failure will be directly returned to the subsystem that sent the data, and the subsystem will no longer try to forward through this route.

[0087] Figure 6 A schematic diagram of interaction between a remote subsystem and a headquarters provided by an embodiment of the present disclosure is schematically shown.

[0088] like Figure 6 As shown in the figure, the remote subsystems can interact with the data management system (i.e., the headquarters). The remote subsystems configure the push content and push frequency, and actively push database information at the set frequency every day. The headquarters receives the database information of the remote subsystems, parses it and stores it in the database. The data management system can issue synchronization tasks to each remote subsystem, and the synchronization tasks include synchronization conditions and data synchronization frequency. The remote subsystems receive the configuration instructions issued by the headquarters and adjust the collected content and configuration.

[0089] In this embodiment, the condition generation plug-in of the collection service can be implemented based on Quartz, and the condition generation is managed as a JOB. The frequency and time range of the condition generation are controlled based on the configuration. And the configuration instruction from the headquarters can be received. The specific process includes: the condition generation plug-in generates the condition information file according to the frequency and time range configured by itself by default, for example, the configuration is: "00 / 5***?", which means it is executed every 5 minutes, where each symbol in "***?" represents an additional configuration information; the condition generation plug-in receives the configuration instruction from the headquarters, parses the configuration information, and readjusts the frequency and time of generating the condition information file according to the requirements of the configuration instruction. For example, if the configuration frequency in the configuration instruction is "00 / 514**?", it will run every 5 minutes from 14:00 to 15:00 every day, starting at 14:00 and ending at 14:55; after the condition generation plug-in adjusts the configuration, it generates a new configuration file and sends it to the headquarters so that the headquarters can grasp the frequency of condition generation in the current remote system. If the configuration adjustment fails, the current configuration is fed back to the headquarters. This design makes it easy for the headquarters to manage the synchronization tasks of remote subsystems and control the frequency and time of synchronization, without having to log in to the remote subsystem to adjust the configuration.

[0090] According to Figures 1 to 6 The remote database synchronization method shown can adapt to situations where network bandwidth is limited, and can adapt to multiple databases and databases from different sources. At the same time, it realizes the headquarters' configuration management of remote subsystems and the dynamic loading of remote subsystem plug-ins, reducing the difficulty of headquarters' management of remote subsystems.

[0091] Figure 7 The structural block diagram of a remote database synchronization device provided by an embodiment of the present disclosure is schematically shown.

[0092] like Figure 7 As shown, another embodiment of the present disclosure provides a remote database synchronization device 700 including: a data collection module 710, a data forwarding module 720, and a data synchronization module 730.

[0093] The data collection module 710 is used to execute synchronization tasks at a preset frequency and according to preset synchronization conditions through the collection services deployed on each remote subsystem to generate data information files. The synchronization tasks include collecting and packaging data, and the data information files are SQL files.

[0094] The data forwarding module 720 is used to package the synchronization conditions and data information files into a first data file package and send it to the data management system. The file name of the first data file package includes the time stamp and task order number of this data collection.

[0095] The data synchronization module 730 is used to start the synchronization service deployed in the data management system to parse the first data file package when the data management system receives the first data file package, and after determining that the time stamp and task order number meet the preset conditions, update the database according to the synchronization conditions and data information file.

[0096] It should be noted that the remote database synchronization device provided in the embodiment of the present disclosure has the same technical features as the remote database synchronization provided in the embodiment of the present disclosure, and therefore can also achieve the same technical effects, so they will not be described in detail here.

[0097] It is understandable that the data collection module 710, the data forwarding module 720, and the data synchronization module 730 can be combined in one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the data collection module 710, the data forwarding module 720, and the data synchronization module 730 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in a proper combination of software, hardware, and firmware. Alternatively, at least one of the data collection module 710, the data forwarding module 720, and the data synchronization module 730 can be at least partially implemented as a computer program module, and when the program is run by a computer, the function of the corresponding module can be executed.

[0098] Figure 8 The structural block diagram of an electronic device provided by an embodiment of the present disclosure is schematically shown.

[0099] like Figure 8 As shown, the electronic device described in this embodiment includes: the electronic device 800 includes a processor 810 and a computer readable storage medium 820. The electronic device 800 can execute the above reference Figure 1 Describes the method to achieve detection of specific operations.

[0100] Specifically, the processor 810 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 810 may also include an onboard memory for cache purposes. The processor 810 may be a processor for executing reference Figure 1 The described method flow according to the embodiment of the present disclosure may be a single processing unit or multiple processing units for different actions.

[0101] Computer readable storage medium 820, for example, can be any medium capable of containing, storing, conveying, propagating or transmitting instructions. For example, readable storage media can include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices, devices or propagation media. Specific examples of readable storage media include: magnetic storage devices, such as magnetic tapes or hard disks (HDD); optical storage devices, such as compact disks (CD-ROMs); memory, such as random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0102] The computer readable storage medium 820 may include a computer program 821, which may include code / computer executable instructions, which when executed by the processor 810 causes the processor 810 to perform, for example, the above-mentioned combination Figure 1 The described method sequences and any variations thereof.

[0103] The computer program 821 may be configured to have, for example, computer program codes including computer program modules. For example, in an exemplary embodiment, the codes in the computer program 821 may include one or more program modules, for example, 821A, 821B, ... It should be noted that the division method and number of modules are not fixed, and those skilled in the art may use appropriate program modules or program module combinations according to actual conditions. When these program module combinations are executed by the processor 810, the processor 810 may execute, for example, the above combination of Figure 1-2 The described method sequences and any variations thereof.

[0104] According to an embodiment of the present invention, at least one of the data collection module 710, the data forwarding module 720, and the data synchronization module 730 can be implemented as a reference. Figure 7 The computer program modules described herein, when executed by the processor 810, can implement the corresponding operations described above.

[0105] The present disclosure also provides a computer-readable medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0106] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0107] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. A remote database synchronization method, applied to a remote deployment system, wherein the remote deployment system includes at least one data management system and multiple remote subsystems, characterized in that: include: The data information file is generated by executing the synchronization task at a preset frequency and according to the preset synchronization conditions through the collection service deployed on each of the remote subsystems, including: calling the condition generation plug-in of the collection service at a preset frequency, generating an information description file according to the preset synchronization conditions, the information description file including the database flag, database table name and data collection conditions, wherein the condition generation plug-in is encapsulated according to the type of database served; calling the data collection plug-in of the collection service, searching for database connection information according to the database flag, connecting to the corresponding database according to the database connection information, and collecting the data in the database table corresponding to the database table name according to the data collection conditions, and generating the data information file; wherein the synchronization task includes collecting and encapsulating data according to the synchronization conditions, and the data information file is an SQL file; Packing the synchronization condition and the data information file into a first data file package and sending it to the data management system, wherein the file name of the first data file package includes a time stamp and a task order number of this data collection; When the data management system receives the first data file package, the synchronization service deployed in the data management system is called to parse the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, the database is updated according to the synchronization conditions and the data information file, including: when the data management system receives the first data file package, the synchronization service is called to parse the time stamp and the task order number from the file name of the first data file package; according to the task order number, the latest time stamp of the synchronization service is queried to determine whether a data file package with the same task order number and a time stamp earlier than the current time stamp has been received; if there is a data file package with the same task order number and a time stamp earlier than the current time stamp, the first data file package is parsed to obtain the synchronization conditions and data information file to update the database; if there is a data file package with the same task order number and a time stamp later than the current time stamp, the first data file package is abandoned.

2. The method according to claim 1, characterized in that The synchronization condition includes a database flag, a database table name and a data collection condition, the data collection condition includes a deletion condition, and updating the database according to the synchronization condition and the data information file includes: Searching for database connection information according to the database identifier to connect to the corresponding database; Passing the database table name and the data collection condition to a preset deletion script to delete the data related to the deletion condition in the corresponding database table in the database; After the data deletion is completed, the data included in the data information file is inserted into the database table to complete the data synchronization.

3. The method according to claim 1, characterized in that The collection service and the synchronization service are designed as plug-ins, and the startup methods of the collection service and the synchronization service include: Scan the XML configuration files of the collection service and / or the synchronization service regularly, start the plug-in that needs to be loaded, and calculate the MD5 value of the plug-in; Compare the current MD5 value of the plug-in with the previous MD5 value, and if the two are different, restart the plug-in; Before restarting the plug-in, confirm that there is no synchronization task being executed in the remote subsystem; If there is a synchronization task being executed, the plug-in is restarted after the synchronization task is completed.

4. The method according to claim 1, characterized in that: The method further comprises: Sending synchronization tasks to each of the remote subsystems through the data management system, wherein the synchronization tasks include the synchronization conditions and data synchronization frequency; The update plug-in of the collection service is issued to each of the remote subsystems through the data management system.

5. The method according to claim 1, characterized in that The method further comprises: Establishing communication between the data management system and each of the remote subsystems and between each of the remote subsystems; When there is a communication failure between one of the remote subsystems and the data management system, the remote subsystem is controlled to forward the data file package through other remote subsystems.

6. A remote database synchronization device, used to execute the remote database synchronization method according to any one of claims 1 to 5, characterized in that: include: A data collection module, used to generate a data information file by executing a synchronization task at a preset frequency and according to preset synchronization conditions through a collection service deployed on each of the remote subsystems, wherein the synchronization task includes collecting and packaging data, and the data information file is an SQL file; A data forwarding module, used for packaging the synchronization condition and the data information file into a first data file package, and sending the first data file package to the data management system, wherein the file name of the first data file package includes a time mark and a task order number of this data collection; The data synchronization module is used to start the synchronization service deployed in the data management system to parse the first data file package when the data management system receives the first data file package, and after determining that the time stamp and the task order number meet the preset conditions, update the database according to the synchronization conditions and the data information file.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the remote database synchronization method described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the remote database synchronization method described in any one of claims 1 to 5 is implemented.

Images