Time series data master-slave synchronization method and system, computer equipment and computer readable storage medium
The method enhances data synchronization in power grids by caching and batch processing time series data across multiple databases, addressing data loss during master failures and ensuring reliable data storage.
Patent Information
- Application Number
- CN202411219585.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-03
AI Technical Summary
Existing master-slave synchronization methods for time series data in power grids suffer from data loss during master database failures, affecting data integrity and accuracy.
A method and system for independent synchronization of time series data to multiple target databases, including data caching, batch execution, and failed command management to ensure reliable data storage.
Improves data integrity and accuracy by writing synchronized data to multiple databases simultaneously and managing failed commands, ensuring redundancy and reliability.
Smart Images
Figure CN119248581A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of data storage, and in particular, relates to a master-slave synchronization method and system for time series data, a computer device, and a computer-readable storage medium. Background Art
[0002] With the construction of smart grids, the scale of power grids is becoming increasingly large, and each substation will generate a large amount of operating data. These operating data contain rich information and have important value. They can be used to monitor the real-time operation of substations, trace the historical operation of substations, and provide necessary data support for fault handling performance optimization.
[0003] For such important data, it is necessary to design a suitable redundant storage system to ensure the reliability and continuity of the data and avoid the loss of critical data in the event of a failure. Summary of the invention
[0004] In view of this, an embodiment of the present application provides a time series data master-slave synchronization method, system, computer device and computer-readable storage medium to solve the problem of data loss during master-slave switching in the prior art.
[0005] A first aspect of an embodiment of the present application provides a master-slave synchronization method for time series data, the method being applied to a target computer device, the target computer device comprising an application module and a synchronization module, the method comprising:
[0006] Get the data to be synchronized;
[0007] The application module sends the data to be synchronized to the synchronization module;
[0008] The synchronization module writes the data to be synchronized into multiple target time series databases respectively.
[0009] In an implementation of the first aspect, before the synchronization module writes the to-be-synchronized data into a plurality of target time series databases respectively, the method further includes:
[0010] The synchronization module establishes connections with multiple target time series databases respectively.
[0011] In an implementation of the first aspect, after the application module sends the data to be synchronized to the synchronization module, the method further includes:
[0012] The synchronization module stores the data to be synchronized in a cache queue in the format of a command to be executed, where the command to be executed includes the data to be synchronized;
[0013] The synchronization module writes the to-be-synchronized data into a plurality of target time series databases respectively, including:
[0014] The synchronization module executes the commands to be executed on the multiple target time series databases respectively, so as to write the data to be synchronized into the multiple target time series databases respectively.
[0015] In an implementation of the first aspect, the step of executing the to-be-executed commands on the multiple target time series databases respectively by the synchronization module includes:
[0016] The synchronization module extracts the commands to be executed from the cache queue in groups, each group including a plurality of commands to be executed;
[0017] The synchronization module executes each group of commands to be executed on the multiple target time series databases to write the data to be synchronized into the multiple target time series databases in batches.
[0018] In an implementation of the first aspect, after the synchronization module executes each group of to-be-executed commands on the multiple target time series databases, the method further includes:
[0019] The synchronization module writes the failed to be executed command as a failed command into a failed command file, wherein the failed command file corresponds to the target time series database of the failed to be synchronized data;
[0020] The synchronization module performs supplementary writing based on the failed command file.
[0021] In an implementation of the first aspect, the performing, by the synchronization module, rewriting based on the failed command file includes:
[0022] The synchronization module detects the node states of the multiple target time series databases respectively;
[0023] The synchronization module executes the failure command in the failure command file on the target time series database with normal node status according to the failure command file and its storage path, so as to write the failed data to be synchronized into its corresponding target time series database.
[0024] In an implementation of the first aspect, the method further includes:
[0025] The synchronization module marks the failed command that succeeded in rewriting as successful rewriting, and marks the failed command that failed in rewriting as failed rewriting;
[0026] The synchronization module clears all the successfully rewritten failed command files and expired failed command files.
[0027] A second aspect of an embodiment of the present application provides a time series data master-slave synchronization system, including:
[0028] A data acquisition module, used to acquire data to be synchronized;
[0029] An application module, used for sending the data to be synchronized to a synchronization module;
[0030] The synchronization module is used to write the data to be synchronized into multiple target time series databases respectively.
[0031] A third aspect of an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the master-slave synchronization method for timing data as described in the first aspect are implemented.
[0032] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the master-slave synchronization method of time series data as described in the first aspect are implemented.
[0033] The beneficial effect of the first aspect of the embodiment of the present application is: after obtaining the data to be synchronized, the application module sends the data to be synchronized to the synchronization module, and the synchronization module writes the data to be synchronized into multiple target time series databases respectively, thereby realizing independent synchronization of the data to be synchronized in multiple target time series databases, thereby improving the integrity and accuracy of data synchronization and making synchronization more reliable.
[0034] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 It is a schematic diagram of the implementation flow of a time series data master-slave synchronization method provided in an embodiment of the present application;
[0037] Figure 2 It is a schematic diagram of the architecture of a time series data master-slave synchronization system provided in an embodiment of the present application;
[0038] Figure 3 It is a schematic diagram of the implementation flow of a time series data master-slave synchronization method provided in an embodiment of the present application;
[0039] Figure 4It is a schematic diagram of the implementation flow of a time series data master-slave synchronization method provided in an embodiment of the present application;
[0040] Figure 5 It is a schematic diagram of the implementation flow of a time series data master-slave synchronization method provided in an embodiment of the present application;
[0041] Figure 6 It is a schematic diagram of a master-slave synchronization system for time series data provided by an embodiment of the present application;
[0042] Figure 7 It is a schematic diagram of the architecture of a time series data master-slave synchronization system provided in an embodiment of the present application;
[0043] Figure 8 It is a schematic diagram of a target computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0046] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0047] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0048] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0049] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0050] The embodiment of the present application provides a master-slave synchronization method for time series data, which performs master-slave synchronization based on logs. The method includes writing the data to be synchronized into the master database and generating corresponding logs; and synchronizing the slave database with the master database based on the logs. However, in this method, if the master database fails to write the data to be synchronized, a master-slave switch is required, which will cause data loss during this period, affecting the integrity and accuracy of the data.
[0051] The embodiment of the present application provides a master-slave synchronization method for time series data applied to a target computer device, which is used to improve the integrity and accuracy of data synchronization and make synchronization more reliable. After acquiring the data to be synchronized, the application module sends the data to be synchronized to the synchronization module, and the synchronization module writes the data to be synchronized into multiple target time series databases respectively, so as to realize independent synchronization of the data to be synchronized in multiple target time series databases, thereby improving the integrity and accuracy of data synchronization and making synchronization more reliable.
[0052] In the application, the target computer device can be any terminal device that can run a computer program with corresponding functions, such as a laptop computer, a desktop computer, a personal computer (PC), an industrial computer, a server, etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.
[0053] like Figure 1 As shown, a master-slave synchronization method for time series data provided by an embodiment of the present application is applied to a target computer device, the target computer device includes an application module and a synchronization module, and the method includes:
[0054] Step S10, obtaining data to be synchronized.
[0055] In the application, the data to be synchronized include voltage and current measurement values, power measurement values, switch status information, relay protection action records, fault recording data, load data, control and dispatch data, communication and network data and other power automation system operation data, which can be obtained through various signal acquisition devices or supervisory control and data acquisition (SCADA) systems. It is understandable that some power automation system operation data can be generated by application modules, such as control and dispatch data and communication and network data.
[0056] Step S20: the application module sends the data to be synchronized to the synchronization module.
[0057] In the application, the target computer device includes at least one application module, and the target computer device sends the data to be synchronized to the synchronization module through the at least one application module. In the case where the target computer device includes multiple application modules, each application module is responsible for different operation data, and the target computer device can send the corresponding data to be synchronized to the synchronization module through each application module according to preset rules.
[0058] Step S30: The synchronization module writes the data to be synchronized into multiple target time series databases respectively.
[0059] In the application, the data to be synchronized is time series data, which is data recorded in chronological order. Each data point has a timestamp and can reflect the trends and patterns that change over time.
[0060] In the application, the multiple target time series databases include a master time series database and at least one slave time series database. The number of slave time series databases is generally 1-3, which can be set according to the importance of the data. In the embodiment of the present application, the multiple target time series databases include a master time series database and a slave time series database as an example for explanation. A master time series database and a slave time series database constitute a redundant database system.
[0061] In the application, multiple target time series databases are on computer devices other than the target computer device, the master time series database and the slave time series database are on different computer devices, and different slave time series databases may or may not be on the same computer device.
[0062] In the application, the master time series database and the slave time series database are pre-configured. Every time the master time series database fails, a master-slave switch is performed, and the slave time series database is set as the new master time series database, and the master time series database is set as the new slave time series database. In the case where multiple target time series databases include a master time series database and more than two slave time series databases, each time the master time series database fails, multiple slave time series databases perform master-slave switch with the master time series database in a preset order. For example, multiple target time series databases include a master time series database A, two slave time series databases B and C. When the first master-slave switch is required, the slave database B is set as the master time series database. When the master-slave switch is required for the second time, the database C is set as the master time series database. When the master-slave switch is required for the third time, the database A is set as the master time series database, and the master-slave switch is performed in this order in a cycle. It can be understood that the master-slave switch can be quickly realized between the master and slave time series databases through the above method. Compared with the cluster system that needs to implement a complex master node election algorithm and a complex data synchronization mechanism between nodes, the method provided by the present application is more lightweight.
[0063] In the application, the synchronization module writes the data to be synchronized into multiple target time series databases respectively, including writing the data to be synchronized into the master time series database and the slave time series database respectively. In the application, the data to be synchronized is written into the master time series database and the slave time series database by performing the same write operation on the master time series database and the slave time series database. It can be understood that writing the data to be synchronized into the master time series database and writing the data to be synchronized into the slave time series database are independent of each other, and can be performed simultaneously or with a certain time difference. By writing the data to be synchronized into the master time series database and the slave time series database respectively, it is possible to store the data to be synchronized into multiple databases to ensure data security, and in the scenario where the master time series database fails, it is ensured that the slave time series database is not affected, thereby avoiding data loss, ensuring data integrity, and improving the reliability of synchronization.
[0064] In one embodiment, before step S30, the method further includes:
[0065] The synchronization module establishes connections with multiple target time series databases respectively.
[0066] In the application, the synchronization module establishes connections with the master time series database and the slave time series database respectively, including establishing connections with the host where the master time series database is located and the host where the slave time series database is located based on the Transmission Control Protocol / Internet Protocol (TCP / IP). TCP / IP includes application layer, transport layer, network layer, data link layer and physical layer, wherein the physical layer defines the physical medium and interface standards for data transmission, such as Ethernet, optical fiber, etc. In the application, the connection established with different target time series databases can realize data communication based on different physical layers to improve the independence of communication with each target time series database, and avoid the failure of the physical communication link to cause the failure of writing the data to be synchronized into multiple target time series libraries.
[0067] like Figure 2 As shown, an architecture diagram of the master-slave synchronization method for time series data provided by an embodiment of the present application, the target computer device obtains the data to be synchronized by connecting the signal acquisition device through a network, and an application module sends the data to be synchronized to the synchronization module, and the synchronization module writes the data to be synchronized into a target time series database including a master time series database and a slave time series database through the network.
[0068] like Figure 3 As shown, in one embodiment, after step S20, the method further includes:
[0069] In step S21, the synchronization module stores the data to be synchronized in a cache queue in the format of commands to be executed, where the commands to be executed include the data to be synchronized.
[0070] In the application, the commands to be executed include CREATE statements for creating databases and data tables, and INSERT statements for inserting data. The CREATE statement is used to create a database or a data table in the database as needed. After the data table is created, the INSERT statement is used to insert the data to be synchronized into the data table.
[0071] The step S30 includes:
[0072] Step S31: The synchronization module executes the to-be-executed commands on the multiple target time series databases respectively, so as to write the to-be-synchronized data into the multiple target time series databases respectively.
[0073] In the application, the instructions to be executed in the cache queue are executed for multiple target timing libraries respectively, including:
[0074] Execute the INSERT statements in the cache queue for multiple target time series databases respectively, so as to write the data to be synchronized into multiple target time series databases respectively.
[0075] In the application, the above steps are implemented based on multi-threading. Through multiple threads in the thread pool, the commands to be executed are respectively executed on multiple target time series databases, and the data to be synchronized are respectively written into the multiple target time series databases. By executing the commands to be executed through multiple threads, the task parallelism is improved, thereby improving the writing efficiency.
[0076] like Figure 4 As shown, in one embodiment, the step S31 includes:
[0077] Step S311: The synchronization module takes out the commands to be executed from the cache queue in groups, each group including a plurality of commands to be executed.
[0078] Step S312: The synchronization module executes each group of commands to be executed on the multiple target time series databases to write the data to be synchronized into the multiple target time series databases in batches.
[0079] In the application, the frequency at which the synchronization module extracts the commands to be executed from the cache queue in groups can be dynamically configured according to the occupancy of the cache queue, the occupancy of system resources such as processor resources, communication bandwidth, etc. The number of commands to be executed included in each group can be set according to the occupancy of the cache queue and the database performance.
[0080] In the application, after taking out the to-be-executed commands from the cache queue in groups, the command integration is also included, which optimizes and merges multiple to-be-executed commands in a group to construct batch execution SQL statements supported by the target timing library, thereby reducing the number of statements and improving execution efficiency.
[0081] In one embodiment, after step S312, the method further includes:
[0082] In step S40, the synchronization module writes the failed to be executed command as a failed command into a failed command file, and the failed command file corresponds to the target time series database of the failed to be synchronized data.
[0083] In the application, in the case of a pending command that fails to execute, the pending command that fails to execute is written as a failed command into the corresponding failed command file, and the name of the failed command file includes the start time of the period described by the timestamp of the pending command failure and the identifier (host name or host address) of the computing device to which the target time series database corresponding to the failed command belongs. The start time of the period described by the timestamp of the pending command failure is calculated as follows: the timestamp of the pending command failure is rounded to the calculated period according to the preset rounding time, for example, rounded forward by five minutes, and the start time of the period to which the current timestamp belongs is calculated (each hour includes 12 periods and 12 start times, and each period is five minutes). The start time includes year, month, day, hour, minute, and second, and is unique. The failed command file is stored in a preset directory. If there is a pending command that fails to execute, and a failed command file with a file name including the same computer device identifier and the same start time is not found in the preset directory, a failed command file including the start time of the cycle described by the timestamp of the failure of the pending command to execute and the identifier of the computing device to which the target time series database corresponding to the failed command belongs is created in the preset directory, for example, a failed command file with the file name host1_cmdlist_20240312_145601.bin, where host1 is the host identifier and 20240312_145601 is the start time of a cycle.
[0084] In the application, the application module that sends the data to be synchronized to the synchronization module is the caller of the synchronization module, and there can be only one or more callers on the target computer device. In the case where there is only one caller on the target computer device, the preset directory can be any preset directory on the target computer device. In the case of multiple callers, it is necessary to distinguish the failure command files of different callers through preset rules. For example, the storage path of the local failure command file directory is constructed according to the caller's identifier (process name or process number, etc.), the identifier of the computer device to which the target timing library belongs (host name or host address), and the name of each database. For each failure command file, it is stored in the local file directory under the corresponding storage path according to the current caller identifier, the identifier of the computer device included in the failure command file name, and the name of the database currently being operated. The storage path can be: GetCygResourceDataPath+" / TmsdbSyncData / Caller ID / Timing Library Node Host Name / DbName", where GetCygResourceDataPath is a custom resource path, "Caller ID" is the process name or process number of the current caller, "Timing Library Node Host Name" is the host name or host address of the computer device described in the target timing database section corresponding to the failed command to be executed, and "DbName" is the database name corresponding to the failed command to be executed.
[0085] In the application, the content structure of the failed command file is as follows:
[0086]
[0087]
[0088] Through the above-mentioned method of naming failed command files and configuring the storage path of failed command files, it is possible not only to manage failed commands according to the start time of the cycle to which they belong, but also to manage failed command files of different callers separately, thereby achieving refined management of failed command files and improving the efficiency of searching and calling failed commands.
[0089] Step S41: the synchronization module performs rewriting based on the failed command file.
[0090] In the application, the synchronization module writes the failed data to be synchronized into the corresponding target time series database based on the failed command file.
[0091] like Figure 5 As shown, in one embodiment, the step S41 includes:
[0092] Step S411: The synchronization module detects the node states of the multiple target time series databases respectively.
[0093] In the application, the failure of the execution of the command to be executed indicates that the corresponding target time series database has a fault. Before rewriting the failed synchronization data, it is necessary to confirm that the node status of the failed target time series database has returned to normal before rewriting, so as to avoid invalid rewriting (failed rewriting) of the failed target time series database and waste system resources. The synchronization module sends the test command to the computer devices to which each target time series database belongs. If a response message is received indicating that the test command is successfully executed corresponding to the computer device to which the target time series database belongs, it means that the node status of the target time series database is normal.
[0094] In addition, for target time series databases with abnormal node status, it is necessary to prompt for troubleshooting, such as prompting to check whether the communication link is interrupted.
[0095] Step S412: The synchronization module executes the failure command in the failure command file on the target time series database with normal node status according to the failure command file and its storage path, so as to write the failed data to be synchronized into the corresponding target time series database.
[0096] In the application, the computer device of the target time series database corresponding to the failed command file can be known according to the file name of the failed command file, and the database name corresponding to the failed command file can be known according to the storage path of the directory to which the failed command file belongs, thereby realizing the writing of the failed to be synchronized data into its corresponding target time series database.
[0097] In the application, the completion of the rewriting of failed commands in the failed command file is implemented based on multiple rewriting threads, thereby improving the concurrency and efficiency of the rewriting.
[0098] In one embodiment, the method further comprises:
[0099] Step S42: the synchronization module marks the failed command that succeeded in rewriting as a successful rewriting, and marks the failed command that failed in rewriting as a failed rewriting.
[0100] In the application, the failed command that successfully rewrites is marked as rewrite success or deleted, and the failed command that failed to rewrite is marked as rewrite failure. The number of rewrites is recorded after each rewrite failure. For the failed command marked as rewrite failure, multiple rewrites can be performed until the number of rewrites exceeds the preset failure threshold. The failed command whose number of rewrites exceeds the preset failure threshold can be deleted.
[0101] Step S43: the synchronization module clears all the successfully rewritten failed command files and expired failed command files.
[0102] In the application, if all failed commands in a failed command file are marked as successfully rewritten, the failed command file can be deleted to optimize storage space; or the failed command file can be calculated based on the start time included in the file name of the command file to determine whether it is within the validity period, and the failed command files that have exceeded the validity period can be cleaned up to reduce storage space usage. The validity period of the failed command file can be flexibly configured, for example, it can be configured to one hour or two hours.
[0103] In the application, the difference between the start time in the file name of the failed command file and the current time can be calculated to determine whether the failed command file has exceeded its validity period.
[0104] In the application, the expired failed command file can also be transmitted to the extension node for continued rewriting. The extension node is other computer equipment, which can be a cloud server with large storage space.
[0105] In one embodiment of the present application, the functions of the master-slave synchronization module are implemented based on C++ program development and encapsulated as a dynamic library for calling by other application modules of the target computer device.
[0106] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0107] The present application also provides a master-slave synchronization system for time series data, which is used to execute the steps in the above-mentioned master-slave synchronization method embodiment. The master-slave synchronization system for time series data can be a virtual appliance in the target computer device, which is run by the processor of the target computer device, or it can be the target computer device itself.
[0108] like Figure 6 As shown, a time series data master-slave synchronization system 5 provided in an embodiment of the present application includes:
[0109] The data acquisition module 501 is used to acquire the data to be synchronized;
[0110] The application module 502 is used to send the data to be synchronized to the synchronization module;
[0111] The synchronization module 503 is used to write the data to be synchronized into multiple target time series databases respectively.
[0112] In an application, the data acquisition module 501 and the application module 502 may be the same module.
[0113] In the application, the synchronization module includes a cache unit, a thread pool unit, a time series library client unit, a failed command management unit, and a rewrite unit. The cache unit (cache queue) is used to store the data to be synchronized (commands to be executed). The thread pool is used to write the data to be synchronized into the master time series database and the slave time series database through each thread. The time series library client unit is used to connect with the master and slave time series databases (target time series databases) and call the failed command management unit to store the failed commands to be executed as failed commands in the failed command file. The rewrite unit is used to rewrite the failed data to be stored in the failed command file into the corresponding target time series database. Figure 7 The figure shows a schematic diagram of an architecture diagram of a master-slave synchronization system for timing data including the above-mentioned units of the synchronization module provided by an embodiment of the present application.
[0114] In one embodiment, the synchronization module 503 is further configured to establish connections with multiple target time series databases respectively.
[0115] In one embodiment, the synchronization module 503 is further used for:
[0116] storing the data to be synchronized in a cache queue in the format of a command to be executed, wherein the command to be executed includes the data to be synchronized;
[0117] The commands to be executed are respectively executed on the multiple target time series databases to write the data to be synchronized into the multiple target time series databases respectively.
[0118] In one embodiment, the synchronization module 503 is further used for:
[0119] Taking out the commands to be executed from the cache queue in groups, each group including a plurality of commands to be executed;
[0120] Each group of commands to be executed is executed on the multiple target time series databases to write the data to be synchronized into the multiple target time series databases in batches.
[0121] In one embodiment, the synchronization module 503 is further used for:
[0122] The to-be-executed command that failed to be executed is written as a failed command into a failed command file, and the failed command file corresponds to the target time series database of the to-be-synchronized data that failed to be written.
[0123] In one embodiment, the synchronization module 503 is further used for:
[0124] Respectively detecting whether the node states of the multiple target time series databases are normal;
[0125] If normal, check whether the failure command files corresponding to each target time series database are empty;
[0126] If it is not empty, the failure command in the corresponding failure command file is executed on the corresponding target time series database to rewrite the failed data to be synchronized into the corresponding target time series database.
[0127] In one embodiment, the synchronization module 503 is further used for:
[0128] Mark the failed command that successfully rewrites as successful rewrite, and mark the failed command that failed to rewrite as failed rewrite;
[0129] Clear all successfully rewritten failed command files and expired failed command files.
[0130] In applications, each module in the time series data master-slave synchronization system may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.
[0131] Figure 8 This is a schematic diagram of the structure of a target computer device provided by an embodiment of the present application. Figure 8 As shown, the target computer device 6 of this embodiment includes: at least one processor 60 ( Figure 8 Only one is shown in the figure) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, and when the processor 60 executes the computer program 62, the steps in any of the above-mentioned embodiments of the master-slave synchronization method for timing data are implemented.
[0132] The target computer device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 8 It is only an example of the target computer device 6 and does not constitute a limitation on the target computer device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.
[0133] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0134] In some embodiments, the memory 61 may be an internal storage unit of the target computer device 6, such as a hard disk or memory of the target computer device 6. In other embodiments, the memory 61 may also be an external storage device of the target computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the target computer device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the target computer device 6. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 61 may also be used to temporarily store data that has been output or is to be output.
[0135] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0136] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0137] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0138] An embodiment of the present application provides a computer program product. When the computer program product is run on a target computer device, a mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0140] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0141] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0142] In the embodiments provided in the present application, it should be understood that the disclosed target computer device and method can be implemented in other ways. For example, the target computer device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0144] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A master-slave synchronization method for time series data, characterized in that: The method is applied to a target computer device, the target computer device includes an application module and a synchronization module, and the method includes: Get the data to be synchronized; The application module sends the data to be synchronized to the synchronization module; The synchronization module writes the data to be synchronized into multiple target time series databases respectively.
2. The master-slave synchronization method for time series data according to claim 1, characterized in that: Before the synchronization module writes the to-be-synchronized data into a plurality of target time series databases respectively, the method further includes: The synchronization module establishes connections with multiple target time series databases respectively.
3. The master-slave synchronization method for time series data according to claim 1 or 2, characterized in that: After the application module sends the data to be synchronized to the synchronization module, the method further includes: The synchronization module stores the data to be synchronized in a cache queue in the format of a command to be executed, where the command to be executed includes the data to be synchronized; The synchronization module writes the to-be-synchronized data into a plurality of target time series databases respectively, including: The synchronization module executes the commands to be executed on the multiple target time series databases respectively, so as to write the data to be synchronized into the multiple target time series databases respectively.
4. The master-slave synchronization method for time series data according to claim 3, characterized in that: The step of executing the to-be-executed commands on the multiple target time series databases respectively by the synchronization module includes: The synchronization module extracts the commands to be executed from the cache queue in groups, each group including a plurality of commands to be executed; The synchronization module executes each group of commands to be executed on the multiple target time series databases to write the data to be synchronized into the multiple target time series databases in batches.
5. The master-slave synchronization method for time series data according to claim 4, characterized in that: After the synchronization module executes each group of commands to be executed on the multiple target time series databases, the method further includes: The synchronization module writes the failed to be executed command as a failed command into a failed command file, and the failed command file corresponds to the target time series database of the failed to be synchronized data; The synchronization module performs supplementary writing based on the failed command file.
6. The master-slave synchronization method for time series data according to claim 5, characterized in that: The rewriting by the synchronization module based on the failed command file includes: The synchronization module detects the node states of the multiple target time series databases respectively; The synchronization module executes the failure command in the failure command file on the target time series database with normal node status according to the failure command file and its storage path, so as to write the failed data to be synchronized into the corresponding target time series database.
7. The master-slave synchronization method for time series data according to claim 6, characterized in that: The method further comprises: The synchronization module marks the failed command that succeeded in rewriting as successful rewriting, and marks the failed command that failed in rewriting as failed rewriting; The synchronization module clears all the successfully rewritten failed command files and expired failed command files.
8. A time series data master-slave synchronization system, characterized in that: include: A data acquisition module, used to acquire data to be synchronized; An application module, used for sending the data to be synchronized to a synchronization module; The synchronization module is used to write the data to be synchronized into multiple target time series databases respectively.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.