Data synchronization method of master application server, storage medium and electronic device

Abstract

The application discloses a data synchronization method of a main application server, which comprises the following steps: analyzing an instruction sent by an external device and obtaining a plurality of instruction data; each kind of instruction data is to be synchronized to a target; a plurality of log files are generated according to each target; each log file comprises a generation time stamp of the file and a host name of the target; an index file is generated according to all the log files corresponding to the same target; each index file comprises the host name of the target, current write file information and current synchronization information; a log file to be written is acquired according to the host name of the target and the current write file information, so as to write corresponding instruction data into the log file to be written; a log file to be synchronized is acquired according to the host name of the target and the current synchronization information, so as to read the instruction data written in the log file to be synchronized; and when a synchronization task triggering condition is met, the instruction data read from the log file to be synchronized is synchronized to the corresponding target.

Description

Technical Field

[0001] This invention relates to the field of rail transit technology, and in particular to a data synchronization method, storage medium, and electronic device for a main application server. Background Technology

[0002] The integrated monitoring system for rail transit, with train operation at its core, is an application of data acquisition and monitoring control systems in the rail transit field. This system is large-scale, encompassing data acquisition and control functions from multiple functional systems, including power monitoring, electromechanical monitoring, fire monitoring, broadcasting, and automatic fare collection. To improve the reliability and security of the integrated monitoring system, redundancy is employed during its design and implementation. This redundancy includes application server redundancy, with a primary and backup application server. Maintaining data consistency between the primary and backup application servers is crucial for system stability.

[0003] Currently, the consistency of data between the primary and backup application servers is guaranteed by the primary application server. That is, the primary application server collects and processes data and receives and processes control commands, and synchronizes the collected data and received commands to the backup application server. Usually, this synchronization is based on the rollback log mode, that is, directly modifying the data on the target machine. If the modification fails, it will roll back to the original state, which will lead to inconsistency between the primary and backup application servers.

[0004] Furthermore, in certain special scenarios, commands sent by the client need to be received and processed by both the primary and backup application servers, such as commands for tag placement, tag removal, manual data setting, and manual position setting in power monitoring systems. If abnormal power outages, system crashes, or network failures occur during command transmission, the reliability and consistency of data on the primary and backup application servers will be affected, potentially leading to serious accidents. Taking a power monitoring system as an example, under normal circumstances, when tag placement / tag removal commands are sent to the primary and backup application servers, the primary application server receives the command, performs subsequent processing, and writes it to the database. Specifically, when construction workers are performing power maintenance on-site, the client needs to send a "hang out maintenance sign" command (or "sign-up command") to the primary and backup application servers. Once the sign-up command is correctly issued, control of various power equipment is no longer permitted. After the workers complete their tasks, the client needs to send a "remove the sign" command (or "remove the sign") to the primary and backup application servers. Once the removal command is correctly issued, control of the power equipment is allowed again. However, in the event of power outages, system crashes, or network failures, assuming the primary application server receives the sign-up command but the backup application server does not, and a switchover occurs, the original backup application server, having not received the sign-up command, will experience data inconsistency. After the original backup application server becomes the primary application server, the removal command sent by the client will not execute correctly, preventing the power equipment from being controlled again. Therefore, it is necessary to take measures or methods to ensure data consistency between the primary and backup application servers. Summary of the Invention

[0005] The purpose of this invention is to provide a data synchronization method, storage medium, and electronic device for a primary application server, which can be applied to data synchronization scenarios between primary and backup application servers and between the primary application server and the database, thereby achieving data consistency between the primary and backup application servers.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] A data synchronization method for a main application server, characterized by comprising:

[0008] Obtain configuration information, including the conditions for triggering the synchronization task;

[0009] The system receives and parses instructions sent by external devices to obtain several types of instruction data; and each type of instruction data is to be synchronized to a target, which is a backup application server or a database.

[0010] Several log files are generated according to each target; each log file includes a log header, which includes the generation timestamp of the corresponding log file and the hostname of the corresponding target;

[0011] An index file is generated based on all the log files corresponding to the same target; each index file includes an index header and an index body, the index header includes the hostname of the corresponding target, and the index body includes current file writing information and current synchronization information;

[0012] The log file to be written is obtained based on the hostname of the target in each index file and the current write file information, so as to write the corresponding instruction data into the log file to be written.

[0013] Based on the hostname of the target in each index file and the current synchronization information, obtain the log file to be synchronized, and read the instruction data already written in the log file to be synchronized; and

[0014] When the synchronization task triggering conditions are met, the instruction data read from the log file to be synchronized will be synchronized to the corresponding target.

[0015] Optionally, each type of instruction data is divided into several records and written to the log file to be written.

[0016] Optionally, the configuration information further includes: the maximum number of records m that the log file can store, where m > 0;

[0017] The synchronous task triggering conditions include a first preset condition and a second preset condition;

[0018] The first preset condition is that a synchronization task is triggered when every n records are read from the log file to be synchronized within a first preset time period, where 0 < n ≤ m; and the first preset time period is t seconds, where t > 0;

[0019] The second preset condition is that if the number of records read from the log file to be synchronized within the first preset time period does not reach n, the synchronization task is triggered at t+i seconds, and i>0.

[0020] Optionally, the log header may also include the type of instruction data stored in the corresponding log file, the maximum number of records that can be stored, the number of records currently written w0, and the number of records currently read r0.

[0021] Optionally, the current file writing information includes a first timestamp and a first record count statistic; and the first timestamp is the generation timestamp of the log file to be written; the first record count statistic is the number of records already written in the log file to be written, denoted as w1.

[0022] Optionally, the step of obtaining the log file to be written based on the hostname of the target in each of the index files and the current write file information includes:

[0023] Read the index header of the index file to obtain the hostname of the target;

[0024] Read the index body of the index file to obtain the first timestamp in the current write file information;

[0025] Based on the hostname of the target in the index header, obtain several log files to be written; and the hostname of the target in the log files to be written is the same as the hostname of the target in the index header;

[0026] The log files to be written are sorted in ascending order according to their generation timestamps;

[0027] Based on the first timestamp in the index body, the current log file to be written is selected from the sorted log files to be written; and the generation timestamp of the selected current log file to be written is the same as the first timestamp; and

[0028] Write the corresponding record to the log file to be written; and increment the first record count value w1 by 1 each time a record is written to the log file to be written.

[0029] Optionally, the step of obtaining the log file to be written based on the hostname of the target in each of the index files and the current write file information further includes:

[0030] When the first record count statistics w1 = m, the first timestamp in the current write file information is updated to the generation timestamp of the next log file to be written, and the first record count statistics w1 = 0 is updated;

[0031] Based on the updated first timestamp, a new log file to be written is selected from the sorted log files to be written; and the generation timestamp of the newly selected new log file to be written is the same as the updated first timestamp.

[0032] Optionally, the current synchronization information includes a second timestamp and a second record count; the second timestamp is the generation timestamp of the log file to be synchronized; the second record count is the number of records read from the log file to be synchronized, denoted as r1.

[0033] Optionally, the step of obtaining the log file to be synchronized based on the hostname of the target in each of the index files and the current synchronization information includes:

[0034] Read the index header of the index file to obtain the hostname of the target;

[0035] Read the index body of the index file to obtain the second timestamp in the current synchronization information;

[0036] Based on the hostname of the target in the index header, obtain several log files to be synchronized; and the hostname of the target in the log files to be synchronized is the same as the hostname of the target in the index header;

[0037] The log files to be synchronized are sorted in ascending order according to their generation timestamps;

[0038] Based on the second timestamp in the index body, the current log file to be synchronized is selected from the sorted log files to be synchronized; and the generation timestamp of the selected current log file to be synchronized is the same as the second timestamp; and

[0039] Read the written records from the log file currently to be synchronized; and increment the second record count value r1 by 1 each time a record is read from the log file currently to be synchronized.

[0040] Optionally, the step of obtaining the log file to be synchronized based on the hostname of the target in each of the index files and the current synchronization information includes:

[0041] When the second record count value r1 = m, the second timestamp in the current synchronization information is updated to the generation timestamp of the next log file to be synchronized, and the second record count value r1 = 0 is updated.

[0042] Based on the updated second timestamp, a new log file to be synchronized is selected from the sorted log files to be synchronized; and the generation timestamp of the newly selected new log file to be synchronized is the same as the updated second timestamp.

[0043] Optionally, before performing the step of reading the written records from the currently synchronized log file, the method further includes: determining whether the currently synchronized log file and the currently written log file are the same log file; if they are the same log file, then ensuring that the second record count value r1 is not greater than the first record count value w1.

[0044] Optionally, the index body further includes the current maximum synchronized record information, which includes a third timestamp and a third record count statistic; and the third timestamp is the generation timestamp of the log file to be synchronized; the third record count statistic is the number of records written to the log file to be synchronized, denoted as w2.

[0045] Optionally, before performing the step of reading the written records from the currently synchronized log file, the method further includes: comparing the second record count value r1 with the third record count value w2, and ensuring that the second record count value r1 is not greater than the third record count value w2.

[0046] Optionally, the sign that the instruction data is successfully synchronized to the backup application server is that the main application server receives the return data sent by the backup server within a second preset time period.

[0047] The sign that the instruction data is successfully synchronized to the database is that no exception to accessing the database is caught when the transaction is committed.

[0048] Optionally, the configuration information also includes: the number of retries r after data synchronization failure, where r ≥ 0, and the persistence attribute value; and the persistence attribute value is 0 or 1.

[0049] If the instruction data is not successfully synchronized to the corresponding target, the persistence attribute value is determined; if the persistence attribute value is 1, it indicates that synchronization persistence is enabled, and the steps of reading the instruction data already written in the log file to be synchronized and synchronizing the instruction data read from the log file to be synchronized to the corresponding target are repeated until the instruction data is successfully synchronized to the corresponding target.

[0050] If the persistence attribute value is 0, it indicates that synchronization persistence is off. The steps of reading the instruction data already written in the log file to be synchronized and synchronizing the instruction data read from the log file to be synchronized to the corresponding target are repeated. The number of synchronization failures is counted, and instruction data with more than r synchronization failures is no longer synchronized. The number of records currently read in the log file r0 is added to the number of records corresponding to this part of the data. The second record count statistics value r1 of the current synchronization information in the index file is also added to the number of records corresponding to this part of the data. Waiting for the next synchronization task trigger condition to be met.

[0051] Optionally, when the target is a backup application server, the hostname of the target is the hostname of the backup application server;

[0052] When the target is a database, the hostname of the target is the hostname of the main application server.

[0053] Optionally, the instruction data and the return-to-school data are transmitted between the main application server and the backup application server using a message queue.

[0054] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the data synchronization method of the main application server as described above.

[0055] Based on the same inventive concept, the present invention also provides an electronic device, characterized in that it includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the data synchronization method of the main application server as described above.

[0056] Compared with the prior art, the present invention has at least one of the following advantages:

[0057] The present invention provides a data synchronization method, storage medium, and electronic device for a primary application server. The data synchronization method includes: parsing instructions sent by an external device to obtain several types of instruction data, each type of instruction data to be synchronized to a target; generating several log files according to each target, each log file including the file's generation timestamp and the target's hostname; generating an index file based on all log files corresponding to the same target, each index file including the target's hostname, current write file information, and current synchronization information; obtaining the log file to be written based on the target's hostname and current write file information in the index file, and writing the corresponding instruction data into the log file to be written; obtaining the log file to be synchronized based on the target's hostname and current synchronization information in the index file, and reading the instruction data already written into the log file to be synchronized; and synchronizing the instruction data read from the log file to be synchronized to the corresponding target when the synchronization task triggering condition is met, thereby achieving synchronization of instruction data between the primary and backup application servers and between the primary application server and the database, thus ensuring data consistency between the primary and backup application servers and providing high reliability.

[0058] Compared to traditional rollback log modes, this invention does not directly modify the target's data during synchronization operations. Instead, it pre-records the data in a log file. When the synchronization task trigger condition is met, the records in the log file are synchronized to the target. Therefore, synchronization operations can occur without writing the original data. This allows data changes to be added to the log file while read operations are performed on the machine where the data has not been changed, achieving concurrent execution of reads and writes.

[0059] The use of index files in this invention makes the synchronization process more accurate and efficient, and the system has good reliability in the face of power failure, network failure and other reasons.

[0060] In this invention, both instruction data and return data are processed using message queues, which enables asynchronous execution of data reception and processing, improving data processing efficiency, and also supports data filtering functions.

[0061] In this invention, both parties synchronize using a response timeout mechanism, while also enabling persistent data operations. Attached Figure Description

[0062] Figure 1 This is a flowchart of a data synchronization method for a main application server provided in an embodiment of the present invention;

[0063] Figure 2 This is a logic diagram of a data synchronization method for a main application server provided in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram illustrating the writing of instruction data and return data into a message queue in a data synchronization method for a main application server provided by an embodiment of the present invention. Detailed Implementation

[0065] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the data synchronization method, storage medium, and electronic device for the main application server proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clearly illustrate the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of this invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0066] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0067] Combined with appendix Figures 1-3 As shown, this embodiment provides a data synchronization method for a primary application server, including: step S110, obtaining configuration information, wherein the configuration information includes synchronization task triggering conditions; step S120, receiving and parsing instructions sent by an external device to obtain several types of instruction data; wherein each type of instruction data is to be synchronized to a target, wherein the target is a backup application server or a database; step S130, generating several log files according to each target; each log file includes a log header, wherein the log header includes the generation timestamp of the corresponding log file and the hostname of the corresponding target; wherein the generation timestamps in any two log files are different; step S140, generating an index file according to all the log files corresponding to the same target; each The index file includes an index header and an index body. The index header includes the hostname of the corresponding target, and the index body includes current file writing information and current synchronization information. Step S150: Obtain the log file to be written according to the hostname of the target and the current file writing information in each index file, and write the corresponding instruction data into the log file to be written. Step S160: Obtain the log file to be synchronized according to the hostname of the target and the current synchronization information in each index file, and read the instruction data already written in the log file to be synchronized. Step S170: When the synchronization task triggering condition is met, synchronize the instruction data read from the log file to be synchronized to the corresponding target.

[0068] It is understood that when the target is a backup application server, the hostname of the target is the hostname of the backup application server; when the target is a database, the hostname of the target is the hostname of the primary application server.

[0069] It is understandable that each type of instruction data is divided into several records and written to the log file to be written.

[0070] Specifically, in this embodiment, the configuration information includes not only the synchronization task triggering conditions, but also: the maximum number of records that the log file can store, m and m > 0; the ability to synchronize the instruction data to a maximum of p backup application servers during the synchronization process, p > 0; the pipe number for communication between the primary application server and the backup application servers; the number of retries r after data synchronization failure, r ≥ 0; and the persistence attribute value; wherein the persistence attribute value is 0 or 1. Based on the configuration information, the synchronization strategy for the instruction data and the communication parameters between the primary application server and the backup application servers can be determined. More specifically, the synchronization task triggering conditions include a first preset condition and a second preset condition; the first preset condition is that the synchronization task is triggered when every n records are read from the log file to be synchronized within a first preset duration, 0 < n ≤ m; and the first preset duration is t seconds, t > 0; the second preset condition is that the synchronization task is triggered at t+i seconds when the number of records read from the log file to be synchronized within the first preset duration does not reach n, and i > 0. Optionally, n can be 1; 0.1≤i≤2; the value of t can be determined according to specific needs, but the present invention is not limited thereto.

[0071] Specifically, in this embodiment, in step S120, the instruction can be parsed according to the target, and the number of types of the instruction data obtained after parsing is the same as the number of targets. For example, if there are two targets, one of which is a backup server and the other is a database, the number of types of instruction data is two: one type is data to be synchronized to the backup application server, and the other type is data to be synchronized to the database, i.e., data to be written to the database. If there are three targets, and the three targets are a first backup server, a second backup server, and a database, the number of types of instruction data is three: the three types of instruction data are data to be synchronized to the first backup application server, data to be synchronized to the second backup application server, and data to be synchronized to the database, respectively. Optionally, the external device can be a client, and the instruction can be commands such as listing, removing, manual data setting, and manual position setting, but the present invention is not limited thereto.

[0072] Specifically, in this embodiment, in step S130, the log header further includes the type of instruction data stored in the corresponding log file, the maximum number of records that can be stored, the number of records currently written (i.e., the number of records w0 currently written to the corresponding log file), and the number of records currently read (i.e., the number of records r0 currently read from the corresponding log file). More specifically, the maximum number of records that the log file can store in the log header, i.e., the storage capacity, is the same as the value of parameter m in the configuration information. Each time a record is written to the log file, w0 is incremented by 1; when the number of records w0 written to the log file reaches m, subsequent records will be written to other log files corresponding to the same target. Each time a record is read from the log file, r0 is incremented by 1; when the number of records r0 read from the log file reaches m, subsequent records will be read from other log files corresponding to the same target. Optionally, the generation timestamp of the log file in the log header is accurate to milliseconds, but this invention is not limited thereto.

[0073] Furthermore, the log file includes not only the log header but also the log body; and the log bodies of several log files generated according to the same target are used to store the instruction data to be synchronized to the target, that is, the instruction data to be synchronized to the target is divided into records and written into the corresponding log bodies. Optionally, the format of the instruction data stored in the log body is not limited; it can be a custom message structure or byte information serialized from protobuf, stored in binary form in the log body.

[0074] Specifically, in this embodiment, in step S140, the number of index files is the same as the number of targets, and one index file is used to store index information of all log files corresponding to one target, such as the current write file information and the current synchronization information. More specifically, in the index body of each index file, the current write file information includes a first timestamp and a first record count statistic; the first timestamp is the generation timestamp of the log file to be written; the first record count statistic is the number of records already written in the log file to be written, denoted as w1. The current synchronization information includes a second timestamp and a second record count statistic; the second timestamp is the generation timestamp of the log file to be synchronized; the second record count statistic is the number of records read from the log file to be synchronized, denoted as r1. Furthermore, the index body of each index file also includes information on the current maximum number of records to be synchronized; this information includes a third timestamp and a third record count statistic; the third timestamp is the generation timestamp of the log file to be synchronized; the third record count statistic is the number of records written to the log file to be synchronized, denoted as w2. More specifically, the log file to be written can be denoted as f1, and the log file to be synchronized can be denoted as f2; f1 and f2 may be the same log file, or they may be two different log files corresponding to the same target. When f1 and f2 are the same log file, w2 = w1; when f1 and f2 are two different log files corresponding to the same target, it indicates that the log file to be synchronized is full of records, and w2 = m. Optionally, the index header of each index file also includes the generation timestamp of the index file; preferably, the generation timestamp of the index file is accurate to milliseconds, but this invention is not limited thereto.

[0075] Specifically, in this embodiment, the content of the index file is dynamically changing. Each time a record is written to the currently pending log file, the first record count statistic w1 is incremented by 1, and it is checked whether the current maximum synchronized record information needs to be updated. When w1 = m, the first timestamp in the current write file information is updated to the generation timestamp of another log file corresponding to the same target, and the first record count statistic w1 is updated to 0. When the second record count statistic r1 < the third record count statistic w2, each time a record is read from the currently pending log file and successfully synchronized to the corresponding target (backup server or database), the second record count statistic r1 is incremented by 1. When r1 = m, the second timestamp in the current synchronization information is updated to the generation timestamp of another log file corresponding to the same target, and the second record count statistic r1 is updated to 0. Preferably, when synchronizing records read from the currently synchronized log file to the corresponding target, the FIFO (First In First Out) principle is followed, that is, the records written to the log file first are synchronized first; the index file can determine the order of the log files to be synchronized based on the generation timestamp of the log files, and the log files with newer generation timestamps are synchronized later, but the present invention is not limited thereto.

[0076] Furthermore, the format of the index body in the index file is not limited. It can be a custom message structure or byte information serialized from protobuf and stored in the index file in binary form, but the present invention is not limited thereto.

[0077] Please continue to refer to this. Figure 1 and Figure 2Step S150 includes: Step S1501, reading the index header of the index file to obtain the hostname of the target; Step S1502, reading the index body of the index file to obtain the first timestamp in the current file writing information; Step S1503, obtaining a plurality of log files to be written according to the hostname of the target in the index header; and the hostname of the target in the log files to be written is the same as the hostname of the target in the index header; Step S1504, processing the plurality of log files to be written according to... The generated timestamps are sorted in ascending order; and the newer the generated timestamp, the later the log file to be written is sorted; step S1505: according to the first timestamp in the index body, select the current log file to be written from the sorted log files to be written; and the generated timestamp of the selected current log file to be written is the same as the first timestamp; and step S1506: write the corresponding record to the current log file to be written; and when each record is written to the current log file to be written, the first record count value w1 is incremented by 1.

[0078] It is understood that step S150 further includes: step S1507, when the first record count statistics w1 = m, updating the first timestamp in the current write file information to the generation timestamp of the next log file to be written, and updating the first record count statistics w1 = 0; step S1508, according to the updated first timestamp, reselecting the current log file to be written from the sorted log files to be written; and the generation timestamp of the reselected current log file to be written is the same as the updated first timestamp.

[0079] Specifically, in this embodiment, steps S1505 to S1508 can be executed repeatedly until all of the parsed instruction data of each type is written into the log file corresponding to the target to be synchronized for that type of instruction data, but the present invention is not limited thereto.

[0080] Please continue to refer to this. Figure 1 and Figure 2Step S160 includes: Step S1601, reading the index header of the index file to obtain the hostname of the target; Step S1602, reading the index body of the index file to obtain the second timestamp in the current synchronization information; Step S1603, obtaining several log files to be synchronized based on the hostname of the target in the index header; and the hostname of the target in the log files to be synchronized is the same as the hostname of the target in the index header; Step S1604, sorting the several log files to be synchronized according to their... The generated timestamps are sorted in ascending order; and the newer the generated timestamp, the later the log file to be synchronized is sorted; step S1605: according to the second timestamp in the index body, select the current log file to be synchronized from the sorted log files to be synchronized; and the generated timestamp of the selected current log file to be synchronized is the same as the second timestamp; and step S1606: read the written records from the current log file to be synchronized; and when each record is read from the current log file to be synchronized, the second record count value r1 is incremented by 1.

[0081] It is understood that step S160 further includes: step S1607, when the second record count statistics value r1 = m, updating the second timestamp in the current synchronization information to the generation timestamp of the next log file to be synchronized, and updating the second record count statistics value r1 = 0; step S1608, according to the updated second timestamp, reselecting the current log file to be synchronized from the sorted log files to be synchronized; and the generation timestamp of the reselected current log file to be synchronized is the same as the updated second timestamp.

[0082] Specifically, in this embodiment, steps S1605 to S1608 and step S170 can be executed repeatedly until all the instruction data written in the log file corresponding to each target is successfully synchronized to that target, but the present invention is not limited thereto.

[0083] Furthermore, in this embodiment, before performing the step of reading the written records from the currently synchronized log file, i.e., step S1606, the method further includes: determining whether the currently synchronized log file and the currently written log file are the same log file. If they are the same log file, the second record count statistic r1 is not greater than the first record count statistic w1. That is, the rate of reading records from the currently synchronized log file is carefully controlled to prevent the records to be read from not being written to the log file before they are written to disk, thereby preventing data synchronization failure. However, this invention is not limited thereto.

[0084] In some embodiments, if the index body of the index file includes the information of the current maximum synchronized record, then before performing the step of reading the written record from the currently synchronized log file (i.e., step S1606), the method further includes: comparing the second record count value r1 with the third record count value w2, and ensuring that the second record count value r1 is never greater than the third record count value w2. This means carefully controlling the rate at which records are read from the currently synchronized log file to prevent data synchronization failures because the records to be read have not yet been written to the log file before being written to disk. Therefore, if the index body includes the information of the current maximum synchronized record, it is no longer necessary to repeatedly determine whether the currently synchronized log file and the currently written log file are the same log file. It is only necessary to ensure that the second record count value r1 is not greater than the third record count value w2, which can effectively improve data synchronization efficiency. However, this invention is not limited to this.

[0085] Specifically, in this embodiment, when the number of records written to a log file reaches the maximum number of records m and all records (i.e., m records) in the log file are successfully synchronized to the backup application server or the database, the log file will be deleted. More specifically, the indication that the instruction data is successfully synchronized to the backup application server is that the primary application server receives the return data sent by the backup server within a second preset time period; otherwise, it fails. The indication that the instruction data is successfully synchronized to the database is that no database access exception is caught when the transaction is committed; otherwise, it fails. Optionally, the instruction data and the return data are transmitted between the primary application server and the backup application server in the form of a message queue. That is, the backup application server uses a message queue to store the instruction data synchronized by the primary application server, and the primary application server uses a message queue to store the return data sent by the backup application server, but the present invention is not limited thereto.

[0086] Furthermore, if the instruction data fails to synchronize to the corresponding target, i.e., synchronization fails, the value of the persistence attribute, i.e., the value of durable, is determined. If the value of durable is 1, it indicates that synchronization persistence is enabled, and steps S160 and S170 are repeated until the instruction data is successfully synchronized. If the value of durable is 0, it indicates that synchronization persistence is disabled, and steps S160 and S170 are repeated. The number of synchronization failures is counted. If the count of synchronization failures exceeds r, the data for that part will no longer be synchronized. The number of records currently read in the log file, r0, is added to the number of records corresponding to that part of the data. Similarly, the second record count count value r1 of the current synchronization information in the index file is added to the number of records corresponding to that part of the data. The system waits for the next synchronization task trigger condition to be met.

[0087] Specifically, in this embodiment, if a primary / backup application server switchover occurs, the data that the original primary application server had not yet completed synchronizing continues to be synchronized, denoted as process s1; the backup application server, which has been promoted to primary application server, will also perform data synchronization operations, denoted as process s2; the synchronization order of the two follows the FIFO principle, that is, operation s2 is performed after operation s1 is completed. Similarly, if the original primary application server has not yet completed writing data to the database, it continues to write data, denoted as process s10; the backup application server, which has been promoted to primary application server, will also perform database write operations, denoted as process s20; the synchronization order of the two follows the FIFO principle, that is, operation s20 is performed after operation s10 is completed.

[0088] Furthermore, in this embodiment, when the backup application server fails, the primary application server can normally synchronize the corresponding instruction data to the database, that is, it can normally synchronize the instruction data already written in the log file corresponding to the database to the database. However, it cannot synchronize the instruction data already written in the log file corresponding to the backup application server to the backup application server. Therefore, the instruction data already written in the log file corresponding to the backup application server will accumulate, and it will contain all unsynchronized records. When the backup application server recovers, the primary application server needs to synchronize the instruction data already written in the log file corresponding to the backup application server to the backup application server according to the FIFO principle. If the backup application server continues to fail, the instruction data already written in the log file will increase over time. As needed, an effective time period can be set to periodically clean up these log files, record abnormal situations, and check the backup application server. It should be noted that since the primary application server is normally synchronizing data to the database, when the backup application server restarts and recovers, it will load the data already synchronized to the database from the database. At this time, the backup application server will also receive the data synchronized from the primary application server. It needs to compare the newness of the two to decide whether to discard or modify them.

[0089] When a database fails, the primary application server can synchronize the corresponding instruction data to the backup application server normally. This means it can synchronize the instruction data already written to the backup server's log file. However, it cannot synchronize the instruction data already written to the database's log file. Therefore, the instruction data already written to the database's log file will accumulate, containing only unsynchronized records. Once the database recovers, the primary application server must synchronize the instruction data already written to the database's log file to the database according to the FIFO principle. If the database continues to fail, the amount of instruction data written to the log file will increase over time. As needed, a valid time period can be set to periodically clean up these log files, record abnormal situations, and check the database.

[0090] When both the backup application server and the database fail, the primary application server cannot properly synchronize the corresponding instruction data to the backup application server and the database. This results in an accumulation of the instruction data already written to the log files of both the backup server and the database. If the failure of both the backup application server and the database continues, the amount of instruction data already written to the log files will increase over time. As needed, a valid time period can be set to periodically clean up these files, record abnormal situations, and check the backup application server and the database.

[0091] In a primary-multiple-standby scenario, if one standby application server fails, the primary application server cannot synchronize the instruction data already written to the corresponding log file of the standby application server. Therefore, the instruction data in the log file of the standby application server will accumulate, containing only unsynchronized records. When the standby application server recovers, the primary application server must synchronize the instruction data already written to the log file of the standby application server according to the FIFO principle. If the standby application server remains unresponsive, the amount of instruction data written to the log file will increase over time. A valid time period can be set to periodically clean up these log files, record anomalies, and check the standby application server. It is important to note that since the primary application server synchronizes data to the database normally, when the standby application server restarts and recovers, it will load the data already synchronized to the database. At this time, the standby application server will also receive data synchronized from the primary application server, and it needs to compare the newness of the two to decide whether to discard or modify the data. Furthermore, the synchronization process from the primary application server to other standby application servers continues normally and is unaffected.

[0092] Furthermore, in this embodiment, if an interruption occurs during index file updates, the main application server will reload all log files corresponding to the same target at once during the next index file read. The log file with the latest generation timestamp among all log files corresponding to the target will be selected as the current log file to be written, and the current write file information in the index file will be updated. The first timestamp will be the generation timestamp (timestamp1) of the current log file to be written, and the first record count (w1) will be equal to the number of records (w0) already written in the current log file to be written. The log file with the oldest generation timestamp among all log files corresponding to the target will be selected as the current log file to be synchronized. The current synchronization information in the index file will be updated, and the second timestamp will be the generation timestamp (timestamp2) of the current log file to be synchronized. The second record count (r1) will be equal to the number of records (r0) already read from the current log file to be synchronized. If timestamp1 ≠ timestamp2, it indicates that the current log file to be written and the current log file to be synchronized are two different log files. In this case, the information of the currently synchronized maximum record in the index file will be updated, and the third timestamp will be timestamp1, and the third record count (w2) will be equal to m. If timestamp1 = timestamp2, it means that the log file to be written and the log file to be synchronized are the same log file. In this case, the information of the currently synchronized maximum record in the updated index file is the same as the information of the currently written file, that is, the third timestamp is timestamp2, and the third record count value w2 equals w1. If the log file is interrupted during the synchronization of the instruction data to the corresponding target, it will not affect the writing and reading of the instruction data, because the index file will still point to the data in the log file, and the entire process will continue as usual. When the next synchronization task trigger condition is met, synchronization will start from the position of the error. If the writing of the instruction data to the log file is interrupted, then the index file is not updated at this time, and the instruction data will be rewritten.

[0093] Specifically, in this embodiment, the instructions sent by the external device to the main application server, the instruction data synchronized by the main application server to the target, and the return data sent by the target to the main application server typically consist of two parts: a data header and a data body. The data header information includes source address information (src), destination address information (trg), pipe number (leaguer), data type (type), agent number (agent), timestamp (timestamp), data body length (len), and data read flag (newflag). The data body information is the data information to be synchronized, and the data format is not limited; it can be a custom message structure or byte information serialized from protobuf. More specifically, the source address information (src) in the header includes the hostname, username, and process ID of the data sender; the destination address information (trg) in the header includes the hostname, username, and process ID of the data receiver; the pipe number (leaguer) is the pipe number used for communication between the primary and backup application servers; the data type (type) is used to distinguish data categories, whether the data is synchronized data, return data, or instructions sent to the primary application server by an external device; the agent number (agent) is the number assigned to the data when the primary application server synchronizes data with the backup application server; the timestamp (timestamp) is the instantaneous moment when the primary application server synchronizes data with the backup application server, accurate to the second; the data read flag (newflag) indicates whether the synchronized data has been read after being written to the shared queue, newflag=0 indicates that the data has been read, and newflag=1 indicates that the data has not been read. Optionally, the pipe number for communication between the primary and backup application servers is assigned to different application servers, with each application server configured with a unique number, and the number is only activated when an application server is designated as the primary application server.

[0094] Specifically, in this embodiment, when the backup application server uses a message queue to store the instruction data synchronized by the main application server, or when the main application server uses a message queue to store the return data sent by the backup application server, the message queue has a filtering function. The synchronization data sender fills in the parameter information described in the data header and sends the synchronized data to the designated receiver. The synchronization data receiver can read matching messages from the message queue according to the configured parameter 'leaguer' and the hostname of the local machine for processing, and send return data. The 'src' parameter in the return data corresponds to the 'trg' parameter of the sender, and the 'trg' parameter in the return data corresponds to the 'src' parameter of the sender. The 'leaguer', 'type', and 'agent' parameters in the return data are all the same as those of the sender. The return data receiver can read matching return data from the message queue according to the configured parameter 'leaguer', the hostname of the local machine, and the agent number (agent) for processing. More specifically, as... Figure 3 As shown, the primary and backup application servers communicate using TCP. After receiving synchronization data (i.e., instruction data), the backup application server writes it to a message queue. Similarly, the primary application server writes return data received from the backup application server to its message queue. In a one-primary-multiple-backup scenario, the primary application server uses configured parameters such as the `leaguer` pipe number and `agent` agent to send instruction data to the backup application servers that need synchronization. Simultaneously, the backup application servers use the `leaguer` pipe number and hostname to filter out the instruction data that needs processing in their message queues. The backup application servers also use the `leaguer` pipe number, `agent` agent, and destination address information `trg` to send return data to the primary application server. Simultaneously, the primary application server uses the configured `leaguer` pipe number, hostname, and `agent` agent to filter the return data corresponding to instruction data destined for a specific backup application server.

[0095] Based on the same inventive concept, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data synchronization method of the main application server as described above.

[0096] Based on the same inventive concept, this embodiment also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the data synchronization method of the main application server as described above.

[0097] In summary, the data synchronization method, storage medium, and electronic device for the main application server provided in this embodiment, compared with the traditional rollback log mode, do not directly modify the target data during synchronization operations. Instead, they are pre-recorded in a log file, and when the synchronization task triggering conditions are met, the records in the log file are synchronized to the target. Therefore, transaction commits can occur without writing the original data, allowing data changes to be added to the log file while read operations are performed on the machine where the data has not been modified, achieving concurrent execution of read and write operations. The use of an index file in this embodiment makes the synchronization process more accurate and efficient, and the system has a certain degree of reliability in the face of power outages, network failures, etc. Instruction data and return data are both processed in the form of message queues, enabling asynchronous execution of data reception and processing, improving data processing efficiency, and supporting data filtering functions. The synchronization parties adopt an acknowledgment timeout mechanism to achieve persistent data operations.

[0098] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A data synchronization method for a main application server, characterized in that, include: Obtain configuration information, including the conditions for triggering the synchronization task; It receives and parses instructions sent by external devices to obtain several types of instruction data; Furthermore, each type of instruction data needs to be synchronized to a target, which is a backup application server or a database; Several log files are generated according to each target; each log file includes a log header, which includes the generation timestamp of the corresponding log file and the hostname of the corresponding target; An index file is generated based on all the log files corresponding to the same target; each index file includes an index header and an index body, the index header includes the hostname of the corresponding target, and the index body includes current file writing information and current synchronization information; The log file to be written is obtained based on the hostname of the target in each index file and the current write file information, so as to write the corresponding instruction data into the log file to be written. The log file to be synchronized is obtained based on the hostname of the target in each index file and the current synchronization information, so as to read the instruction data already written in the log file to be synchronized; as well as When the synchronization task triggering conditions are met, the instruction data read from the log file to be synchronized will be synchronized to the corresponding target.

2. The data synchronization method for the main application server as described in claim 1, characterized in that, Each type of instruction data is divided into several records and written to the log file to be written.

3. The data synchronization method for the main application server as described in claim 2, characterized in that, The configuration information also includes: the maximum number of records m that the log file can store, where m > 0; The synchronous task triggering conditions include a first preset condition and a second preset condition; The first preset condition is that a synchronization task is triggered when every n records are read from the log file to be synchronized within a first preset time period, where 0 < n ≤ m; and the first preset time period is t seconds, where t > 0; The second preset condition is that if the number of records read from the log file to be synchronized within the first preset time period does not reach n, the synchronization task is triggered at t+i seconds, and i>0.

4. The data synchronization method for the main application server as described in claim 3, characterized in that, The log header also includes the type of instruction data stored in the corresponding log file, the maximum number of records that can be stored, the number of records currently written w0, and the number of records currently read r0.

5. The data synchronization method for the main application server as described in claim 4, characterized in that, The current file writing information includes a first timestamp and a first record count statistic; the first timestamp is the generation timestamp of the log file to be written; the first record count statistic is the number of records already written to the log file to be written, denoted as w1.

6. The data synchronization method for the main application server as described in claim 5, characterized in that, The steps for obtaining the log file to be written based on the hostname of the target in each of the index files and the current write file information include: Read the index header of the index file to obtain the hostname of the target; Read the index body of the index file to obtain the first timestamp in the current write file information; Based on the hostname of the target in the index header, obtain several log files to be written; and the hostname of the target in the log files to be written is the same as the hostname of the target in the index header; The log files to be written are sorted in ascending order according to their generation timestamps; Based on the first timestamp in the index body, the current log file to be written is selected from the sorted list of log files to be written; and the generation timestamp of the selected current log file to be written is the same as the first timestamp; and Write the corresponding record to the log file to be written; and increment the first record count value w1 by 1 each time a record is written to the log file to be written.

7. The data synchronization method for the main application server as described in claim 6, characterized in that, The step of obtaining the log file to be written based on the hostname of the target in each of the index files and the current write file information further includes: When the first record count statistics w1 = m, the first timestamp in the current write file information is updated to the generation timestamp of the next log file to be written, and the first record count statistics w1 = 0 is updated. Based on the updated first timestamp, a new log file to be written is selected from the sorted log files to be written; and the generation timestamp of the newly selected new log file to be written is the same as the updated first timestamp.

8. The data synchronization method for the main application server as described in claim 7, characterized in that, The current synchronization information includes a second timestamp and a second record count; the second timestamp is the generation timestamp of the log file to be synchronized; the second record count is the number of records read from the log file to be synchronized, denoted as r1.

9. The data synchronization method for the main application server as described in claim 8, characterized in that, The steps for obtaining the log file to be synchronized based on the hostname of the target in each index file and the current synchronization information include: Read the index header of the index file to obtain the hostname of the target; Read the index body of the index file to obtain the second timestamp in the current synchronization information; Based on the hostname of the target in the index header, obtain several log files to be synchronized; and the hostname of the target in the log files to be synchronized is the same as the hostname of the target in the index header; The log files to be synchronized are sorted in ascending order according to their generation timestamps; Based on the second timestamp in the index body, the current log file to be synchronized is selected from the sorted log files to be synchronized; and the generation timestamp of the selected current log file to be synchronized is the same as the second timestamp; and Read the written records from the log file currently to be synchronized; and increment the second record count value r1 by 1 each time a record is read from the log file currently to be synchronized.

10. The data synchronization method for the main application server as described in claim 9, characterized in that, The steps for obtaining the log file to be synchronized based on the hostname of the target in each index file and the current synchronization information include: When the second record count value r1 = m, the second timestamp in the current synchronization information is updated to the generation timestamp of the next log file to be synchronized, and the second record count value r1 = 0 is updated. Based on the updated second timestamp, a new log file to be synchronized is selected from the sorted log files to be synchronized; and the generation timestamp of the newly selected new log file to be synchronized is the same as the updated second timestamp.

11. The data synchronization method for the main application server as described in claim 9, characterized in that, Before performing the step of reading the written records from the currently synchronized log file, the method further includes: determining whether the currently synchronized log file and the currently written log file are the same log file; if they are the same log file, then the second record count value r1 is not greater than the first record count value w1.

12. The data synchronization method for the main application server as described in claim 9, characterized in that, The index body also includes information on the current maximum number of records synchronized, which includes a third timestamp and a third record count; and the third timestamp is the generation timestamp of the log file to be synchronized. The third record count is the number of records already written to the log file currently to be synchronized, denoted as w2.

13. The data synchronization method for the main application server as described in claim 12, characterized in that, Before performing the step of reading the written records from the currently synchronized log file, the method further includes: comparing the second record count value r1 with the third record count value w2, and ensuring that the second record count value r1 is not greater than the third record count value w2.

14. The data synchronization method for the main application server as described in claim 1, characterized in that, The sign that the instruction data is successfully synchronized to the backup application server is that the main application server receives the return data sent by the backup server within a second preset time period. The sign that the instruction data has been successfully synchronized to the database is that no database access exception has been caught.

15. The data synchronization method for the main application server as described in claim 1, characterized in that, The configuration information also includes: the number of retries r after data synchronization failure, where r ≥ 0, and the persistence attribute value; where the persistence attribute value is 0 or 1. If the instruction data is not successfully synchronized to the corresponding target, the persistence attribute value is determined; if the persistence attribute value is 1, the steps of reading the instruction data already written in the log file to be synchronized and synchronizing the instruction data read from the log file to be synchronized to the corresponding target are repeated until the instruction data is successfully synchronized to the corresponding target. If the persistence attribute value is 0, repeat the steps of reading the instruction data already written in the log file to be synchronized and synchronizing the instruction data read from the log file to be synchronized to the corresponding target, and count the number of synchronization failures, and no longer synchronize instruction data with more than r synchronization failures.

16. The data synchronization method for the main application server as described in claim 1, characterized in that, When the target is a backup application server, the hostname of the target is the hostname of the backup application server; When the target is a database, the hostname of the target is the hostname of the main application server.

17. The data synchronization method for the main application server as described in claim 14, characterized in that, The instruction data and the return-to-school data are transmitted between the main application server and the backup application server using a message queue.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data synchronization method of the main application server as described in any one of claims 1 to 17.

19. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the data synchronization method of the main application server as described in any one of claims 1 to 17.

Images