Data replication method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a data replication method and device, electronic equipment and storage medium, which can be applied to the fields of big data technology, finance or other fields. The method comprises: in response to triggering a data replication event, creating a data playback process; using a detection tool and a parameter interaction table of a host device to reach a consensus on data replication with the host device, and obtaining a consensus result; in a case where it is determined that the consensus result indicates that a consensus is reached, loading a data playback script into the data playback process; running the data playback process to pull a target message from a message queue, and playing back target data obtained by analyzing the target message in a database of a platform device; the host device is configured to load a log analysis script into a log analysis process, and run the log analysis process to analyze a database log of the host device to obtain target data, and send a target message encapsulated by the target data to the message queue.

Description

Technical Field

[0001] This disclosure relates to the fields of big data technology and finance, and more specifically, to a data replication method, apparatus, electronic device, and storage medium. Background Technology

[0002] To adapt to the needs of business innovation and development, meet the requirements of continuous business growth, business continuity and processing performance, and alleviate the pressure on host resources caused by business growth, the relevant technologies have built an application architecture that combines the host and the open platform. Under this application architecture, data replication from the host to the platform is inevitable.

[0003] In related technologies, data replication from host to platform is generally done manually by technicians. However, with the rapid increase in the number of platform servers, the workload of manual operation by technicians has also increased, making it impossible to guarantee the accuracy of data replication. Summary of the Invention

[0004] In view of the above, this disclosure provides a data copying method, apparatus, electronic device, readable storage medium, and computer program product.

[0005] One aspect of this disclosure provides a data replication method, comprising: in response to triggering a data replication event, creating a data playback process; using a parameter interaction table between a detection tool and a host device, reaching a data replication consensus with the host device to obtain a consensus result; if the consensus result indicates that a consensus has been reached, loading a data playback script into the data playback process; and running the data playback process to pull a target message from a message queue and replay the target data obtained by parsing the target message in the database of the platform device; wherein the host device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host device to obtain the target data and send the target message encapsulated from the target data to the message queue.

[0006] According to embodiments of this disclosure, the above-mentioned creation of a data playback process in response to triggering a data replication event includes: stopping the current data playback process in response to triggering the data replication event; importing a table replication list of data to be replicated from the host device; and creating the data playback process based on the table replication list.

[0007] According to embodiments of this disclosure, running the data replay process to retrieve target messages from the message queue and replay the target data obtained by parsing the target messages in the database of the platform device includes: running the data replay process to sequentially retrieve messages from the message queue; filtering the messages using the table copy list to obtain filtering results; parsing the target messages to obtain the target data when the filtering results indicate that the message is the target message; and replaying the target data in the database of the platform device using the data replay script.

[0008] According to embodiments of this disclosure, the method further includes: if the filtering result indicates that the message is not the target message, returning the message to the message queue.

[0009] According to an embodiment of this disclosure, the method further includes: determining the time information of the data to be copied from the task log; and sending the time information to the host device; wherein the host device is configured to update the parameter value of the first field in the parameter interaction table based on the time information.

[0010] According to an embodiment of this disclosure, the host device is configured to: load the log parsing script into the log parsing process; run the log parsing process to parse the database logs of the host device based on the time information to obtain the data to be copied; and encapsulate the data to be copied into multiple messages and send the multiple messages sequentially to the message queue.

[0011] According to embodiments of this disclosure, the above-mentioned use of a detection tool and a parameter interaction table of a host device to achieve data replication consensus with the host device and obtain a consensus result includes: when it is determined that the data playback process has been successfully created, updating the parameter value of the second field of the parameter interaction table to a first preset value, wherein the host device is configured to create the log parsing process and update the parameter value of the third field of the parameter interaction table to the second preset value when the parameter value of the second field of the parameter interaction table is found to be the first preset value; when the detection tool detects that the parameter value of the third field of the parameter interaction table is... If the value is the second preset value, the parameter value of the fourth field of the parameter interaction table is updated to the third preset value. The host device is configured to configure the log parsing process based on the parameter value of the first field of the parameter interaction table when the parameter value of the fourth field of the parameter interaction table is found to be the third preset value, and to update the parameter value of the fifth field of the parameter interaction table to the fourth preset value. And when the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, a consensus result is obtained that the data replication consensus is reached with the host device.

[0012] According to an embodiment of this disclosure, the method further includes: updating the parameter value of the sixth field of the parameter interaction table to a fifth preset value when the data playback process is running; wherein the host device is configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table is found to be the fifth preset value.

[0013] According to an embodiment of this disclosure, the data replication event includes the parameter value of the seventh field of the parameter interaction table being a sixth preset value. The parameter value of the seventh field of the parameter interaction table is obtained by detecting the parameter interaction table using the detection tool. The host device is configured to stop the current log parsing process and update the parameter value of the seventh field to the sixth preset value in response to the triggered table access event.

[0014] According to embodiments of this disclosure, the platform-side device includes a platform server; the method further includes: in response to a server online event, determining a target platform server to be accessed; sending the detection tool and the data playback script to the target platform server through a server management system; and sending server online information to the host-side device, wherein the host-side device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

[0015] Another aspect of this disclosure provides a data replication apparatus, comprising: a creation module for creating a data playback process in response to a triggered data replication event; a consensus module for reaching a data replication consensus with the host device using a parameter interaction table between a detection tool and the host device, and obtaining a consensus result; a loading module for loading a data playback script into the data playback process when the consensus result indicates that a consensus has been reached; and a running module for running the data playback process to retrieve a target message from a message queue and replay the target data obtained by parsing the target message in the database of the platform device; wherein the host device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host device, obtain the target data, and send the target message encapsulated from the target data to the message queue.

[0016] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0017] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0018] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0019] According to embodiments of this disclosure, when performing data replication from a host device to a platform device, the host device and the platform device can reach a consensus using a parameter exchange table. After reaching a consensus, the platform device can run a data playback process, and the host device can run a log parsing process, using a message queue to complete the data replication from the host device to the platform device. Through the above technical means, consensus based on a parameter exchange table can replace manual operations by technicians, thus at least partially overcoming the technical problem of low accuracy in data replication caused by significant manual operations in related technologies, thereby effectively improving the efficiency and accuracy of data replication. Attached Figure Description

[0020] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0021] Figure 1An exemplary system architecture for which data copying methods and apparatus can be applied according to embodiments of this disclosure is illustrated.

[0022] Figure 2 A flowchart illustrating a data copying method according to an embodiment of the present disclosure is shown schematically.

[0023] Figure 3 A flowchart illustrating a data replication consensus method according to an embodiment of this disclosure is shown schematically.

[0024] Figure 4 A timing diagram of a data copying method according to an embodiment of the present disclosure is illustrated schematically.

[0025] Figure 5 A block diagram of a data copying apparatus according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 6 A block diagram of an electronic device suitable for implementing a data copying method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0031] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0032] To adapt to the needs of business innovation and development, meet the requirements of continuous business growth, business continuity, and processing performance, and alleviate the pressure on host resources caused by business growth, an application architecture combining the host and an open platform has been constructed in related technologies. Under this application architecture, data replication from the host to the platform is inevitable. With more and more application data being migrated to the platform, the platform servers are growing rapidly, resulting in a huge workload for technical personnel in manual operations, making it impossible to guarantee the accuracy of data replication.

[0033] In view of this, embodiments of the present disclosure provide a data replication method, apparatus, electronic device, storage medium, and computer program product to achieve interactive automatic production between a host and a platform. Specifically, the data replication method is applied to a platform-side device, and the method includes: creating a data playback process in response to triggering a data replication event; using a detection tool and a parameter interaction table of the host-side device to achieve data replication consensus with the host-side device and obtain a consensus result; if it is determined that the consensus result indicates that a consensus has been reached, loading a data playback script into the data playback process; and running the data playback process to pull a target message from a message queue and replay the target data obtained by parsing the target message in the database of the platform-side device; wherein, the host-side device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host-side device to obtain the target data and send a target message encapsulated from the target data to a message queue.

[0034] It should be noted that the data replication method and apparatus determined in the embodiments of this disclosure can be used in the field of big data technology or the financial field, or in any field other than the field of big data technology and the financial field. The application field of the data replication method and apparatus determined in the embodiments of this disclosure is not limited.

[0035] Figure 1 This illustration schematically depicts an exemplary system architecture to which data replication methods and apparatus can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a host device 101, a network 102, and platform devices 103, 104, and 105.

[0037] The host device 101 can be a server cluster consisting of various servers. These servers can include physical servers located locally or cloud servers located in the cloud, and are not limited here.

[0038] Network 102 serves as a medium for providing a communication link between host device 101 and platform devices 103, 104, and 105. Network 102 may include various connection types, such as wired and / or wireless communication links.

[0039] Platform devices 103, 104, and 105 can include various electronic devices, such as smartphones, tablets, laptops, and desktop computers. Alternatively, platform devices 103, 104, and 105 can also be various servers.

[0040] It should be noted that the data copying method provided in this embodiment can generally be executed by platform devices 103, 104, and 105. Accordingly, the data copying apparatus provided in this embodiment can generally be disposed in platform devices 103, 104, and 105.

[0041] It should be understood that Figure 1 The number of host-side devices, network-side devices, and platform-side devices shown is merely illustrative. Depending on implementation needs, any number of host-side devices, network-side devices, and platform-side devices can be included.

[0042] Figure 2 A flowchart illustrating a data copying method according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2As shown, the method includes operations S201 to S204.

[0044] In operation S201, in response to the triggering of a data copy event, a data playback process is created.

[0045] In operation S202, the parameter interaction table between the detection tool and the host device is used to perform data replication consensus with the host device to obtain a consensus result.

[0046] In operation S203, if the consensus result indicates that a consensus has been reached, the data replay script is loaded into the data replay process.

[0047] In operation S204, the data playback process is run to retrieve the target message from the message queue and replay the target data obtained by parsing the target message in the database of the platform device.

[0048] According to the embodiments of this disclosure, the methods of operating S201 to S204 can be executed by a platform-side device. The platform-side device can refer to any platform server. The platform server can be a physical server set up in a computer room or a cloud server leased or purchased in the cloud. No limitation is made here.

[0049] According to embodiments of this disclosure, the data replication event can be a periodically triggered event, meaning it is triggered at regular intervals to replicate data from the host device to the platform device. Alternatively, the data replication event can be generated and sent to the platform device by the host device when certain conditions are met. These conditions could be, for example, that the amount of incremental data on the host device reaches a threshold, or that a field in the parameter interaction table has a specific value. Furthermore, the data replication event can also be generated by the platform device when certain conditions are met. These conditions could be, for example, that the platform device is in an idle state, or that an application running on the platform device reports an error due to a lack of necessary data.

[0050] According to embodiments of this disclosure, the data playback process can be an instance of a data playback program in a platform-side device. Accordingly, creating a data playback process can be a cold start of the data playback program, which may include processes such as loading, starting, and rendering views of the data playback program.

[0051] According to embodiments of this disclosure, the detection tool can be an application program installed on the platform-side device. This detection tool can perform cyclic detection of specific fields in the parameter interaction table of the host-side device to periodically obtain the parameter values ​​of those specific fields. This detection tool can be compiled from various programming languages, such as C++, JAVA, etc., and is not limited thereto.

[0052] According to embodiments of this disclosure, the parameter interaction table may be a data table stored in the host device and associated with the platform device. The parameter interaction table may include multiple fields, each of which may have a parameter value.

[0053] According to embodiments of this disclosure, data replication consensus can be a process of confirming data replication preparation work between a platform device and a host device based on parameter values ​​of fields in a parameter interaction table. For example, this data replication consensus process may include the platform device and the host device respectively writing data to the parameter interaction table to indicate that the platform device and the host device are ready.

[0054] According to embodiments of this disclosure, the data playback script can be a script file containing message acquisition, message parsing, and data playback logic. After the data playback script is loaded into the data playback process, the data playback process can execute the script logic included in the data playback script during runtime.

[0055] According to embodiments of this disclosure, the message queue can be provided by a message middleware, which can be set on a host device, a platform device, or other devices capable of communicating with both the host device and the platform device, without limitation.

[0056] According to embodiments of this disclosure, replaying target data in the database of a platform device can be achieved by writing the target data into the database of the platform device in its original order.

[0057] According to embodiments of this disclosure, after the platform-side device runs the data playback process, the host-side device can be configured to load the log parsing script into the log parsing process and run the log parsing process in order to parse the database logs of the host-side device, obtain the target data, and send the target message encapsulated from the target data to the message queue.

[0058] According to embodiments of this disclosure, when performing data replication from a host device to a platform device, the host device and the platform device can reach a consensus using a parameter exchange table. After reaching a consensus, the platform device can run a data playback process, and the host device can run a log parsing process, using a message queue to complete the data replication from the host device to the platform device. Through the above technical means, consensus based on a parameter exchange table can replace manual operations by technicians, thus at least partially overcoming the technical problem of low accuracy in data replication caused by significant manual operations in related technologies, thereby effectively improving the efficiency and accuracy of data replication.

[0059] The following is for reference. Figures 3-4 In conjunction with specific embodiments, Figure 2The method shown will be further explained.

[0060] According to embodiments of this disclosure, a data replication event may include a parameter value of a sixth preset value in the seventh field of a parameter interaction table. That is, the host device or platform device can generate this data replication event when it detects that the parameter value of the seventh field of the parameter interaction table is the sixth preset value.

[0061] According to embodiments of this disclosure, the parameter value of the seventh field of the parameter interaction table can be actively modified by the host device. For example, the host device can be configured to stop the current log parsing process and update the parameter value of the seventh field to the sixth preset value in response to a triggered table access event. The table access event could be the creation of a new data table in the host device's database or the import of a new data table from an external data source. Stopping the current log parsing process avoids interference from existing processes on newly added data replication tasks, thus ensuring data consistency between the host device and the platform device. The sixth preset value can be set according to specific application scenarios, such as 1, Y, etc., and is not limited here.

[0062] It should be noted that the descriptions used here, such as "first field", "seventh field", "first preset value", "sixth preset value", etc., are only used to distinguish different "fields" or "preset values", and do not limit the order of "fields" or "preset values".

[0063] According to embodiments of this disclosure, operation S201 may specifically include the following operations:

[0064] In response to a triggered data replication event, the current data playback process is stopped; a table replication list of the data to be replicated is imported from the host device; and a data playback process is created based on the table replication list.

[0065] According to embodiments of this disclosure, stopping the current data playback process can be achieved by the platform device using tools such as a task manager to close the current data playback process. By stopping the current data playback process, interference from existing processes on newly added data replication tasks can be avoided, thereby ensuring data consistency between the host device and the platform device.

[0066] According to embodiments of this disclosure, the table replication list of data to be replicated may include multiple data table identifiers, and the data table corresponding to each data table identifier may contain incremental data, which may refer to the new data added by the host device between the last data replication task and the current data replication task.

[0067] According to embodiments of this disclosure, creating a data playback process based on a table copy list can involve cold-starting the data playback process and importing the table copy list into the data playback process.

[0068] According to embodiments of this disclosure, data replication consensus can be achieved using a parameter interaction table of the host device.

[0069] Figure 3 A flowchart illustrating a data replication consensus method according to an embodiment of this disclosure is shown schematically.

[0070] like Figure 3 As shown, the method includes operations S301 to S305, wherein operations S301, S303 and S305 can be executed by the platform-side device, and operations S302 and S304 can be executed by the host-side device.

[0071] In operation S301, if it is confirmed that the data playback process has been successfully created, the parameter value of the second field of the parameter interaction table is updated to the first preset value.

[0072] In operation S302, if the parameter value of the second field of the parameter interaction table is found to be the first preset value, a log parsing process is created, and the parameter value of the third field of the parameter interaction table is updated to the second preset value.

[0073] In operation S303, if the detection tool detects that the parameter value of the third field of the parameter interaction table is the second preset value, the parameter value of the fourth field of the parameter interaction table is updated to the third preset value.

[0074] In operation S304, if the parameter value of the fourth field of the parameter interaction table is found to be the third preset value, the log parsing process is configured based on the parameter value of the first field of the parameter interaction table, and the parameter value of the fifth field of the parameter interaction table is updated to the fourth preset value.

[0075] When operating S305, if the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, a consensus result is obtained that a data replication consensus is reached with the host device.

[0076] According to embodiments of this disclosure, the first preset value, the second preset value, the third preset value, and the fourth preset value can be set according to specific application scenarios, such as 1, Y, etc., without limitation.

[0077] According to embodiments of this disclosure, creating a log parsing process may involve the host device performing a cold start on the log parsing process.

[0078] According to embodiments of this disclosure, updating the parameter value of the second field of the parameter interaction table to a first preset value by the platform device can inform the host device that the cold start of the data playback process has been completed. Correspondingly, updating the parameter value of the third field of the parameter interaction table to a second preset value by the host device can inform the platform device that the cold start of the log parsing process has been completed. Subsequently, the platform device and the host device can sequentially complete a handshake operation by modifying the parameter values ​​of the fourth and fifth fields to provide information on the preparatory work before data replication. This consensus mechanism ensures the automatic execution of the data replication task and improves the accuracy of data replication.

[0079] According to embodiments of this disclosure, if the host device finds that the parameter value of the second field is not the first preset value, or the parameter value of the fourth field is not the third preset value, the host device can continue querying the second or fourth field after a certain interval. This interval can be set according to the specific application scenario, for example, it can be set to 1 second. Correspondingly, if the platform device detects that the parameter value of the third field is not the second preset value, or the parameter value of the fifth field is not the fourth preset value, the platform device can use a detection tool to perform cyclic detection.

[0080] According to embodiments of this disclosure, operation S204 may include the following operations:

[0081] Run the data replay process to retrieve messages sequentially from the message queue; filter the messages using a table copy list to obtain the filtering results; if the filtering results indicate that the message is the target message, parse the target message to obtain the target data; and use the data replay script to replay the target data in the database of the platform device.

[0082] According to embodiments of this disclosure, the message queue may not contain any messages before the log parsing process of the host device runs, i.e., the data replay process fails to retrieve messages. In this case, the data replay process can periodically retry message retrieval.

[0083] According to embodiments of this disclosure, in a single data replication task, the host device can use a single message queue to transmit data to multiple platform devices, and the data required by the multiple platform devices can be different. Each platform device's table replication list can record the data required by that platform device. By using the table replication list to filter messages, the accuracy of data replication can be ensured.

[0084] According to embodiments of this disclosure, if the filtering result indicates that the message is not the target message, the message can be returned to the message queue so that other platform devices can obtain the message and filter it.

[0085] According to embodiments of this disclosure, when data copying is performed using methods such as S201 to S204, the platform device can determine the time information of the data to be copied from the task log and send the time information to the host device. The host device can be configured to update the parameter value of the first field in the parameter interaction table based on the time information.

[0086] According to embodiments of this disclosure, data in the database of the host device can be stored in the form of records, and each record can be configured with the time information of its creation. After completing the data replication task, the platform device can write the time information of the most recent record being replayed into the task log. This time information can be the time information of the data to be replicated in the next data replication.

[0087] According to embodiments of this disclosure, the host device can start the log parsing process after the platform device has completed the data playback process. Specifically, while the data playback process is running, the platform device can update the parameter value of the sixth field of the parameter interaction table to the fifth preset value. The host device can be configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table is found to be the fifth preset value.

[0088] According to embodiments of this disclosure, the fifth preset value can be set according to specific application scenarios, such as 1, Y, etc., without limitation.

[0089] According to embodiments of this disclosure, during the process of loading the log parsing script into the log parsing process and running the log parsing process, the host device can be specifically configured to perform the following operations:

[0090] Load the log parsing script into the log parsing process; run the log parsing process to parse the database logs of the host device based on time information to obtain the data to be replicated; and encapsulate the data to be replicated into multiple messages and send the multiple messages to the message queue in sequence.

[0091] In the fundamentally disclosed embodiment, the database log of the host device is parsed based on time information to obtain the data to be replicated. This can be done by determining all data records in the database of the host device that are located after the time information from the database log, and then pulling all the data records from the host device to obtain the data to be replicated.

[0092] According to embodiments of this disclosure, the data to be copied may include data corresponding to different data tables. Encapsulating the data to be copied into multiple messages can be achieved by encapsulating the data corresponding to each data table in the data to be copied into one or more messages according to the data volume, that is, all data contained in a single message belongs to the same data table. Alternatively, when encapsulating the data to be copied into multiple messages, the data to be copied can also be classified into sub-data related to each platform device according to the table copy list of each platform device, and then each sub-data can be encapsulated into one or more messages according to the data volume.

[0093] According to embodiments of this disclosure, the platform-side device may include a platform server. After the platform server connects to the platform-side device, it can trigger a server online event. The platform-side device and the host-side device can configure the newly connected platform server to complete the data replication task. Specifically, in response to triggering the server online event, the platform-side device can determine the target platform server to be connected; send detection tools and data playback scripts to the target platform server through the server management system; and send server online information to the host-side device. The host-side device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

[0094] According to embodiments of this disclosure, the platform-side device may be a different platform server from the newly connected platform server, or other terminal devices, and is not limited thereto.

[0095] According to the embodiments of this disclosure, the following describes the timing process of data replication from the host device to the platform server, taking the newly accessed platform server as an example. In the following embodiments, the first preset value, the second preset value, the third preset value, the fourth preset value, the fifth preset value, and the sixth preset value are all set to Y.

[0096] Figure 4 A timing diagram of a data copying method according to an embodiment of the present disclosure is illustrated schematically.

[0097] like Figure 4 As shown, for the newly connected platform server 402, the data replication process can include an online process and a replication process.

[0098] According to embodiments of this disclosure, in the online process, after the platform server 402 goes online, it can send a server online event to the platform device 403. The platform device 403 can upload detection tools and data playback scripts to the platform server through the server management system. Simultaneously, the platform device 403 can send a server online message to the host device 401, and the host device 401 can establish and initialize a parameter interaction table. As an optional implementation, in the online process, the platform server 402 can also send server online events to both the host device 401 and the platform device 403 separately. Alternatively, the platform server 402 can first send a server online event to the host device 401, and after the parameter interaction table is established, the host device 401 notifies the platform device 403. That is, the execution sequence of the operation of the platform device 403 uploading the detection tools and data playback scripts and the operation of the host device 401 establishing the parameter interaction table is not limited.

[0099] According to embodiments of this disclosure, during the replication process, the platform server 402 can cyclically check whether the parameter value of the seventh field in the parameter interaction table is Y. After triggering a table access event, the host device 401 can stop the current log parsing process and update the parameter value of the seventh field to Y. After detecting that the parameter value of the seventh field is Y, the platform server 402 can close the current data playback process and import the table replication list of the data to be replicated from the host device 401. Then, the platform server 402 can query the time information of the most recent data record from the task log and use this time information to update the parameter value of the first field in the parameter interaction table. Then, it performs a cold start on the data playback process. After the cold start is completed, it can update the parameter value of the second field in the parameter interaction table to Y. The host device 401 can use various tools to perform cyclic queries on the parameter interaction table. After querying that the parameter value of the second field is Y, it can cold start the log parsing process and update the parameter value of the third field in the parameter interaction table to Y. After platform server 402 detects that the parameter value of the third field is Y, it can update the parameter value of the fourth field in the parameter interaction table to Y. After host device 401 detects that the parameter value of the fourth field is Y, it can pause the log parsing process, configure the log parsing process based on the parameter value of the first field, and update the parameter value of the fifth field to Y. After platform server 402 detects that the parameter value of the fifth field is Y, it can run the data replay process and update the parameter value of the sixth field to Y. After host device 401 detects that the parameter value of the sixth field is Y, it can run the log parsing process. At this point, the log parsing process of host device 401 can send the data to be copied to the message queue, and the data replay process of platform server 402 can replay the messages in the message queue in the database of platform server 402, thereby realizing data replication from host device 401 to platform server 402. After completing the data replication, host device 401 can also initialize the parameter values ​​of each field in the parameter interaction table to prepare for the next data replication task.

[0100] Figure 5 A block diagram of a data copying apparatus according to an embodiment of the present disclosure is shown schematically.

[0101] like Figure 5 As shown, the data replication device 500 includes a creation module 510, a consensus module 520, a loading module 530, and a running module 540. This data replication device 500 can be installed on a platform-side device.

[0102] Create module 510 to create a data playback process in response to a triggered data copy event.

[0103] The consensus module 520 is used to perform data replication consensus with the host device by utilizing the parameter exchange table between the detection tool and the host device, and obtain the consensus result.

[0104] Loading module 530 is used to load the data replay script into the data replay process when the consensus result indicates that a consensus has been reached.

[0105] The execution module 540 is used to run the data playback process, so as to pull the target message from the message queue and play back the target data obtained by parsing the target message in the database of the platform device.

[0106] According to embodiments of this disclosure, the host device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host device, obtain target data, and send a target message encapsulated from the target data to a message queue.

[0107] According to embodiments of this disclosure, when performing data replication from a host device to a platform device, the host device and the platform device can reach a consensus using a parameter exchange table. After reaching a consensus, the platform device can run a data playback process, and the host device can run a log parsing process, using a message queue to complete the data replication from the host device to the platform device. Through the above technical means, consensus based on a parameter exchange table can replace manual operations by technicians, thus at least partially overcoming the technical problem of low accuracy in data replication caused by significant manual operations in related technologies, thereby effectively improving the efficiency and accuracy of data replication.

[0108] According to embodiments of this disclosure, the creation module 510 includes a first creation unit, a second creation unit, and a third creation unit.

[0109] The first creation unit is used to stop the current data playback process in response to a triggered data copy event.

[0110] The second creation unit is used to import the table copy list of data to be copied from the host device.

[0111] The third creation unit is used to create a data replay process based on the table copy list.

[0112] According to embodiments of this disclosure, the operation module 540 includes a first operation unit, a second operation unit, a third operation unit, and a fourth operation unit.

[0113] The first execution unit is used to run the data playback process, retrieving messages sequentially from the message queue.

[0114] The second execution unit is used to filter messages by copying the list from the table and obtain the filtering results.

[0115] The third execution unit is used to parse the target message and obtain the target data when the filtering result indicates that the message is the target message.

[0116] The fourth execution unit is used to replay the target data in the database of the platform device using a data playback script.

[0117] According to embodiments of this disclosure, the operation module 540 further includes a fifth operation unit.

[0118] The fifth execution unit is used to return the message to the message queue if the filtering result indicates that the message is not the target message.

[0119] According to embodiments of this disclosure, the data copying apparatus 500 further includes a first determining module and a first sending module.

[0120] The first determination module is used to determine the time information of the data to be copied from the task log.

[0121] The first sending module is used to send time information to the host device.

[0122] According to embodiments of this disclosure, the host device is configured to update the parameter value of the first field in the parameter interaction table based on time information.

[0123] According to embodiments of this disclosure, the host device is configured to: load a log parsing script into a log parsing process; run the log parsing process to parse the database logs of the host device based on time information to obtain data to be copied; and encapsulate the data to be copied into multiple messages and send the multiple messages sequentially to a message queue.

[0124] According to embodiments of this disclosure, consensus module 520 includes a first consensus unit, a second consensus unit, and a third consensus unit.

[0125] The first consensus unit is used to update the parameter value of the second field of the parameter interaction table to the first preset value when it is determined that the data playback process has been successfully created. The host device is configured to create a log parsing process and update the parameter value of the third field of the parameter interaction table to the second preset value when the parameter value of the second field of the parameter interaction table is found to be the first preset value.

[0126] The second consensus unit is used to update the parameter value of the fourth field of the parameter interaction table to the third preset value when the parameter value of the third field of the parameter interaction table is detected by the detection tool. The host device is configured to configure the log parsing process based on the parameter value of the first field of the parameter interaction table when the parameter value of the fourth field of the parameter interaction table is found to be the third preset value, and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value.

[0127] The third consensus unit is used to obtain a consensus result that reaches a data replication consensus with the host device when the parameter value of the fifth field of the parameter interaction table is detected by the detection tool as the fourth preset value.

[0128] According to embodiments of this disclosure, the data copying apparatus 500 further includes an update module.

[0129] The update module is used to update the parameter value of the sixth field of the parameter interaction table to the fifth preset value when the data playback process is running.

[0130] According to an embodiment of this disclosure, the host device is configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table is found to be a fifth preset value.

[0131] According to an embodiment of this disclosure, the data copying event includes a parameter value of a sixth preset value in the seventh field of the parameter interaction table, and the parameter value of the seventh field of the parameter interaction table is obtained by detecting the parameter interaction table using a detection tool.

[0132] According to an embodiment of this disclosure, the host device is configured to stop the current log parsing process and update the parameter value of the seventh field to the sixth preset value in response to a triggered table access event.

[0133] According to embodiments of this disclosure, the platform-side device includes a platform server.

[0134] According to embodiments of this disclosure, the data copying apparatus 500 further includes a second determining module, a second sending module, and a third sending module.

[0135] The second determination module is used to determine the target platform server to be connected in response to the trigger server online event.

[0136] The second sending module is used to send detection tools and data playback scripts to the target platform server through the server management system.

[0137] The third sending module is used to send server online information to the host device, wherein the host device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

[0138] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0139] For example, any plurality of the creation module 510, consensus module 520, loading module 530, and running module 540 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the creation module 510, consensus module 520, loading module 530, and running module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the creation module 510, consensus module 520, loading module 530, and running module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0140] It should be noted that the data copying device part in the embodiments of this disclosure corresponds to the data copying method part in the embodiments of this disclosure. For a detailed description of the data copying device part, please refer to the data copying method part, which will not be repeated here.

[0141] Figure 6 A block diagram of an electronic device suitable for implementing a data copying method according to an embodiment of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0142] like Figure 6 As shown, a computer electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0143] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0144] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0145] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0146] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0147] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0148] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0149] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the data copying method provided in the embodiments of this disclosure.

[0150] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0151] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0152] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0154] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data replication method, applied to a platform-side device, the method comprising: In response to a data replication event, a data playback process is created; Using the parameter interaction table between the detection tool and the host device, a data replication consensus is achieved with the host device to obtain a consensus result; If the consensus result indicates that a consensus has been reached, the data replay script is loaded into the data replay process; as well as The data playback process is run to retrieve the target message from the message queue and replay the target data obtained by parsing the target message in the database of the platform device. The host device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host device, obtain the target data, and send the target message encapsulated from the target data to the message queue. The step of using the parameter interaction table between the detection tool and the host device to perform data replication consensus with the host device and obtain a consensus result includes: If the data playback process is successfully created, the parameter value of the second field of the parameter interaction table is updated to the first preset value. The host device is configured to create the log parsing process and update the parameter value of the third field of the parameter interaction table to the second preset value when the parameter value of the second field of the parameter interaction table is found to be the first preset value. If the detection tool detects that the parameter value of the third field of the parameter interaction table is the second preset value, the parameter value of the fourth field of the parameter interaction table is updated to the third preset value. The host device is configured to, upon finding that the parameter value of the fourth field of the parameter interaction table is the third preset value, configure the log parsing process based on the parameter value of the first field of the parameter interaction table, and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value. If the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, the consensus result of reaching the data replication consensus with the host device is obtained.

2. The method according to claim 1, wherein, The process of creating a data playback process in response to a triggered data replication event includes: In response to triggering the data copy event, stop the current data playback process; Import the table copy list containing the data to be copied from the host device; and The data replay process is created based on the table copy list.

3. The method according to claim 2, wherein, Running the data replay process to retrieve the target message from the message queue and replay the target data obtained by parsing the target message in the database of the platform device includes: Run the data playback process to retrieve messages sequentially from the message queue; The messages are filtered using the copied list to obtain the filtering results; If the filtering results indicate that the message is the target message, then the target message is parsed to obtain the target data; and The target data is replayed in the database of the platform device using the data playback script.

4. The method according to claim 3, further comprising: If the filtering results indicate that the message is not the target message, the message is returned to the message queue.

5. The method according to claim 2, further comprising: Determine the time information of the data to be copied from the task log; as well as Send the time information to the host device; The host device is configured to update the parameter value of the first field in the parameter interaction table based on the time information.

6. The method according to claim 5, wherein, The host device is configured as follows: Load the log parsing script into the log parsing process; Run the log parsing process to parse the database logs of the host device based on the time information to obtain the data to be copied; as well as The data to be copied is encapsulated into multiple messages, and the multiple messages are sent sequentially to the message queue.

7. The method according to claim 1, further comprising: While the data playback process is running, update the parameter value of the sixth field of the parameter interaction table to the fifth preset value; The host device is configured to load the log parsing script into the log parsing process and run the log parsing process when the parameter value of the sixth field of the parameter interaction table is the fifth preset value.

8. The method according to claim 1, wherein, The data copying event includes the parameter value of the seventh field of the parameter interaction table being a sixth preset value, and the parameter value of the seventh field of the parameter interaction table is obtained by detecting the parameter interaction table using the detection tool; The host device is configured to stop the current log parsing process and update the parameter value of the seventh field to the sixth preset value in response to a triggered table access event.

9. The method according to claim 1, wherein, The platform-side equipment includes a platform server; The method further includes: In response to the server going online event, determine the target platform server to be connected; The detection tool and the data playback script are sent to the target platform server through the server management system; and Send server online information to the host device, wherein the host device is configured to establish a parameter interaction table associated with the target platform server based on the server online information.

10. A data replication apparatus, disposed on a platform-side device, the apparatus comprising: Create a module to create a data playback process in response to a data copy event; The consensus module is used to perform data replication consensus with the host device by utilizing the parameter interaction table between the detection tool and the host device to obtain a consensus result. A loading module is used to load the data replay script into the data replay process when it is determined that the consensus result indicates that a consensus has been reached; as well as The running module is used to run the data playback process in order to pull the target message from the message queue and play back the target data obtained by parsing the target message in the database of the platform device; The host device is configured to load a log parsing script into a log parsing process and run the log parsing process to parse the database logs of the host device, obtain the target data, and send the target message encapsulated from the target data to the message queue. The consensus module is used for: If the data playback process is successfully created, the parameter value of the second field of the parameter interaction table is updated to the first preset value. The host device is configured to create the log parsing process and update the parameter value of the third field of the parameter interaction table to the second preset value when the parameter value of the second field of the parameter interaction table is found to be the first preset value. If the detection tool detects that the parameter value of the third field of the parameter interaction table is the second preset value, the parameter value of the fourth field of the parameter interaction table is updated to the third preset value. The host device is configured to, upon finding that the parameter value of the fourth field of the parameter interaction table is the third preset value, configure the log parsing process based on the parameter value of the first field of the parameter interaction table, and update the parameter value of the fifth field of the parameter interaction table to the fourth preset value. If the detection tool detects that the parameter value of the fifth field of the parameter interaction table is the fourth preset value, the consensus result of reaching the data replication consensus with the host device is obtained.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 9.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 9.

Images