Data processing method and related device

Abstract

The invention provides a data processing method and a related device, and the method comprises the steps: obtaining a transaction identifier of a transaction triggering a data synchronization task under the condition of triggering the data synchronization task; a data change list with the transaction identifier is searched, the data change list comprises first list information of first data, and the first data is data written into a first storage component based on a first task of the transaction; determining second list information corresponding to second data of the first storage component at the current moment; if the first list information is not matched with the second list information, backup data of the first data are searched from the first storage component based on the transaction identifier, and the backup data are data for backing up the first data under the condition that the first storage component receives a second task of the transaction; and synchronizing the backup data to a second storage component of the second server. The method can reduce the abnormal rate of data synchronization.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method, device, server, computer storage medium and computer program product. Background Art

[0002] A large number of applications are usually deployed in the computer room of a large business system to provide services to system users. In order to avoid service interruption, the server cluster in the computer room is usually configured with disaster recovery (also known as disaster recovery) to back up the data in the server cluster.

[0003] In the data synchronization process of executing the disaster recovery task, the data processing method usually synchronizes the data based on the data difference between the source server and the target server. It is necessary to read all the stored data in the source server and the target server at the same time for comparison, which takes a long time. In addition, before or during the execution of data synchronization, if there are two data processing tasks, it is easy to cause synchronization anomalies. For example, the execution of the data processing task requires deleting the data in the storage component before writing new data. Therefore, when synchronizing the data written by the first data processing task to the storage component, it may occur that the second data processing task has deleted the data written by the first data processing task and wrote new data in the storage component, resulting in the inability to find the data written by the first data processing task and causing synchronization anomalies. Summary of the invention

[0004] The embodiments of the present application disclose a data processing method, device, server, computer storage medium and computer program product.

[0005] In a first aspect, the present application provides a data processing method, which is applied to a first server, wherein the first server includes a first storage component, and the method includes: in the case of triggering a data synchronization task, obtaining a transaction identifier of a transaction that triggers the data synchronization task; searching for a data change list with the transaction identifier, wherein the data change list includes first list information of first data, wherein the first data is data written to the first storage component based on the first task of the transaction; determining second list information corresponding to second data of the first storage component at the current moment; if the first list information does not match the second list information, searching for backup data of the first data from the first storage component based on the transaction identifier, wherein the backup data is data for backing up the first data when the first storage component receives the second task of the transaction; and synchronizing the backup data to the second storage component of the second server.

[0006] In a second aspect, the present application provides a data processing device, which is applied to a first server, wherein the first server includes a first storage component, and the data processing device includes: a synchronization task triggering module, which, when a data synchronization task is triggered, obtains a transaction identifier of a transaction that triggers the data synchronization task; a table lookup module, which is used to search for a data change list with the transaction identifier, wherein the data change list includes first list information of first data, wherein the first data is data written to the first storage component based on the first task of the transaction; a determination module, which is used to determine second list information corresponding to second data of the first storage component at the current moment; a judgment module, which is used to search for backup data of the first data from the first storage component based on the transaction identifier if the first list information does not match the second list information, wherein the backup data is data for backing up the first data when the first storage component receives the second task of the transaction; and a synchronization module, which is used to synchronize the backup data to the second storage component of the second server.

[0007] In a third aspect, the present application provides a server, comprising a processor and a memory, wherein the processor is used to execute instructions stored in the memory so that the server performs a data processing method.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction implements a data processing method when executed by a processor.

[0009] In a fifth aspect, the present application provides a computer program product, wherein instructions are stored in the computer program product, and when the computer program product is executed on a computer, the computer implements a data processing method.

[0010] The data processing method of the present application is applied to a first server, which includes a first storage component. In the case of triggering a data synchronization task, the transaction identifier of the transaction that triggers the data synchronization task is obtained, so as to use the transaction identifier to find the data / list involved in executing the transaction. In the case of obtaining the transaction identifier, the data change list with the transaction identifier is found. Since the data change list includes the first list information of the first data, and the first data is the data written to the first storage component by the first task based on the transaction, the first list information of the data change list can be used for subsequent verification of whether the data in the first storage component is the first data. In order to verify, the second list information corresponding to the second data of the first storage component at the current moment is determined. If the first list information does not match the second list information, it means that there is a second task to back up the first data to generate backup data, and delete the first data after the backup, and write the second data in the first storage component. Therefore, in the case of the first list information not matching the second list information, the backup data of the first data can be found from the first storage component based on the transaction identifier, and the backup data can be synchronized to the second storage component of the second server. Avoid abnormal situations caused by the failure to find the data to be synchronized in the case of data synchronization, reduce the synchronization abnormality rate, and improve the synchronization efficiency to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application.

[0012] Figure 2 It is a flowchart of the data processing method provided in the embodiment of the present application.

[0013] Figure 3 It is a triggering flow diagram of a data synchronization task provided in an embodiment of the present application.

[0014] Figure 4 It is a schematic diagram of the synchronization process between the first analysis component and the second analysis component provided in an embodiment of the present application.

[0015] Figure 5 It is a schematic diagram of the architecture of the source server and the target server provided in the embodiment of the present application.

[0016] Figure 6 It is a schematic diagram of the architecture of the first server and the second server provided in an embodiment of the present application.

[0017] Figure 7 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0018] Figure 8 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0020] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0021] As mentioned above, a large number of applications are usually deployed in the computer room of a large business system to provide services to system users. In order to avoid service interruptions, the server cluster in the computer room is usually configured with disaster recovery to back up the data in the server cluster. Both the source server and the target server include a storage component and an analysis component, wherein the storage component can be a distributed file storage system in Hadoop, which has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware for data storage. Hadoop is an open source big data processing platform developed by the Apache Foundation. The analysis component can be a data warehouse analysis component of Hadoop, which is used to provide a structured query language (SQL) query method to analyze the data stored in the storage component.

[0022] Specifically, the analysis component can present a SQL command line interactive interface for users to set the storage strategy of the partition table data and related parameters of the partition table. For example, if the user chooses to establish partitions according to "days", the generated partition directory will be generated according to days. It can be understood that the above only takes "days" as an example, and it can also support partitions of other time intervals or partitions of other data types, which is not limited by this application. The analysis component can map structured data files to a database table and provide complete SQL query functions, such as querying the partition where the data is located in the storage component; or, the analysis component stores the data uploaded by the terminal device in the storage component based on the storage policy partition.

[0023] Generally, during the data synchronization process of performing disaster recovery tasks, data synchronization is usually performed based on the data differences between the source server and the target server's stored data. This requires reading all the stored data in the source server and the target server for comparison, which is time-consuming.

[0024] In addition, since the storage component and the analysis component have a sequential synchronization mechanism, if there are two data processing tasks, synchronization anomalies are easily caused based on the sequential synchronization mechanism. The sequential synchronization mechanism may include: when synchronizing the data stored in the source server to the storage component of the target server, the data of the storage component will be synchronized first, and then the data of the analysis component will be synchronized.

[0025] In one example, the execution of a data processing task requires deleting the data in the storage component before writing new data. When synchronizing the data written to the storage component by the first data processing task, if a second data processing task appears, the second data processing task will delete the data written by the first data processing task and write new data. Because the data of the first data processing task is deleted and the data synchronization corresponding to the first data processing task is not completed, a synchronization exception will occur because the data written based on the first data processing task cannot be found.

[0026] Therefore, in order to solve the technical problem that synchronization anomalies are prone to occur when performing data synchronization of disaster recovery tasks in related data processing methods, the embodiments of the present application provide a data processing method, device, server, computer storage medium and computer program product, which determines whether the data in the first storage component of the first server has been modified by comparing the first list information corresponding to the first task with the second list information corresponding to the current moment, thereby determining the corresponding data synchronization method according to the comparison between the first list information and the second list information. The abnormal rate of data synchronization is reduced and the accuracy of data synchronization is improved.

[0027] For ease of understanding, the technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 Schematic diagram of the application scenario of the data processing method provided in the embodiment of the present application. Figure 1 As shown, the data processing method provided in the embodiment of the present application can be applied to Figure 1 In the illustrated scenario, the scenario includes: a first server 10, a second server 20 and at least one terminal device 30. The first server 10 includes a first storage component 110 and a first analysis component 120, and the second server 20 includes a second storage component 210 and a second analysis component 220.

[0029] In some embodiments of the present application, the first server 10 can be connected to the second server 20 and at least one terminal device 30 in communication, and the first server 10 is used to perform data synchronization tasks on the second server 20 according to the data operation sent by at least one terminal device 30. Among them, the communication connection mode includes wired communication connection and wireless communication connection, and the wired communication connection can include one or more of the wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication connection can include one or more of the wireless fidelity (Wireless Fidelity, Wi-Fi), Bluetooth (Bluetooth, BT), mobile communication network, Frequency Modulation (Frequency Modulation, FM), Near Field Communication (Near Field Communication, NFC), infrared technology (Infrared, IR), etc.

[0030] The first server 10 and the second server 20 can be any server in a server cluster composed of multiple servers, or a central control server that manages the server cluster. The first server 10 and the second server 20 can be servers with the same functions and configurations, and the second server 20 can be used as a data backup device for the first server 10 to avoid loss of data stored in the first server 10.

[0031] The terminal device 30 may be a mobile phone, a tablet computer, a television (also referred to as a smart screen, a large-screen device, etc.), a laptop computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a vehicle-mounted device, a virtual reality device, etc.; or, the terminal device 30 may also include a server, such as an independent physical server. The embodiments of the present application do not impose any restrictions on this.

[0032] In one example, the first server 10 receives a data storage instruction from at least one terminal device 30, the data storage instruction carries data that the terminal device 30 needs to store, and uses the first analysis component 120 of the first server 10 to process the data to be stored and then store it in the first storage component 110. When the first storage component 110 receives the data to be stored, it first deletes the stored data and then writes the data to be stored into the first storage component 110. The first storage component 110 determines a data change list according to the deleted and written data, and modifies the data in the first analysis component 120 according to the data change list, thereby triggering a data synchronization task. In the case of triggering the data synchronization task, the first list information in the data change list and the second list information corresponding to the first storage component 110 at the current moment are obtained. If the first list information does not match the second list information, indicating that the data in the first storage component 110 is modified, the backup data is searched according to the transaction identifier. The backup data is synchronized to the second storage component 210 of the second server 20. After completing the data synchronization between the first storage component 110 and the second storage component 210, the data synchronization between the first analysis component 120 and the second analysis component 220 is performed.

[0033] It should be noted that the above application scenarios are only shown for the purpose of facilitating understanding of the principles of the present application, and the embodiments of the present application are not limited thereto. On the contrary, the embodiments of the present application can be applied to any scenario that may be applicable, for example, the disaster recovery system that performs the data synchronization task may not belong to the first server 10. In another application scenario, the first server 10 can be connected to an independent server, which can be used to deploy the disaster recovery system and monitor the data status of the first server 10 to trigger the first server 10 to perform data disaster recovery.

[0034] Figure 2 is a flow chart of a data processing method provided in an embodiment of the present application, which is applied to a first server (eg Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0035] Step S201, when a data synchronization task is triggered, obtain a transaction identifier of a transaction that triggers the data synchronization task.

[0036] In some embodiments of the present application, a data synchronization task indicates that the data of the first server is to be synchronized to the second server so that the data of the first server is consistent with the data of the second server. The second server may be a backup device of the first server. The first server may trigger a data synchronization task under preset conditions. For example, when the first server polls a new transaction log and determines that the preset conditions are met, the data synchronization task is triggered. The preset conditions may be set according to actual needs, and the present application does not limit this. In addition, the situation of triggering a data synchronization task may refer to the following example. Figure 3 The steps of the embodiment shown.

[0037] In some embodiments of the present application, a transaction represents a collection of a series of operations, and a transaction may be a transaction from a user of a system or an application. In other words, a transaction may be triggered or initiated by a user or an application. For example, a transaction may include a write request for writing data to a first storage component of a first server, or a read request for reading data from a first storage component.

[0038] In some embodiments of the present application, when the first server receives a transaction, it can create a directory for the transaction, and the directory can be used to store information corresponding to the transaction, such as the transaction identifier of the transaction. When a data synchronization task is triggered, the first server can obtain the transaction identifier from the directory so as to subsequently search for data / lists involved in executing the transaction based on the transaction identifier.

[0039] Step S202, searching for a data change list with a transaction identifier.

[0040] In some embodiments of the present application, the data change list includes first list information of first data, and the first data is data written to the first storage component based on the first task of the transaction. Among them, a transaction may include multiple tasks, and each task may perform different operations, such as a deletion operation, a write data operation, and a read data operation. The first task may be a write data operation. The first list information may include the first data, a check code of the first data, etc. Among them, the check code may be a cyclic redundancy check (CRC) code. After the first server finds the transaction identifier, it can find the data change list with the transaction identifier based on the transaction identifier. The first list information of the data change list can be used to subsequently verify whether the data in the first storage component is the first data.

[0041] Step S203, determining the second list information corresponding to the second data of the first storage component at the current moment.

[0042] In some embodiments of the present application, in order to verify the data in the first storage component, the data currently stored in the first storage component is used as the second data. The second list information corresponding to the second data is obtained, and the second list information includes the second data and the CRC code of the second data.

[0043] Step S204: if the first list information does not match the second list information, searching the first storage component for backup data of the first data based on the transaction identifier.

[0044] In some embodiments of the present application, since the first data is data written into the first storage component based on the first task, it is possible to determine whether the second task has modified the first data before data synchronization to reduce the synchronization anomaly rate. In some embodiments of the present application, determining whether the second task has modified the first data can be determined by comparing the first list information with the second list information.

[0045] If the first list information matches the second list information, it means that the second data is the first data, and when the first storage component executes the data synchronization task corresponding to the first task, there is no second task to modify the first data.

[0046] If the first list information does not match the second list information, it means that the second data is data written based on the second task. In order to write the second data, it is necessary to first delete the first data and back up the first data. Then, in the case where the first list information does not match the second list information, the backup data of the first data can be found from the first storage component according to the transaction identifier.

[0047] Step S205: synchronizing the backup data to the second storage component of the second server.

[0048] In some embodiments of the present application, in the process of executing the data synchronization task, the bandwidth resources between the first server and the second server will also have a certain impact on the data synchronization. Therefore, before synchronizing the backup data to the second server, the bandwidth resources between the first server and the second server can be detected to avoid synchronization failure due to full bandwidth. Among them, full bandwidth means that the transmission capacity of the network is fully utilized and the maximum transmission rate is reached. Full bandwidth may cause network delays, data loss, service interruptions and other problems.

[0049] In some embodiments of the present application, the first bandwidth resource required for synchronous backup data is obtained, and the current bandwidth resource between the first server and the second server is obtained, and the current bandwidth resource is the bandwidth that is not occupied between the first server and the second server at the current moment. In order to avoid full bandwidth, the second bandwidth resource can be dynamically set according to the current bandwidth resource, and the second bandwidth resource is less than the current bandwidth resource. If the first bandwidth resource is less than the second bandwidth resource, it means that the synchronous backup data will not have a full bandwidth situation, and the backup data can be synchronized to the second storage component of the second server.

[0050] If the first bandwidth resource is greater than or equal to the second bandwidth resource, it means that the running tasks between the first server and the second server at the current moment occupy more bandwidth resources, and the synchronization of backup data may be full of bandwidth. In the case where the first bandwidth resource is greater than or equal to the second bandwidth resource, you can wait for the first server and the second server to release bandwidth resources. The first server can continuously compare the first bandwidth resource with the second bandwidth resource until the first bandwidth resource is less than the second bandwidth resource, and synchronize the backup data to the second storage component of the second server.

[0051] In an embodiment of the present application, in the case of triggering a data synchronization task, the transaction identifier of the transaction that triggers the data synchronization task is obtained so that the transaction identifier is used to find the data / list involved in executing the transaction. In the case of obtaining the transaction identifier, the data change list with the transaction identifier is searched. Since the data change list includes the first list information of the first data, and the first data is the data written to the first storage component by the first task based on the transaction, the first list information of the data change list can be used to subsequently verify whether the data in the first storage component is the first data. In order to verify, the second list information corresponding to the second data of the first storage component at the current moment is determined. If the first list information does not match the second list information, it means that there is a second task to back up the first data to generate backup data, and delete the first data after the backup, and write the second data in the first storage component. Therefore, in the case of the first list information not matching the second list information, the backup data of the first data can be found from the first storage component based on the transaction identifier, and the backup data can be synchronized to the second storage component of the second server. Avoid abnormal situations caused by the failure to find the data to be synchronized in the case of data synchronization, reduce the synchronization abnormality rate, and improve the synchronization efficiency to a certain extent.

[0052] Figure 3 1 is a schematic diagram of the triggering process of the data synchronization task provided in the embodiment of the present application. In order to speed up the timeliness of disaster recovery, the data synchronization task can be triggered according to the preset rules. Figure 3 The process shown includes the following steps.

[0053] Step S301, when the first storage component receives the first task, delete the third data in the first storage component and write the first data.

[0054] In some embodiments of the present application, the first server is connected to at least one terminal device and is used to store data of the at least one terminal device. For example, if the terminal device is a device for managing enterprise operations, the data stored by the terminal device to the first server is business data. The first server includes a first analysis component and a first storage component. The first analysis component can present a structured query language (SQL) command interaction interface for the user to set the storage strategy of the partition table data and the related parameters of the partition table. The first storage component is used to store data, for example, business data sent by the terminal device.

[0055] In some embodiments of the present application, the first server may receive the data to be written sent by the terminal device, and the data to be written is first processed by the first analysis component. The processing of the data to be written using the SQL engine may include data filtering, data sorting, data aggregation, data connection, and data conversion. After the first analysis component completes the data processing, it generates a first task and sends the first task to the first storage component.

[0056] In some embodiments of the present application, a plurality of data storage partitions divided by date may be provided in the first storage component, for example, partition 1 is used to store data of No. 1, partition 2 is used to store data of No. 2, etc. The above are just examples, and the partitions may also be divided by day or time, and the present application does not limit the division of partitions. When the first storage component receives the first task, it determines the partition to be updated by the first task and deletes the third data in the partition. After deleting the third data, the first data is written to the partition.

[0057] In one example, partition 1 stores the third data on August 1st. When the first storage component receives the first task on September 1st, it deletes the third data in partition 1 and writes the first data into partition 1.

[0058] Step S302, writing the first list information of the first data and the third list information of the third data into the data change list.

[0059] In some embodiments of the present application, after the first storage component deletes the third data and writes the first data, a data change list is generated according to the first data and the third data, and the first list information of the first data and the third list information of the third data are determined. The first list information includes the CRC code of the first data, and the third list information includes the CRC code of the third data. The CRC code of the first data and the CRC code of the third data are written into the data change list.

[0060] Step S303: modify the data in the first analysis component in the first server based on the data change list to obtain first modified data.

[0061] In some embodiments of the present application, since the data change list records the data modification of the first storage component based on the first task, in order to ensure that the metadata of the first analysis component has corresponding storage data in the first storage component, the data in the first analysis component is modified using the data change list, thereby obtaining the first modified data. The metadata refers to the general term for detailed information of all objects (such as tables, partitions, columns, etc.) in the first server.

[0062] Step S304: Record a transaction log corresponding to the first modification data in the first analysis component.

[0063] In some embodiments of the present application, after obtaining the first modified data, the first analysis component may record a transaction log corresponding to the first modified data, wherein the transaction log includes a data change list, a transaction identifier, and the first modified data.

[0064] Step S305: If a transaction log exists, trigger a data synchronization task.

[0065] In some embodiments of the present application, the disaster recovery system can be deployed in the first server or in an independent server connected to the first server. The disaster recovery system monitors in real time whether there is a new transaction log (such as the transaction log generated in step S304) of the first analysis component, and triggers the data synchronization task if there is a new transaction log. By triggering the data synchronization task by monitoring the transaction log, the trigger interval of the disaster recovery can be reduced from the day / hour level to the minute / second level, thereby increasing the timeliness of the data disaster recovery.

[0066] In addition, the time to trigger disaster recovery can be preset in the disaster recovery system. For example, when the disaster recovery system detects a new transaction log, it starts calculating the duration. If the preset duration is reached, the data synchronization task is triggered. This application does not limit the triggering time of the data synchronization task.

[0067] In the embodiment of the present application, by monitoring the transaction log to trigger the data synchronization task, the triggering interval of disaster recovery can be reduced from days / hours to minutes / seconds, thereby increasing the timeliness of data disaster recovery.

[0068] Figure 4 is a schematic diagram of the synchronization process between the first analysis component and the second analysis component provided in an embodiment of the present application, such as Figure 2 In step S205 shown, after synchronizing the backup data to the second storage component of the second server, based on the sequential synchronization mechanism between the first server and the second server, after the first storage component and the second storage component complete the synchronization, synchronization between the first analysis component and the second analysis component is performed.

[0069] Step S401: Obtain a record table with a transaction identifier from a database of a first server.

[0070] In some embodiments of the present application, after each data modification is completed, the first analysis component will record the source table and the output table in the record table of the database. The record table will include the source table and the output table corresponding to each task in the transaction and the transaction identifier.

[0071] Step S402, determining whether there is a source table and an output table matching the data table in the record table.

[0072] In some embodiments of the present application, the first modified data includes a data table, which is a database table used by the first task during the operation. The first modified data is data obtained by modifying the data in the first analysis component in the first server based on the data change list when the first storage component receives the first task. Since the first task generates many data tables during the operation, these data tables may be temporary tables. In order to reduce the bandwidth resources occupied by temporary tables during the data synchronization process, it can be determined whether the first modified data contains temporary tables.

[0073] In one example, the SQL statement included in the execution of the first task is: insert t_target select fromt_source where c = 'a';insert t_target2 select from t_target; This SQL statement is used to copy data from one data table to another. Specifically, select all records with the c field value of 'a' from the t_source table, and then insert these records into the t_target table. Next, insert all records in the t_target table into the t_target2 table. Therefore, in the above process, the t_target table is an output table relative to the t_source table, and is a source table relative to the t_target2 table. Therefore, the source tables recorded in the record table include: t_source table and t_target table; the output tables recorded in the record table include: t_target table and t_target2 table. Therefore, if the data table is the t_target table, then the data table is both a source table and an output table, and the data table is determined to be a temporary table.

[0074] In some embodiments of the present application, if there are source tables and output tables matching the data table in the record table, step S403 is executed. If there is no source table matching the data table in the record table, and there is an output table matching the data table in the record table, the process jumps to step S405.

[0075] Step S403, deleting the data table from the first modified data to obtain second modified data.

[0076] In some embodiments of the present application, if it is determined that the data table is a temporary table, in order to avoid the temporary table occupying bandwidth resources, the data table can be deleted from the first modified data, thereby obtaining the second modified data that does not contain the temporary table.

[0077] Step S404: Synchronize the copy data of the second modified data to the second analysis component of the second server.

[0078] In some embodiments of the present application, before synchronizing the copy data of the second modified data to the second analysis component of the second server, it is also possible to determine the unoccupied bandwidth resources between the first server and the second server to avoid the situation where the bandwidth is full. Obtain a third bandwidth resource for synchronizing the copy data of the second modified data. If the third bandwidth resource is less than the first bandwidth resource, synchronize the copy data of the second modified data to the second analysis component of the second server. If the third bandwidth resource is greater than or equal to the first bandwidth resource, wait for the first server and the second server to release the bandwidth resources until the third bandwidth resource is less than the first bandwidth resource, and then synchronize the copy data of the second modified data to the second analysis component of the second server.

[0079] Step S405: synchronize the copy data of the first modified data to the second analysis component.

[0080] In some embodiments of the present application, if it is determined that the data table is not a temporary table, a fourth bandwidth resource required for synchronizing the copy data of the first modified data is determined. If the fourth bandwidth resource is less than the first bandwidth resource, the copy data of the first modified data is synchronized to the second analysis component of the second server. If the fourth bandwidth resource is greater than or equal to the first bandwidth resource, the first server and the second server may be waited for to release the bandwidth resource until the fourth bandwidth resource is less than the first bandwidth resource, and then the copy data of the first modified data is synchronized to the second analysis component of the second server.

[0081] In the embodiment of the present application, by deleting the temporary table in the first modified data, the bandwidth resources required for data synchronization are saved, the synchronization exception rate is reduced to a certain extent, and the efficiency of data synchronization is improved.

[0082] To better understand the Figure 2 , Figure 3 and Figure 4 To better understand the improvements and advantages of the embodiments of the present application compared to the related art, the following is combined with Figure 5 and Figure 6 Provide explanation.

[0083] Figure 5 It is a schematic diagram of the architecture of the source server and the target server provided in the embodiment of the present application, including the execution order of related technologies. Figure 6 It is a schematic diagram of the architecture of the first server and the second server provided in the embodiment of the present application, including the execution order of the embodiment of the present application.

[0084] like Figure 5 As shown, the source server 31 includes a disaster recovery system that triggers data synchronization tasks at regular intervals (such as Figure 5 ①), triggering the storage component 312 of the source server 31 to start synchronizing data (such as Figure 5 For example, the data of partition 3 in table 2 of storage component 312 of source server 31 is synchronized to partition 3 in table 2 of storage component 322 of target server 32. After the disaster recovery system determines that data synchronization is completed, it starts synchronizing metadata. The disaster recovery system triggers the analysis component 31 of source server 31 to synchronize metadata (such as Figure 5 Line ④ shown), for example, synchronizes partition 3 of the metadata of table 2 in the analysis component 311 of the source server 31 to partition 3 of the metadata of table 2 in the target server 32.

[0085] like Figure 5The scheme shown may fail to find the data to be synchronized, resulting in synchronization anomalies. For example, after the above process is executed, the first data is stored in the storage component 312. Before the data synchronization between the storage component 312 of the source server 31 and the storage component 322 of the target server 32 is executed, there is another task that deletes the first data and writes the second data. After the data synchronization task is triggered, when the data synchronization between the storage component 312 and the storage component 322 is executed, the synchronization fails because the first data cannot be found.

[0086] In addition, due to the sequential synchronization mechanism, the data of the storage component 312 needs to be synchronized first and then the data of the analysis component 311. Therefore, when performing data synchronization between the storage component 312 and the storage component 322, if there is another task that writes the second data and modifies the metadata of the analysis component 311, since the synchronization task of the storage component 312 has been started, the second data written at this time will not be synchronized to the storage component 322, but when synchronizing the metadata of the analysis component 311, the metadata corresponding to the second data will be synchronized to the analysis component 321. This results in that in the second server, the metadata stored in the analysis component 321 does not have corresponding storage data in the storage component 322.

[0087] In order to avoid the Figure 5 In the case of the example shown, the present application embodiment provides Figure 6 Example shown.

[0088] like Figure 6 As shown, the first server 10 includes a disaster recovery system. The first server executes the following sequence ① to ⑦: Based on the first task, the first data is written to the first storage component 110 (such as Figure 6 1) of the table 2 data partition 3, delete the third data pre-stored in partition 3. Generate a data change list based on the first data and the third data. Use the data change list to modify the data in the first analysis component 120 (such as Figure 6 The first analysis component 120 records a transaction log (such as Figure 6 Line ③ shown in the figure), the disaster recovery system monitors the transaction log (such as Figure 6 The data synchronization task is triggered when the line ④ shown in Figure 6 When the bandwidth resource meets the data synchronization condition, it is determined whether the first data stored in the first storage component 110 is modified.

[0089] The first server 10 obtains the second list information corresponding to the second data of the first storage component 110 at the current moment. Compare the first list information with the second list information in the data change list. If the first list information matches the second list information, it means that there is no second task to modify the first data. Synchronize the copy data of the first data to the second storage component 210. Continuing with the above example, the second storage component 210 first deletes the data in partition 3 of table 2 in the second storage component 210, and then writes the copy data of the first data, completing the data synchronization between the first storage component 110 and the second storage component 210.

[0090] If the first list information does not match the second list information, it means that there is a second task that deletes the first data and writes the second data. If there is a second task, the first server 10 deletes the first data and backs up the first data to obtain backup data. After deleting the first data, the second data is written. In an example, the first server 10 deletes the first data of partition 3, writes the second data in partition 3, and backs up the first data to obtain backup data (such as Figure 6 Then, when executing the data synchronization task corresponding to the first task, the data of the snapshot of partition 3 is synchronized to the second storage component 210 (such as Figure 6 In the second storage component 210, the data stored in partition 3 is first deleted, and then the snapshot data of partition 3 is written. After the data synchronization between the first storage component 110 and the second storage component 210 is completed, the data synchronization between the first analysis component 120 and the second analysis component 220 is performed based on the first modified data (such as Figure 6 Line ⑦ shown).

[0091] like Figure 6 The execution order of ① to ⑦ shown can avoid synchronization anomalies caused by the deletion of the first data. At the same time, it can also ensure that after data synchronization, the metadata in the second analysis component 220 has corresponding storage data in the second storage component 210, avoiding the scenario where the storage data does not exist but the metadata exists.

[0092] Figure 7 Schematic diagram of the structure of the data processing device provided in the embodiment of the present application. Figure 7 As shown, the data processing device 70 may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the data processing device 70 may be stored in the memory of the first server and executed by at least one processor to perform (see Figure 2 Description) Data processing functions.

[0093] In this embodiment, the data processing device 70 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a synchronization task trigger module 710, a table lookup module 720, a determination module 730, a judgment module 740, and a synchronization module 750. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the data processing device 70 can be used to implement the following Figures 2 to 6 The data processing method of the embodiment shown is as follows: Figure 7 As shown, the data processing device 70 is applied to the first server (such as Figure 1 The first server 10 shown in the figure includes a first storage component, and the data processing device 70 includes the following modules: The synchronization task triggering module 710, when triggering a data synchronization task, obtains the transaction identifier of the transaction that triggers the data synchronization task. The table lookup module 720, is used to search for a data change list with the transaction identifier, the data change list includes first list information of first data, the first data is data written to the first storage component based on the first task of the transaction. The determination module 730, is used to determine the second list information corresponding to the second data of the first storage component at the current moment. The judgment module 740, is used to search for backup data of the first data from the first storage component based on the transaction identifier if the first list information does not match the second list information, the backup data is the data for backing up the first data when the first storage component receives the second task of the transaction. The synchronization module 750, is used to synchronize the backup data to the second storage component of the second server.

[0094] In some embodiments of the present application, before triggering the data synchronization task, the data processing device 70 also includes: when the first storage component receives the first task, deleting the third data in the first storage component and writing the first data; writing the first list information of the first data and the third list information of the third data into the data change list; modifying the data in the first analysis component in the first server based on the data change list to obtain first modified data; and recording the transaction log corresponding to the first modified data in the first analysis component.

[0095] In some embodiments of the present application, the synchronization task triggering module 710 is further used to trigger the data synchronization task when the transaction log exists.

[0096] In some embodiments of the present application, the first modified data includes a data table, which is a database table used by the first task during the operation process. After synchronizing the backup data to the second storage component of the second server, the data processing device 70 is also used to: obtain a record table with the transaction identifier from the database of the first server, and the record table includes a source table and an output table for each task in the transaction; if there is a source table and an output table matching the data table in the record table, delete the data table from the first modified data to obtain the second modified data; synchronize the copy data of the second modified data to the second analysis component of the second server.

[0097] In some embodiments of the present application, the data processing device 70 is also used to: if there is no source table matching the data table in the record table, and there is an output table matching the data table in the record table, then synchronize the copy data of the first modified data to the second analysis component.

[0098] In some embodiments of the present application, the synchronization module 750 is used to: synchronize the backup data to the second storage component of the second server if the first bandwidth resources required to synchronize the backup data are less than the second bandwidth resources, and the second bandwidth resources are determined based on the unoccupied bandwidth resources between the first server and the second server.

[0099] Figure 8 Schematic diagram of the structure of the server provided in the embodiment of the present application. Figure 1 The first server 10 and the second server 20 shown may be servers 80 having the same structure. Figure 8 As shown, the server 80 may be a central server, an edge server, or a local server of a local data center. Each server 80 may include a communication module 801, a memory 802, a processor 803, an input / output (I / O) interface 804, and a bus 805. The processor 803 is coupled to the communication module 801, the memory 802, and the I / O interface 804 through the bus 805.

[0100] The communication module 801 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), bluetooth (bluetooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0101] The memory 802 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory may be directly read and written by the processor 803, and may be used to store executable programs (such as machine instructions) of an operating system or other running programs, and may also be used to store user and application data, etc. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0102] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0103] The memory 802 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 803. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 803, the data disaster recovery method executed on the server 80 can be implemented.

[0104] In other embodiments, the server 80 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the server 80 .

[0105] The processor 803 may include one or more processing units, for example, the processor 803 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0106] The processor 803 provides computing and control capabilities. For example, the processor 803 is used to execute computer programs stored in the memory 802 .

[0107] The I / O interface 804 is used to provide a channel for user input or output. For example, the I / O interface 804 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0108] The bus 805 is at least used to provide a channel for mutual communication among the communication module 801 , the memory 802 , the processor 803 , and the I / O interface 804 in the server 80 .

[0109] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the server 80. In other embodiments of the present application, the server 80 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0110] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data extraction method embodiment or each process of the above-mentioned data extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is, such as a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a disk or an optical disk, etc.

[0111] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the server reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the server executes the data extraction method described above in the embodiment of the present application.

[0112] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0114] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data processing method, applied to a first server, characterized in that: The first server includes a first storage component, and the method includes: In the case of triggering a data synchronization task, obtaining a transaction identifier of the transaction that triggers the data synchronization task; Searching for a data change list having the transaction identifier, where the data change list includes first list information of first data, where the first data is data written into the first storage component based on a first task of the transaction; Determine second list information corresponding to second data of the first storage component at the current moment; If the first list information does not match the second list information, searching the first storage component for backup data of the first data based on the transaction identifier, the backup data being data used to back up the first data when the first storage component receives the second task of the transaction; The backup data is synchronized to a second storage component of a second server.

2. The data processing method according to claim 1, characterized in that: Before triggering the data synchronization task, the method further includes: When the first storage component receives the first task, deleting the third data in the first storage component and writing the first data; Writing the first list information of the first data and the third list information of the third data into the data change list; Modify the data in the first analysis component in the first server based on the data change list to obtain first modified data; In the first analysis component, a transaction log corresponding to the first modification data is recorded.

3. The data processing method according to claim 2, characterized in that: The triggering data synchronization task includes: In the case where the transaction log exists, the data synchronization task is triggered.

4. The data processing method according to claim 2, characterized in that: The first modified data includes a data table, which is a database table used by the first task during operation. After synchronizing the backup data to the second storage component of the second server, the method further includes: Acquire a record table with the transaction identifier from a database of the first server, wherein the record table includes a source table and an output table of each task in the transaction; If there is a source table and an output table matching the data table in the record table, the data table is deleted from the first modified data to obtain second modified data; The copy of the second modified data is synchronized to the second analysis component of the second server.

5. The data processing method according to claim 4, characterized in that: The method further comprises: If there is no source table matching the data table in the record table, and there is an output table matching the data table in the record table, the copy data of the first modified data is synchronized to the second analysis component.

6. The data processing method according to claim 1, characterized in that: The second storage component for synchronizing the backup data to the second server includes: If the first bandwidth resource required for synchronizing the backup data is less than the second bandwidth resource, the backup data is synchronized to the second storage component of the second server, and the second bandwidth resource is determined according to the unoccupied bandwidth resource between the first server and the second server.

7. A data processing device, applied to a first server, characterized in that: The first server includes a first storage component, and the data processing device includes: A synchronization task triggering module, when a data synchronization task is triggered, obtains a transaction identifier of a transaction that triggers the data synchronization task; a table lookup module, configured to search for a data change list having the transaction identifier, wherein the data change list includes first list information of first data, where the first data is data written into the first storage component based on a first task of the transaction; A determination module, used to determine second list information corresponding to second data of the first storage component at a current moment; a judgment module, configured to search the first data's backup data from the first storage component based on the transaction identifier if the first list information does not match the second list information, the backup data being data used to back up the first data when the first storage component receives the second task of the transaction; The synchronization module is used to synchronize the backup data to the second storage component of the second server.

8. A server, characterized in that: The server includes a processor and a memory, and the processor is used to execute instructions stored in the memory, so that the server executes the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product stores instructions, and when the computer program product is executed, the computer implements the data processing method according to any one of claims 1 to 6.

Images