Data synchronization system, method, device, equipment, medium and product

Abstract

The present application discloses a data synchronization system, method, device, equipment, medium and product, which relates to the field of computer technology. The system includes a management platform, an intermediate storage terminal and a data synchronization receiving platform; the intermediate storage terminal is used to obtain the first data from the data synchronization sending platform; the management platform is used to obtain the data synchronization task corresponding to the first data; the data synchronization receiving platform is used to receive the data synchronization request; in response to the data synchronization request, the first data is read from the intermediate storage terminal; data synchronization processing is performed on the first data; when the synchronization of the first data is completed, a first status update message is sent to the management platform; the management platform is used to switch the task status of the data synchronization task to the completion state after receiving the first status update message. The system allows the data synchronization process of the first data to maintain the consistency of the task status in a cross-platform environment, and ensures the data synchronization efficiency when the data is synchronized across platforms.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization system, method, device, equipment, medium and product. Background Art

[0002] In a wide range of application scenarios in the Internet field, heterogeneous data of different types, versions or structures are often generated in different clients and background applications. These heterogeneous data may be stored on different hardware platforms, or in different types of databases, distributed file storage systems, or directly in files. A significant requirement in the process of data processing and use is the need to synchronize multi-source heterogeneous data from multiple platforms into the same storage unit using data synchronization tools and methods for further high-level data analysis and calculation.

[0003] In the related art, traditional data synchronization methods are used to achieve synchronization of heterogeneous data between different platforms, such as MySQL binary log replication, platform application programming interface (Application Programming Interface, API) calls, etc.

[0004] However, faced with the diversity of data sources and the complexity of cross-platform synchronization, during the cross-platform synchronization of heterogeneous data, data synchronization failures occur due to differences in software and hardware between platforms, thereby reducing the data synchronization efficiency of asynchronous data during cross-platform synchronization. Summary of the invention

[0005] The embodiments of the present application provide a data synchronization system, method, device, equipment, medium and product. The technical solution is as follows:

[0006] On the one hand, a data synchronization system is provided, the system comprising a management platform, an intermediate storage terminal and a data synchronization receiving platform;

[0007] The intermediate storage end is used to obtain first data from the data synchronization sending platform, where the first data is data to be synchronized to the data synchronization receiving platform;

[0008] The management platform is used to obtain a data synchronization task corresponding to the first data, wherein the task status of the data synchronization task is in a pending synchronization status;

[0009] The data synchronization receiving platform is used to receive a data synchronization request; in response to the data synchronization request, read the first data from the intermediate storage end; perform data synchronization processing on the first data; and send a first status update message to the management platform when the synchronization of the first data is completed;

[0010] The management platform is used to switch the task state of the data synchronization task to a completed state after receiving the first status update message.

[0011] In some optional embodiments, the system further includes a first data platform;

[0012] The first data platform is used to cyclically detect the task status of the data synchronization task stored in the management platform, and determine that the data synchronization task is completed when it is detected that the task status is a success status.

[0013] In some optional embodiments, the first data platform is further configured to send the data synchronization request to the data synchronization receiving platform when a task start condition of the data synchronization task is met.

[0014] In some optional embodiments, the first data platform is further used to obtain a data synchronization task notification corresponding to the first data, the data synchronization task notification being used to indicate that the task state of the data synchronization task is in the pending synchronization state; and send the data synchronization task notification to the management platform;

[0015] The management platform is further configured to, upon receiving the data synchronization task notification sent by the first data platform, record that the task status of the data synchronization task is in the pending synchronization status.

[0016] In some optional embodiments, the first data platform is the data synchronization sending platform, and the first data platform includes a first data synchronization module and a first storage unit;

[0017] The first data synchronization module is used to read the first data from the first storage unit; write the first data to the intermediate storage end; and generate the data synchronization task notification corresponding to the first data.

[0018] In some optional embodiments, the system further includes a second data platform, which is the data synchronization sending platform;

[0019] The second data platform is used to write the first data into the intermediate storage end; generate the data synchronization task notification corresponding to the first data; and send the data synchronization task notification to the first data platform.

[0020] In some optional embodiments, the data synchronization receiving platform further includes a second data synchronization module and a second storage unit;

[0021] The second data synchronization module is used to obtain the first attribute information of the first data before data synchronization; when the first data is read from the intermediate storage end, detect the second attribute information corresponding to the first data; when the first attribute information and the second attribute information match, write the first data into the second storage unit; when it is determined that the first data has been written into the second storage unit, determine that the synchronization of the first data is completed.

[0022] In some optional embodiments, the first attribute information includes at least one of the following:

[0023] The data volume of the first data;

[0024] a data type of the first data;

[0025] a data format of the first data;

[0026] index information of the first data;

[0027] A hash value of the first data.

[0028] In some optional embodiments, the management platform is used to obtain a data synchronization task notification corresponding to the first data, the data synchronization task notification including the first attribute information; and send the first attribute information to the second data synchronization module based on the data synchronization task notification.

[0029] In some optional embodiments, the data synchronization receiving platform further includes a second data synchronization module and a second storage unit;

[0030] The second data synchronization module is used to perform data conversion processing on the first data to obtain second data when the first data and the data stored in the second storage unit are heterogeneous data; write the second data into the second storage unit; and determine that the synchronization of the first data is completed when it is determined that the second data has been written into the second storage unit.

[0031] In some optional embodiments, the data synchronization receiving platform is further used to, when it is determined that the synchronization of the first data has failed, clear the acquired first data; re-read the first data from the intermediate storage end; and perform data synchronization processing on the re-read first data.

[0032] In some optional embodiments, the data synchronization receiving platform is further configured to send a second status update message to the management platform when the number of first data synchronization failures reaches a first threshold;

[0033] The management platform is further used to switch the task status of the data synchronization task to a failure status when receiving the second status update message; and generate a data synchronization failure notification, wherein the data synchronization failure notification is used to notify manual intervention in the data synchronization task.

[0034] In some optional embodiments, the system further includes a first data platform;

[0035] The first data platform is further used to cyclically detect the task status of the data synchronization task stored in the management platform, and stop the cyclic detection of the task status when it is detected that the task status is a failure status.

[0036] On the other hand, a data synchronization method is provided, the method being executed by a management platform, the method comprising:

[0037] Acquire a data synchronization task corresponding to first data, wherein the task status of the data synchronization task is in a pending synchronization state, the first data is data written to an intermediate storage end by a data synchronization sending platform, and the first data is data to be synchronized to a data synchronization receiving platform;

[0038] When the first status update message sent by the data synchronization receiving platform is received, the task status of the data synchronization task is switched to the completion status; the first status update message is sent by the data synchronization receiving platform in response to the received data synchronization request, reading the first data from the intermediate storage end, and performing data synchronization processing on the first data after the first data synchronization is completed, and the data synchronization request is sent after the data synchronization task is started.

[0039] In another aspect, a data synchronization device is provided, the device comprising:

[0040] An acquisition module is used to acquire a data synchronization task corresponding to first data, wherein the task status of the data synchronization task is in a to-be-synchronized state, the first data is data written to an intermediate storage end by a data synchronization sending platform, and the first data is data to be synchronized to a data synchronization receiving platform;

[0041] A switching module is used to switch the task status of the data synchronization task to a completion status when a first status update message sent by the data synchronization receiving platform is received; the first status update message is a response by the data synchronization receiving platform to a received data synchronization request, the first data is read from the intermediate storage end, and data synchronization processing is performed on the first data, and the message is sent after the first data synchronization is completed, and the data synchronization request is sent after the data synchronization task is started.

[0042] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a data synchronization method as described in any of the above-mentioned embodiments of the present application.

[0043] On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a data synchronization method as described in any of the above-mentioned embodiments of the present application.

[0044] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data synchronization method described in any of the above embodiments.

[0045] The technical solution provided by this application includes at least the following beneficial effects:

[0046] When realizing data synchronization of the first data between the data synchronization sending platform and the data synchronization receiving platform, the first data is temporarily stored through the intermediate storage end, and the task status of the data synchronization task of the first data during the data synchronization process is recorded and updated through the management platform, that is, after the data synchronization receiving platform completes the data synchronization of the first data, the data synchronization receiving platform updates the task status recorded in the management platform to the completion status through the first status update message, so that the data synchronization sending platform can know whether the data synchronization task of the first data is completed at any time through the management platform, so that the data synchronization sending platform and the data synchronization receiving platform are coordinated through the management platform, so that the data synchronization process of the first data maintains consistent task status in a cross-platform environment, so that in the cross-platform data synchronization process, the information recognition of the task status between different platforms is unified to ensure the data synchronization efficiency when the data is synchronized across platforms.

[0047] At the same time, the intermediate storage end, as an intermediate storage of data, opens up the data transmission channel between different platforms. Since the management platform is set as the intermediate storage of the task status, after the data synchronization sending platform sends the first data in the data synchronization task to the intermediate storage end, the data synchronization sending platform only needs to pay attention to the task status in the management platform. Even if the data synchronization receiving platform fails to synchronize, it can also clear the data and re-acquire the first data from the intermediate storage end for data synchronization. Different platforms no longer need to re-acquire the first data through retransmission instructions after data synchronization fails, which reduces the data interaction requirements between platforms and avoids the repeated cross-platform transmission of the first data when it is heterogeneous data. It avoids the high failure rate of heterogeneous data when it is transmitted across platforms, and improves the data synchronization efficiency when the data is synchronized across platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0050] Figure 2 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0051] Figure 3 is an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application;

[0052] Figure 4 is an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a data synchronization system provided by an exemplary embodiment of the present application;

[0054] Figure 6 is an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application;

[0055] Figure 7 is a schematic diagram of a data synchronization system provided by an exemplary embodiment of the present application;

[0056] Figure 8 It is a schematic diagram of the overall architecture of a message synchronization system provided by an exemplary embodiment of the present application;

[0057] Fig. 9is a flow chart of a data synchronization method provided by an exemplary embodiment of the present application;

[0058] Fig.10 is a schematic diagram of a visual interface provided to a user by a management platform provided by an exemplary embodiment of the present application;

[0059] Fig.11 is a structural block diagram of a data synchronization device provided by an exemplary embodiment of the present application;

[0060] Fig.12 is a structural block diagram of a data synchronization device provided by an exemplary embodiment of the present application;

[0061] Fig.13 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0063] In this application, the terms "first", "second", etc. are used to distinguish identical items or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first" and "second", nor are there any limitations on quantity and execution order.

[0064] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0065] Heterogeneous data: refers to data of different types, different versions or different structures between data. This data may have differences in computer architecture, basic operating system, database management system itself, etc. For example, a heterogeneous database system may include databases running on different hardware platforms (e.g., mainframes, minicomputers, workstations, personal computers (PCs) or embedded systems), these databases may use different operating systems (e.g., Unix, Windows NT, Mac OS, etc.), and can be relational databases (such as Mysql, PostgreSql, etc.), non-relational databases (such as MongoDB, Redis, etc.) or data warehouses (such as hive, StarRocks).

[0066] Data synchronization: refers to the real-time or periodic updating and matching of data in two or more different locations or systems to maintain data consistency. Data synchronization is critical in many scenarios, such as backup, disaster recovery, distributed databases, multi-data center operations, etc.

[0067] Secondly, the implementation environment of this application is introduced. Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application, the implementation environment includes: a data synchronization sending end 110, a management end 120, an intermediate storage server 130, and a data synchronization receiving end 140.

[0068] Among them, the data synchronization sending end 110 is a device that provides hardware support for the data synchronization sending platform; the management end 120 is a device that provides hardware support for the management platform; the intermediate storage server 130 is a device that provides hardware support for the intermediate storage end; the data synchronization receiving end 140 is a device that provides hardware support for the data synchronization receiving platform.

[0069] Illustratively, the data synchronization sending platform is a platform that provides the first data to be synchronized, the management platform is a platform that manages the task status of the data synchronization task, the intermediate storage end is a storage end that provides data transit storage for the first data, and the data synchronization receiving platform is a platform that performs data synchronization on the first data, that is, it is the synchronization target of the first data when completing the data synchronization task.

[0070] In one example, the data synchronization sending end 110 stores the first data to the intermediate storage server 130, and continuously detects the task status of the data synchronization task recorded in the management end 120 after the management end 120 sends a data synchronization task notification. The data synchronization sending end 110 sends a data synchronization request to the data synchronization receiving end 140. After receiving the data synchronization request, the data synchronization receiving end 140 reads the first data from the intermediate storage server 130 and synchronizes the first data. When the first data synchronization is completed, a first status update message is sent to the management end 120 to update the task status of the data synchronization task recorded in the management end 120 to a completed status.

[0071] Optionally, the data synchronization sending end 110 management end 120 and the data synchronization receiving end 140 can be implemented as a terminal, or as a server, or as a device group obtained by combining a terminal and a server. The figure only takes the implementation as a server as an example for illustration, and does not impose specific restrictions on its hardware implementation.

[0072] Optionally, the terminal may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a device equipped with an embedded system (eg, a smart home, smart traffic management equipment, etc.), which is not limited in the embodiments of the present application.

[0073] Optionally, the above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud security, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0074] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0075] In some embodiments, when there is isolation between the data synchronization sending end 110 and the management end 120, or when there is isolation between the management end 120 and the data synchronization receiving end 140, the implementation environment further includes a control transfer end.

[0076] In one example, if Figure 2 As shown, when the management end 120 and the data synchronization receiving end 140 are facilities of the first enterprise, and the data synchronization sending end 110 is a facility of the second enterprise, there is security isolation between the data synchronization sending end 110 and the management end 120, and between the data synchronization sending end 110 and the data synchronization receiving end 140. Therefore, a control transfer end 150 is set to transfer the instructions generated during the data synchronization process.

[0077] Combined with the above-mentioned noun introduction and implementation environment, the interactive process of the data synchronization system provided by this application is explained, please refer to Figure 3 , which shows an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application, wherein the system includes a management platform, an intermediate storage terminal and a data synchronization receiving platform, and the interactive process includes the following steps:

[0078] Step 301: The intermediate storage end obtains first data from a data synchronization sending platform.

[0079] In the embodiment of the present application, the first data is the data to be synchronized to the data synchronization receiving platform, that is, the task object of the data synchronization task currently executed by the data synchronization system is the first data.

[0080] The intermediate storage end is a storage device that provides temporary data storage for data synchronization tasks. Optionally, the intermediate storage end can be implemented as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud storage.

[0081] The data synchronization sending platform is the source platform of the first data. Optionally, the data synchronization sending platform can be a platform belonging to the same network environment as the data synchronization receiving platform. In one example, the data synchronization sending platform is implemented as the enterprise intranet platform of the first enterprise, and the data synchronization receiving platform is implemented as the enterprise public cloud platform of the first enterprise; or, the data synchronization sending platform can be a network environment different from that of the data synchronization receiving platform. In one example, the data synchronization sending platform is implemented as the enterprise network platform of the second enterprise, and the data synchronization receiving platform is implemented as the enterprise public cloud platform of the first enterprise.

[0082] In some embodiments, when the data synchronization sending platform detects that the task start condition of the data synchronization task is met, the first data is sent to the intermediate storage end.

[0083] Optionally, the task start condition can be implemented as at least one of the following:

[0084] 1. The current time is the preset data synchronization time point;

[0085] 2. The time difference between the current time and the last execution of the data synchronization task satisfies the data synchronization frequency corresponding to the data synchronization task;

[0086] 3. The first data load of the data synchronization sending platform reaches the first load threshold;

[0087] 4. The second data load of the data synchronization receiving platform is lower than the second load threshold;

[0088] 5. The data volume difference between the first platform data stored on the data synchronization sending platform and the second platform data stored on the data synchronization receiving platform reaches a preset threshold.

[0089] Step 302: The management platform obtains a data synchronization task corresponding to the first data, and the task status of the data synchronization task is in a pending synchronization status.

[0090] In some embodiments, a data synchronization task for synchronizing data stored on the data synchronization sending platform and data stored on the data synchronization receiving platform is preconfigured in the management platform.

[0091] In some embodiments, the management platform provides a visual interface to the user for creating a new data synchronization task. Schematically, a terminal equipped with the management platform displays a task creation interface, which is used to create a data processing task, wherein the task creation interface includes multiple task type candidates, and the multiple task type candidates include a data synchronization type; a first selection operation for a data source type is received, and a second selection operation for a data target type is received, wherein the data source type is used to indicate the data type of the first data, and the data target type is used to indicate the data type of the first data after the data processing task is completed; in the case of receiving a selection of a data synchronization type from multiple task type candidates, a data synchronization sending platform is determined based on the data source type, and a data synchronization receiving platform is determined based on the data target type; a data synchronization task is created based on the data synchronization sending platform and the data synchronization receiving platform.

[0092] In other embodiments, the data synchronization sending platform configures the data synchronization task to the management platform. Schematically, the data synchronization sending platform creates a data synchronization task corresponding to the first data, and sends the task information corresponding to the data synchronization task to the management platform, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type of the first data after synchronization is completed, and the task start condition of the data synchronization task. Alternatively, the data synchronization receiving platform configures the data synchronization task to the management platform. Schematically, the data synchronization receiving platform creates a data synchronization task corresponding to the first data, and sends the task information corresponding to the data synchronization task to the management platform.

[0093] Optionally, the data synchronization task can be implemented as a single execution task or a cyclic execution task.

[0094] First, when the data synchronization task is implemented as a single execution task, the management platform automatically records the task status of the data synchronization task as a pending synchronization status after obtaining the data synchronization task.

[0095] Second, when the data synchronization task is a cyclically executed task, the management platform receives a data synchronization task notification corresponding to the data synchronization task, and sets the task state of the managed data synchronization task to a pending synchronization state according to the data synchronization task notification.

[0096] In some embodiments, the data synchronization system also includes a first data platform. Schematically, the first data platform obtains a data synchronization task notification corresponding to the first data, wherein the data synchronization task notification is used to indicate that the task status of the data synchronization task is in a pending synchronization state, and the first data platform sends the data synchronization task notification to the management platform; when the management platform receives the data synchronization task notification sent by the first data platform, it records that the task status of the data synchronization task is in a pending synchronization state.

[0097] In one example, when a data synchronization task has been configured in the management platform and the data synchronization task notification is only used to update the task status of the data synchronization task, the data synchronization task notification includes a task identifier corresponding to the data synchronization task. After receiving the data synchronization task notification, the management platform determines the data synchronization task according to the task identifier indicated in the data synchronization task notification and switches the task status of the data synchronization task to a pending synchronization state. In one example, before receiving the data synchronization task notification, the task status corresponding to the data synchronization task in the management platform can be empty or in an unstarted state.

[0098] In some embodiments, the data synchronization task configured in the management platform corresponds to a data synchronization frequency, and the data synchronization frequency is used to indicate the frequency of the cyclic execution of the cyclically executed task. In an illustrative embodiment, the management platform obtains the data synchronization frequency of the data synchronization task, and the management platform sends a task deployment instruction including the data synchronization frequency to the first data platform; when the first data platform receives the task deployment instruction, it generates and sends a data synchronization task notification to the management platform based on the data synchronization frequency.

[0099] In another example, when the data synchronization task is not configured in the management platform, the data synchronization task notification is used to indicate the task information of the data synchronization task to the management platform and to set the task status of the data synchronization task, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type after the first data is synchronized, and the task start condition of the data synchronization task. Schematically, after receiving the data synchronization task notification, the management platform records the data synchronization task according to the task information indicated in the data synchronization task notification, and sets the task status of the data synchronization task to the pending synchronization status.

[0100] Optionally, the above-mentioned first data platform can be implemented as a data synchronization sending platform, that is, the first data platform is the data provider of the first data; or, the above-mentioned first data platform can also be implemented as a transit platform between the data synchronization sending platform and the management platform / data synchronization receiving platform, that is, the data synchronization sending platform generates a data synchronization task notification, sends the data synchronization task notification to the first data platform, and the first data platform sends the data synchronization task notification to the management platform.

[0101] Step 303: The data synchronization receiving platform receives the data synchronization request.

[0102] In the embodiment of the present application, the data synchronization receiving platform is a platform responsible for receiving the first data and performing data synchronization processing on the first data in the data synchronization task.

[0103] The data synchronization request is sent after the data synchronization task is started.

[0104] Optionally, the data synchronization request may be a request sent by the management platform to the data synchronization receiving platform, that is, the management platform sends the data synchronization request to the data synchronization receiving platform; or, the data synchronization request may be a request sent by the data synchronization sending platform to the data synchronization receiving platform, that is, the data synchronization sending platform sends the data synchronization request to the data synchronization receiving platform; or, the data synchronization request may be a request sent by the transit platform to the data synchronization receiving platform.

[0105] In some embodiments, the data synchronization system further includes a first data platform, and the first data platform sends a data synchronization request to the data synchronization receiving platform when the task start condition of the data synchronization task is met. Optionally, the first data platform can be implemented as a data synchronization sending platform, that is, the first data platform is the data provider of the first data; or, the first data platform can also be implemented as a transfer platform between the data synchronization sending platform and the management platform / data synchronization receiving platform.

[0106] Step 304: In response to the data synchronization request, the data synchronization receiving platform reads the first data from the intermediate storage end.

[0107] In an embodiment of the present application, the data synchronization request is used to instruct the data synchronization receiving platform to perform data synchronization. Then, after receiving the data synchronization request, the data synchronization receiving platform reads the first data corresponding to the data synchronization task from the intermediate storage end.

[0108] In some embodiments, the data synchronization request includes a data identifier of the first data. After the data synchronization receiving platform parses the data identifier from the data synchronization request, it reads the first data from the intermediate storage end according to the data identifier and performs data synchronization on the first data.

[0109] Step 305: the data synchronization receiving platform performs data synchronization processing on the first data.

[0110] In the embodiment of the present application, after reading the first data from the intermediate storage end, the data synchronization receiving platform performs data synchronization processing on the first data.

[0111] Optionally, the data synchronization process includes at least one of the following operations:

[0112] 1. Storage operation:

[0113] Illustratively, the data synchronization receiving platform stores the first data in a storage area in the data synchronization receiving platform. In one example, the data synchronization receiving platform includes a second storage unit, and the data synchronization receiving platform stores the first data in the second storage unit.

[0114] 2. Verification operation:

[0115] Illustratively, the data synchronization receiving platform performs data verification on the first data to determine whether the first data meets the verification condition. Optionally, the data verification includes at least one of a consistency check on the first data before and after synchronization and an integrity check on the read first data.

[0116] 3. Conversion operation:

[0117] Illustratively, the data synchronization receiving platform performs data conversion on the first data to obtain the second data. Optionally, the data conversion includes at least one of converting the data format of the first data, converting the data type of the first data, and converting the storage method of the first data.

[0118] 4. Update operation:

[0119] Schematically, the data synchronization receiving platform updates the original data stored in the data synchronization receiving platform based on the first data. Exemplarily, the data difference between the first data and the original data is compared, and the first difference data in the first data that is different from the original data is stored, or the first difference data is used to replace the second difference data in the original data that is different from the first data.

[0120] 5. Logging operations:

[0121] Illustratively, the data synchronization receiving platform updates the data storage log in the data synchronization receiving platform based on the read first data, and records at least one of the following information: the amount of data synchronized with the first data, the synchronization time, the number of synchronizations, and the synchronization success rate.

[0122] 6. Permission setting operation:

[0123] Illustratively, the data synchronization receiving platform sets data permissions for the first data read, wherein the data permissions include at least one of permissions to read the first data from the data synchronization receiving platform, permissions to update the first data stored by the data synchronization receiving platform, permissions to delete the first data stored by the data synchronization receiving platform, and permissions to modify the first data stored by the data synchronization receiving platform.

[0124] Step 306: When the first data synchronization is completed, the data synchronization receiving platform sends a first status update message to the management platform.

[0125] In an embodiment of the present application, when the data synchronization receiving platform determines that the data synchronization processing performed on the first data has been successfully executed, it determines that the synchronization of the first data is completed, generates a first status update message, and sends the first status update message to the management platform. The first status update message is used to indicate to the management platform that the data synchronization task of the first data has been completed.

[0126] Step 307: After receiving the first status update message, the management platform switches the task status of the data synchronization task to a completed status.

[0127] In an embodiment of the present application, the task status of the data synchronization task managed in the management platform is used to keep the task status of the data synchronization sending platform and the data synchronization receiving platform consistent when executing the data synchronization task.

[0128] In some embodiments, the data synchronization system further includes a first data platform, which is a data synchronization sending platform, or a transit platform between the data synchronization sending platform and the management platform / data synchronization receiving platform.

[0129] In an illustrative manner, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform, and determines that the data synchronization task is completed when the task status is detected to be a success status. That is, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform to ensure the consistency of the task status of the data synchronization task at both ends during the data synchronization process.

[0130] In some embodiments, when the first data platform is a data synchronization sending platform, when the first data platform detects that the task status is a success status during the cyclic detection process, the first data platform determines that the data synchronization task is completed and releases the computing resources and memory resources used to execute the data synchronization task.

[0131] In other embodiments, when the first data platform is a transit platform between the data synchronization sending platform and the management platform / data synchronization receiving platform, when the first data platform detects that the task status is a success status during the cyclic detection process, the first data platform generates a task completion instruction and sends the task completion instruction to the data synchronization sending platform.

[0132] In other embodiments, the data synchronization receiving platform will re-synchronize the data when data synchronization fails. Schematically, when the data synchronization receiving platform determines that the first data synchronization has failed, it clears the acquired first data, re-reads the first data from the intermediate storage end, and performs data synchronization processing on the re-read first data.

[0133] In some embodiments, when the data synchronization receiving platform fails multiple times during data synchronization, a data synchronization failure notification is generated through the management platform to notify manual intervention. In an exemplary embodiment, when the number of first data synchronization failures reaches a first threshold, the data synchronization receiving platform sends a second status update message to the management platform; when the management platform receives the second status update message, it switches the task status of the data synchronization task to a failed state and generates a data synchronization failure notification, wherein the data synchronization failure notification is used to notify manual intervention in the data synchronization task.

[0134] In some embodiments, when the first data platform detects that the task status is a failure status, it stops the cyclic detection of the task status.

[0135] In some embodiments, the management platform records the information generated during the data synchronization of the first data to form log data, so that users can analyze the log data to maintain the system and reduce fault analysis costs.

[0136] In summary, when realizing data synchronization between the data synchronization sending platform and the data synchronization receiving platform, the first data is temporarily stored through the intermediate storage end, and the task status of the data synchronization task of the first data during the data synchronization process is recorded and updated through the management platform, that is, after the data synchronization receiving platform completes the data synchronization of the first data, the data synchronization receiving platform updates the task status recorded in the management platform to the completion status through the first status update message, so that the data synchronization sending platform can know whether the data synchronization task of the first data is completed at any time through the management platform, so that the management platform coordinates the data synchronization sending platform and the data synchronization receiving platform, so that the data synchronization process of the first data keeps the task status consistent in the cross-platform environment, so that in the cross-platform data synchronization process, by unifying the information cognition of the task status between different platforms, the data synchronization efficiency of the data during cross-platform synchronization is guaranteed. In addition, by uniformly managing the status of the data synchronization task on the management platform, different data platforms associated with the data synchronization task (such as the data synchronization sending platform and the data synchronization receiving platform) can always maintain a unified status during the data synchronization process, avoiding the problem of data synchronization errors or failures caused by the asynchronous status between different platforms.

[0137] At the same time, the intermediate storage end, as an intermediate storage of data, opens up the data transmission channel between different platforms. Since the management platform is set as the intermediate storage of the task status, after the data synchronization sending platform sends the first data in the data synchronization task to the intermediate storage end, the data synchronization sending platform only needs to pay attention to the task status in the management platform. Even if the data synchronization receiving platform fails to synchronize, it can also clear the data and re-acquire the first data from the intermediate storage end for data synchronization. Different platforms no longer need to re-acquire the first data through retransmission instructions after data synchronization fails, which reduces the data interaction requirements between platforms and avoids the repeated cross-platform transmission of the first data when it is heterogeneous data. It avoids the high failure rate of heterogeneous data when it is transmitted across platforms, and improves the data synchronization efficiency when the data is synchronized across platforms.

[0138] In some embodiments, the data synchronization sending platform and the data synchronization receiving platform can be implemented as platforms in the same enterprise network environment, or can be implemented as platforms in different enterprise network environments.

[0139] The first is that the data synchronization sending platform and the data synchronization receiving platform are implemented as platforms within the same enterprise network environment:

[0140] Schematically, the data synchronization system also includes a first data platform. In the embodiment of the present application, the first data platform is taken as a data synchronization sending platform for the first data as an example. The first data platform includes a first data synchronization module and a first storage unit. Please refer to Figure 4 , which shows an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application, the interactive process includes the following steps:

[0141] Step 401: A first data synchronization module reads first data from a first storage unit.

[0142] In the embodiment of the present application, the first data platform is a data synchronization sending platform for the first data, and therefore, the first data is stored in the first storage unit of the first data platform.

[0143] In an embodiment of the present application, the first data synchronization module in the first data platform is used to provide functional support for data synchronization.

[0144] In some embodiments, when the first data platform detects that a task start condition of a data synchronization task is met, the first data is read from the first storage unit.

[0145] Optionally, the task start condition can be implemented as at least one of the following:

[0146] 1. The current time is the preset data synchronization time point;

[0147] 2. The time difference between the current time and the last execution of the data synchronization task satisfies the data synchronization frequency corresponding to the data synchronization task;

[0148] 3. The first data load of the first data platform reaches a first load threshold;

[0149] 4. The second data load of the data synchronization receiving platform is lower than the second load threshold;

[0150] 5. The data volume difference between the first platform data stored on the first data platform and the second platform data stored on the data synchronization receiving platform reaches a preset threshold.

[0151] Step 402: The first data synchronization module writes the first data into the intermediate storage end.

[0152] The intermediate storage end is a storage device that provides temporary data storage for data synchronization tasks. Optionally, the intermediate storage end can be implemented as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud storage.

[0153] In one example, the intermediate storage end can be implemented as a Cloud Object Storage (COS) cloud service end, where the COS cloud service is a distributed storage service for storing massive files provided by a cloud server, with the advantages of high scalability, low cost, reliability and security. Through a variety of methods such as consoles, APIs, software development kits (SDKs) and tools, users can simply and quickly access COS to upload, download and manage files in multiple formats, and realize massive data storage and management.

[0154] Step 403: The first data synchronization module generates a data synchronization task notification corresponding to the first data.

[0155] The data synchronization task notification is used to indicate that the task status of the data synchronization task is in a pending synchronization state.

[0156] In some embodiments, the first data synchronization module stores multiple candidate data synchronization tasks and continuously detects whether the task start conditions of each candidate data synchronization task are met. When it is determined that the target data synchronization task among the multiple candidate data synchronization tasks meets its corresponding task start conditions, the first data synchronization module determines the data to be synchronized (first data) corresponding to the target data synchronization task and generates a data synchronization task notification corresponding to the first data.

[0157] In one example, when a data synchronization task has been configured in the management platform and the data synchronization task notification is only used to update the task status of the data synchronization task, the data synchronization task notification generated by the first data synchronization module includes the task identifier and task status corresponding to the data synchronization task.

[0158] In another example, when the data synchronization task is not configured in the management platform, the data synchronization task notification is used to indicate the task information of the data synchronization task to the management platform and set the task status of the data synchronization task, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type after the synchronization of the first data is completed, and the task start condition of the data synchronization task. That is, the data synchronization task notification generated by the first data synchronization module includes the task information and task status corresponding to the data synchronization task.

[0159] Step 404: The first data synchronization module sends a data synchronization task notification to the management platform.

[0160] Step 405: upon receiving the data synchronization task notification, the management platform stores the task status of the data synchronization task as a pending synchronization status.

[0161] In an embodiment of the present application, the first data synchronization module in the first data platform sends a data synchronization task notification to the management platform to indicate the task status of the data synchronization task by sending the data synchronization task notification.

[0162] In some embodiments, after receiving a data synchronization task notification, the management platform stores the task status of the data synchronization task as a pending synchronization status according to the data synchronization task notification, and sends a first feedback message to the first data synchronization module. The first feedback message is used to indicate that the management platform has successfully received the data synchronization task notification and has updated the task status of the data synchronization task.

[0163] In one example, when a data synchronization task has been configured in the management platform and the data synchronization task notification is only used to update the task status of the data synchronization task, the data synchronization task notification includes a task identifier corresponding to the data synchronization task. After receiving the data synchronization task notification, the management platform determines the data synchronization task according to the task identifier indicated in the data synchronization task notification and switches the task status of the data synchronization task to a pending synchronization state. In one example, before receiving the data synchronization task notification, the task status corresponding to the data synchronization task in the management platform can be empty or in an unstarted state.

[0164] In another example, when the data synchronization task is not configured in the management platform, the data synchronization task notification is used to indicate the task information of the data synchronization task to the management platform and to set the task status of the data synchronization task, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type after the first data is synchronized, and the task start condition of the data synchronization task. Schematically, after receiving the data synchronization task notification, the management platform records the data synchronization task according to the task information indicated in the data synchronization task notification, and sets the task status of the data synchronization task to the pending synchronization status.

[0165] Step 406: The first data synchronization module cyclically detects the task status of the data synchronization tasks stored in the management platform.

[0166] In some embodiments, after sending the data synchronization task notification, the first data synchronization module starts a cyclic detection of the task status of the data synchronization task stored in the management platform.

[0167] In other embodiments, after sending a data synchronization task notification and receiving a feedback message fed back by the management platform, the first data synchronization module starts a cyclic detection of the task status of the data synchronization task stored in the management platform.

[0168] Step 407: The first data synchronization module sends a data synchronization request to the data synchronization receiving platform.

[0169] In an embodiment of the present application, a first data synchronization module of a first data platform sends a data synchronization request to a data synchronization receiving platform, where the data synchronization request is used to instruct the data synchronization receiving platform to perform data synchronization of the first data.

[0170] The data synchronization request is sent after the data synchronization task is started.

[0171] Step 408: upon receiving the data synchronization request, the data synchronization receiving platform reads the first data from the intermediate storage end.

[0172] In some embodiments, the data synchronization request includes a data identifier of the first data. After the data synchronization receiving platform parses the data identifier from the data synchronization request, it reads the first data from the intermediate storage end according to the data identifier and performs data synchronization on the first data.

[0173] Step 409: the data synchronization receiving platform performs data synchronization processing on the first data.

[0174] In the embodiment of the present application, after reading the first data from the intermediate storage end, the data synchronization receiving platform performs data synchronization on the first data.

[0175] Optionally, the data synchronization process includes at least one of the following operations:

[0176] 1. Storage operation:

[0177] Illustratively, the data synchronization receiving platform stores the first data in a storage area in the data synchronization receiving platform. In one example, the data synchronization receiving platform includes a second storage unit, and the data synchronization receiving platform stores the first data in the second storage unit.

[0178] 2. Verification operation:

[0179] Illustratively, the data synchronization receiving platform performs data verification on the first data to determine whether the first data meets the verification condition. Optionally, the data verification includes at least one of a consistency check on the first data before and after synchronization and an integrity check on the read first data.

[0180] 3. Conversion operation:

[0181] Illustratively, the data synchronization receiving platform performs data conversion on the first data to obtain the second data. Optionally, the data conversion includes at least one of converting the data format of the first data, converting the data type of the first data, and converting the storage method of the first data.

[0182] 4. Update operation:

[0183] Schematically, the data synchronization receiving platform updates the original data stored in the data synchronization receiving platform based on the first data. Exemplarily, the data difference between the first data and the original data is compared, and the first difference data in the first data that is different from the original data is stored, or the first difference data is used to replace the second difference data in the original data that is different from the first data.

[0184] 5. Logging operations:

[0185] Illustratively, the data synchronization receiving platform updates the data storage log in the data synchronization receiving platform based on the read first data, and records at least one of the following information: the amount of data synchronized with the first data, the synchronization time, the number of synchronizations, and the synchronization success rate.

[0186] 6. Permission setting operation:

[0187] Illustratively, the data synchronization receiving platform sets data permissions for the first data read, wherein the data permissions include at least one of permissions to read the first data from the data synchronization receiving platform, permissions to update the first data stored by the data synchronization receiving platform, permissions to delete the first data stored by the data synchronization receiving platform, and permissions to modify the first data stored by the data synchronization receiving platform.

[0188] In some embodiments, the data synchronization receiving platform also includes a second data synchronization module and a second storage unit. The second data synchronization module is used to perform data verification on the read first data, and write the first data into the second storage unit if the first data meets the verification condition.

[0189] In an illustrative manner, the second data synchronization module obtains the first attribute information of the first data before data synchronization; in the case of reading the first data from the intermediate storage end, the second attribute information corresponding to the first data is detected; in the case of matching the first attribute information and the second attribute information, the first data is written into the second storage unit; in the case of determining that the first data has been written into the second storage unit, it is determined that the synchronization of the first data is completed. That is, the second data synchronization module verifies the first data by comparing the difference between the data attribute information of the first data before data synchronization and the data attribute information of the first data read from the intermediate storage end.

[0190] Optionally, the first attribute information includes at least one of the following:

[0191] 1. The amount of the first data, that is, comparing whether the amount of the first data read from the intermediate storage end by the second data synchronization module is the same as the amount of data written to the intermediate storage end by the first data synchronization module in the first data platform;

[0192] 2. The data type of the first data, that is, comparing whether the data type of the first data read by the second data synchronization module from the intermediate storage end is the same as the data type written to the intermediate storage end by the first data synchronization module in the first data platform;

[0193] 3. The data format of the first data, that is, comparing whether the data format of the first data read by the second data synchronization module from the intermediate storage end is the same as the data format written to the intermediate storage end by the first data synchronization module in the first data platform;

[0194] 4. Index information of the first data, that is, comparing whether the index information of the first data read from the intermediate storage end by the second data synchronization module is the same as the index information written into the intermediate storage end by the first data synchronization module in the first data platform;

[0195] 5. The hash value of the first data, that is, after reading the first data, the second data synchronization module inputs the first data into a preset hash function to obtain a second hash value. Before writing the first data into the intermediate storage end, the first data synchronization module in the first data platform inputs the first data into a preset hash function to obtain the first hash value. The second data synchronization module compares the first hash value and the second hash value to see if they are the same.

[0196] In some embodiments, the data synchronization task notification sent by the first data platform to the management platform includes the above-mentioned first attribute information, that is, the management platform obtains the data synchronization task notification corresponding to the first data, and the data synchronization task notification includes the first attribute information; the management platform sends the first attribute information to the second data synchronization module based on the data synchronization task notification.

[0197] In other embodiments, the data synchronization request sent by the first data platform to the second data synchronization module includes the above-mentioned first attribute information.

[0198] In some embodiments, the data stored in the first storage unit of the first data platform and the data stored in the second storage unit of the data synchronization receiving platform are heterogeneous data, that is, there is a difference in at least one of the data format, data type, hardware medium, and storage system between the stored data, then the second data synchronization module needs to perform data conversion on the first data after reading the first data. Schematically, when the first data and the data stored in the second storage unit are heterogeneous data, the second data synchronization module performs data conversion processing on the first data to obtain the second data, writes the second data into the second storage unit, and determines that the synchronization of the first data is completed when it is determined that the second data has been written into the second storage unit.

[0199] Step 410: When the first data synchronization is completed, the data synchronization receiving platform sends a first status update message to the management platform.

[0200] In an embodiment of the present application, when the data synchronization receiving platform determines that the data synchronization processing performed on the first data has been successfully executed, it determines that the synchronization of the first data is completed, generates a first status update message, and sends the first status update message to the management platform. The first status update message is used to indicate to the management platform that the data synchronization task of the first data has been completed.

[0201] In one example, after writing the first data / the second data into the second storage unit, the second data synchronization module generates a first status update message and sends the first status update message to the management platform.

[0202] Step 411: After receiving the first status update message, the management platform switches the task status of the data synchronization task to the completed status.

[0203] In an embodiment of the present application, the task status of the data synchronization task managed in the management platform is used to keep the task status of the data synchronization sending platform and the data synchronization receiving platform consistent when executing the data synchronization task.

[0204] Step 412: When the first data synchronization module detects that the task status is a success status, it determines that the data synchronization task is completed.

[0205] In an illustrative manner, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform, and determines that the data synchronization task is completed when the task status is detected to be a success status. That is, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform to ensure the consistency of the task status of the data synchronization task at both ends during the data synchronization process.

[0206] In some embodiments, when the first data platform detects that the task status is a success status during the cyclic detection process, the first data platform determines that the data synchronization task is completed and releases the computing resources and memory resources used when executing the data synchronization task.

[0207] For example, please refer to Figure 5 , which shows a schematic diagram of a data synchronization system 500 provided by an exemplary embodiment of the present application. Taking the data synchronization system 500 providing data synchronization functions for different platforms in a certain enterprise network environment as an example, the enterprise network environment includes an enterprise intranet platform 510 and an enterprise public cloud platform 520. The enterprise intranet platform 510 includes a first central data synchronization module 511 and an offline table storage unit 512, and the enterprise public cloud platform 520 includes a second central data synchronization module 521 and a hive storage unit 522. The data synchronization system 500 also includes an object storage cloud service 530 and a management platform 540.

[0208] Specifically, taking the target service provided by the enterprise to synchronize account data from the enterprise intranet platform 510 to the enterprise public cloud platform 520 as an example, the information processing flow of the data synchronization system 500 is as follows:

[0209] Step 1: The target business party collects user-authorized compliance account data through information collection points such as clients, web pages, and applets and stores the data in the offline table storage unit 512 of the enterprise intranet platform 510 .

[0210] Step 2, the first central data synchronization module 511 deployed by the management platform 540 on the enterprise intranet platform 510 periodically reads the data to be synchronized in the offline table storage unit 512 of the corresponding partition, and writes the data to be synchronized into the object storage cloud service 530, wherein the object storage cloud service 530 allows cross-platform access through the https protocol of a secure channel.

[0211] Step 3: The first central data synchronization module 511 abstracts the current synchronization task into a task notification, records the offline table partition data volume, marks the task notification status as newly created, and registers the notification with the management platform 540 .

[0212] Step 4: the first central data synchronization module 511 starts to cyclically detect whether the task notification status is switched from new creation to other statuses.

[0213] Step 5: The first central data synchronization module 511 initiates a data rerun or re-recording request to the second central data synchronization module 521 of the enterprise public cloud platform 520 .

[0214] Step 6: The second central data synchronization module 521 receives the request and reads the data to be synchronized in the object storage cloud service 530.

[0215] Step 7: The second central data synchronization module 521 verifies the consistency of the data according to the task notification details.

[0216] Step 8: If the data verification is successful, the second central data synchronization module 521 writes the data into the hive storage unit 522 and updates the task status to the management platform 540 as successful.

[0217] Step 9 (not shown in the figure), if the data verification fails, the second central data synchronization module 521 cleans the data and tries again; if the retry is successful, the task status is updated to success; if the number of retries reaches the upper limit and the verification still fails, the task status is updated to failure, and the management platform 540 initiates a system notification to notify the business party and platform administrator to intervene manually.

[0218] To summarize, when realizing data synchronization of the first data between the data synchronization sending platform and the data synchronization receiving platform, the first data is temporarily stored through the intermediate storage end, and the task status of the data synchronization task of the first data during the data synchronization process is recorded and updated through the management platform, that is, after the data synchronization receiving platform completes the data synchronization of the first data, the data synchronization receiving platform updates the task status recorded in the management platform to a completed status through a first status update message, so that the data synchronization sending platform can know whether the data synchronization task of the first data is completed at any time through the management platform, thereby coordinating the data synchronization sending platform and the data synchronization receiving platform through the management platform, so that the data synchronization process of the first data maintains a consistent task status in a cross-platform environment, so that in the cross-platform data synchronization process, the unified message notification mechanism implemented by the management platform is used to unify the information cognition of the task status between different platforms, coordinate and control the task status flow between multiple platforms, and ensure the data synchronization efficiency when the data is synchronized across platforms. At the same time, the intermediate storage end, as an intermediate storage of data, opens up the data transmission channel between different platforms. Since the management platform is set as the intermediate storage of the task status, after the data synchronization sending platform sends the first data in the data synchronization task to the intermediate storage end, the data synchronization sending platform only needs to pay attention to the task status in the management platform. Even if the data synchronization receiving platform fails to synchronize, it can also clear the data and re-acquire the first data from the intermediate storage end for data synchronization. Different platforms no longer need to re-acquire the first data through retransmission instructions after data synchronization fails, which reduces the data interaction requirements between platforms and avoids the repeated cross-platform transmission of the first data when it is heterogeneous data. It avoids the high failure rate of heterogeneous data when it is transmitted across platforms, and improves the data synchronization efficiency when the data is synchronized across platforms.

[0219] Furthermore, in an embodiment of the present application, the first data synchronization module and the second data synchronization module are modules deployed on different platforms as management platforms to assist in data synchronization, thereby opening up the information and data transmission channels between multiple platforms and reducing the processing difficulty during data synchronization. At the same time, the beginning and end of the data transmission channel of the first data synchronization module-management platform / intermediate storage end-second data synchronization module can also be integrated into a standardized custom plug-in access module, so that the above-mentioned data transmission channel can connect any two data platforms, so that the business party can access and perform secondary development of the data transmission channel according to the characteristics of the business data, thereby accessing the first data synchronization module and the second data synchronization module deployed by the management platform, and realizing more and richer heterogeneous data transmission capabilities.

[0220] The second type is that the data synchronization sending platform and the data synchronization receiving platform are implemented as platforms in different enterprise network environments:

[0221] Schematically, the data synchronization system also includes a first data platform and a second data platform. In the embodiment of the present application, the second data platform is taken as an example of a data synchronization sending platform for the first data. The first data platform includes a first data synchronization module. Please refer to Figure 6 , which shows an interactive flow chart of a data synchronization system provided by an exemplary embodiment of the present application, the interactive process includes the following steps:

[0222] Step 601: The second data platform writes the first data into an intermediate storage end.

[0223] In an embodiment of the present application, the second data platform is a data synchronization sending platform for the first data. In some embodiments, the second data platform includes a control node and a third storage unit. Schematically, the computing node reads the first data from the third storage unit, and the computing node writes the first data to the intermediate storage end.

[0224] The intermediate storage end is a storage device that provides temporary data storage for data synchronization tasks. Optionally, the intermediate storage end can be implemented as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud storage.

[0225] In one example, the intermediate storage end can be implemented as a Cloud Object Storage (COS) cloud service end, where the COS cloud service is a distributed storage service for storing massive files provided by a cloud server, with the advantages of high scalability, low cost, reliability and security. Through a variety of methods such as consoles, APIs, software development kits (SDKs) and tools, users can simply and quickly access COS to upload, download and manage files in multiple formats, and realize massive data storage and management.

[0226] In some embodiments, when the computing node detects that a task start condition of the data synchronization task is met, the computing node reads the first data from the third storage unit.

[0227] Optionally, the task start condition can be implemented as at least one of the following:

[0228] 1. The current time is the preset data synchronization time point;

[0229] 2. The time difference between the current time and the last execution of the data synchronization task satisfies the data synchronization frequency corresponding to the data synchronization task;

[0230] 3. The first data load of the second data platform reaches a first load threshold;

[0231] 4. The second data load of the data synchronization receiving platform is lower than the second load threshold;

[0232] 5. The data volume difference between the first platform data stored on the second data platform and the second platform data stored on the data synchronization receiving platform reaches a preset threshold.

[0233] Step 602: The second data platform generates a data synchronization task notification corresponding to the first data.

[0234] The data synchronization task notification is used to indicate that the task status of the data synchronization task is in a pending synchronization state.

[0235] In some embodiments, the second data platform stores multiple candidate data synchronization tasks and continuously detects whether the task start conditions of each candidate data synchronization task are met. When it is determined that the target data synchronization task among the multiple candidate data synchronization tasks meets its corresponding task start conditions, the second data platform determines the data to be synchronized (first data) corresponding to the target data synchronization task, and the data synchronization receiving platform that serves as the data recipient in the data synchronization task, and generates a data synchronization task notification corresponding to the first data.

[0236] In one example, when a data synchronization task has been configured in the management platform and the data synchronization task notification is only used to update the task status of the data synchronization task, the data synchronization task notification generated by the second data platform includes the task identifier and task status corresponding to the data synchronization task.

[0237] In another example, when the data synchronization task is not configured in the management platform, the data synchronization task notification is used to indicate the task information of the data synchronization task to the management platform and set the task status of the data synchronization task, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type after the synchronization of the first data is completed, and the task start condition of the data synchronization task. That is, the data synchronization task notification generated by the second data platform includes the task information and task status corresponding to the data synchronization task.

[0238] Step 603: The second data platform sends a data synchronization task notification to the first data synchronization module.

[0239] In some embodiments, the second data platform determines the data synchronization receiving platform as the data recipient based on the target data synchronization task determined from multiple candidate data synchronization tasks. After determining the data synchronization receiving platform, it determines whether it is possible to directly communicate messages with the data synchronization receiving platform. If it is detected that it is impossible to directly communicate messages with the data synchronization receiving platform, it is determined that the transit platform in the enterprise network environment where the data synchronization receiving platform is located is the first data platform, and a data synchronization task notification is sent to the first data synchronization module in the first data platform.

[0240] In some embodiments, the first data synchronization module in the first data platform and the second data platform implement message communication through an external dependent module (eg, a gateway) in the first data platform.

[0241] Step 604: The first data synchronization module forwards the received data synchronization task notification to the management platform.

[0242] Step 605: upon receiving the data synchronization task notification, the management platform stores the task status of the data synchronization task as a pending synchronization status.

[0243] In an embodiment of the present application, the first data synchronization module in the first data platform sends a data synchronization task notification to the management platform to indicate the task status of the data synchronization task by sending the data synchronization task notification.

[0244] In some embodiments, after receiving a data synchronization task notification, the management platform stores the task status of the data synchronization task as a pending synchronization status according to the data synchronization task notification, and sends a first feedback message to the first data synchronization module. The first feedback message is used to indicate that the management platform has successfully received the data synchronization task notification and has updated the task status of the data synchronization task.

[0245] In one example, when a data synchronization task has been configured in the management platform and the data synchronization task notification is only used to update the task status of the data synchronization task, the data synchronization task notification includes a task identifier corresponding to the data synchronization task. After receiving the data synchronization task notification, the management platform determines the data synchronization task according to the task identifier indicated in the data synchronization task notification and switches the task status of the data synchronization task to a pending synchronization state. In one example, before receiving the data synchronization task notification, the task status corresponding to the data synchronization task in the management platform can be empty or in an unstarted state.

[0246] In another example, when the data synchronization task is not configured in the management platform, the data synchronization task notification is used to indicate the task information of the data synchronization task to the management platform and to set the task status of the data synchronization task, wherein the task information includes at least one of the platform identifier of the data synchronization sending platform, the platform identifier of the data synchronization receiving platform, the data type of the first data to be synchronized, the data type after the first data is synchronized, and the task start condition of the data synchronization task. Schematically, after receiving the data synchronization task notification, the management platform records the data synchronization task according to the task information indicated in the data synchronization task notification, and sets the task status of the data synchronization task to the pending synchronization status.

[0247] Step 606: The first data synchronization module cyclically detects the task status of the data synchronization tasks stored in the management platform.

[0248] In some embodiments, after sending the data synchronization task notification, the first data synchronization module starts a cyclic detection of the task status of the data synchronization task stored in the management platform.

[0249] In other embodiments, after sending a data synchronization task notification and receiving a feedback message fed back by the management platform, the first data synchronization module starts a cyclic detection of the task status of the data synchronization task stored in the management platform.

[0250] Step 607: The first data synchronization module sends a data synchronization request to the data synchronization receiving platform.

[0251] In an embodiment of the present application, a first data synchronization module of a first data platform sends a data synchronization request to a data synchronization receiving platform, where the data synchronization request is used to instruct the data synchronization receiving platform to perform data synchronization of the first data.

[0252] The data synchronization request is sent after the data synchronization task is started.

[0253] Step 608: Upon receiving the data synchronization request, the data synchronization receiving platform reads the first data from the intermediate storage end.

[0254] In some embodiments, the data synchronization request includes a data identifier of the first data. After the data synchronization receiving platform parses the data identifier from the data synchronization request, it reads the first data from the intermediate storage end according to the data identifier and performs data synchronization on the first data.

[0255] Step 609: the data synchronization receiving platform performs data synchronization processing on the first data.

[0256] In the embodiment of the present application, after reading the first data from the intermediate storage end, the data synchronization receiving platform performs data synchronization on the first data.

[0257] Optionally, the data synchronization process includes at least one of the following operations: a storage operation, a verification operation, a conversion operation, an update operation, a log recording operation, and a permission setting operation.

[0258] In some embodiments, the data synchronization receiving platform also includes a second data synchronization module and a second storage unit. The second data synchronization module is used to perform data verification on the read first data, and write the first data into the second storage unit if the first data meets the verification condition.

[0259] In an illustrative manner, the second data synchronization module obtains the first attribute information of the first data before data synchronization; in the case of reading the first data from the intermediate storage end, the second attribute information corresponding to the first data is detected; in the case of matching the first attribute information and the second attribute information, the first data is written into the second storage unit; in the case of determining that the first data has been written into the second storage unit, it is determined that the synchronization of the first data is completed. That is, the second data synchronization module verifies the first data by comparing the difference between the data attribute information of the first data before data synchronization and the data attribute information of the first data read from the intermediate storage end.

[0260] Optionally, the first attribute information includes at least one of the following: data volume of the first data, data type of the first data, data format of the first data, index information of the first data, and a hash value of the first data.

[0261] In some embodiments, the data synchronization task notification forwarded by the first data platform to the management platform includes the above-mentioned first attribute information, that is, the management platform obtains the data synchronization task notification corresponding to the first data, and the data synchronization task notification includes the first attribute information; the management platform sends the first attribute information to the second data synchronization module based on the data synchronization task notification.

[0262] In other embodiments, the data synchronization request sent by the first data platform to the second data synchronization module includes the above-mentioned first attribute information.

[0263] In some embodiments, the data stored in the third storage unit of the second data platform and the data stored in the second storage unit of the data synchronization receiving platform are heterogeneous data, that is, there is a difference in at least one of the data format, data type, hardware medium, and storage system between the stored data, then the second data synchronization module needs to perform data conversion on the first data after reading the first data. Schematically, when the first data and the data stored in the second storage unit are heterogeneous data, the second data synchronization module performs data conversion processing on the first data to obtain the second data, writes the second data into the second storage unit, and determines that the synchronization of the first data is completed when it is determined that the second data has been written into the second storage unit.

[0264] Step 610: When the first data synchronization is completed, the data synchronization receiving platform sends a first status update message to the management platform.

[0265] In an embodiment of the present application, when the data synchronization receiving platform determines that the data synchronization processing performed on the first data has been successfully executed, it determines that the synchronization of the first data is completed, generates a first status update message, and sends the first status update message to the management platform. The first status update message is used to indicate to the management platform that the data synchronization task of the first data has been completed.

[0266] In one example, after writing the first data / the second data into the second storage unit, the second data synchronization module generates a first status update message and sends the first status update message to the management platform.

[0267] Step 611: After receiving the first status update message, the management platform switches the task status of the data synchronization task to the completed status.

[0268] In an embodiment of the present application, the task status of the data synchronization task managed in the management platform is used to keep the task status of the data synchronization sending platform and the data synchronization receiving platform consistent when executing the data synchronization task.

[0269] Step 612: When the first data synchronization module detects that the task status is a success status, it determines that the data synchronization task is completed and sends a data synchronization success message to the second data platform.

[0270] In an illustrative manner, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform, and determines that the data synchronization task is completed when the task status is detected to be a success status. That is, the first data platform cyclically detects the task status of the data synchronization task stored in the management platform to ensure the consistency of the task status of the data synchronization task at both ends during the data synchronization process.

[0271] In some embodiments, when the first data platform detects that the task status is a success status during the cyclic detection process, the first data platform generates a data synchronization success message and sends the data synchronization success message to the second data platform.

[0272] For example, please refer to Figure 7 , which shows a schematic diagram of a data synchronization system 700 provided by an exemplary embodiment of the present application. Taking the data synchronization system 700 providing data synchronization functions on different platforms in a cross-enterprise network environment formed by a first enterprise and a second enterprise as an example, the cross-enterprise network environment includes a first enterprise platform 710 and a second enterprise platform 720, wherein the second enterprise platform 720 includes an enterprise intranet platform 721 and an enterprise public cloud platform 722. The first enterprise platform 710 includes a computing node 711, the enterprise intranet platform 721 includes a first central data synchronization module 7211 and an external dependency module 7212, and the enterprise public cloud platform 722 includes a second central data synchronization module 7221 and a hive storage unit 7222. The second enterprise platform 720 also includes an object storage cloud service 723 and a management platform 724.

[0273] Specifically, taking the target business provided by the first enterprise synchronizing account data from the first enterprise platform 710 to the enterprise public cloud platform 722 in the second enterprise platform 720 as an example, the information processing flow of the data synchronization system 700 is as follows:

[0274] Step 1: The target business party of the first enterprise collects user-authorized compliance account data through information collection points such as clients, web pages, and mini-programs and stores it in the first enterprise platform 710.

[0275] Step 2: The computing node 711 in the first enterprise platform 710 writes the data to be synchronized into the object storage cloud service 723, wherein the object storage cloud service 723 allows cross-platform access through the https protocol of a secure channel.

[0276] Step 3, the computing node 711 abstracts this synchronization task into a task notification, records the amount of data to be synchronized, marks the task notification status as new, and sends the notification to the first central data synchronization module 7211 through the external dependency module 7212. The first central data synchronization module 7211 forwards the received notification to the management platform 724.

[0277] Step 4: The first central data synchronization module 7211 starts to cyclically detect whether the task notification status is switched from new creation to other status.

[0278] Step 5: The first central data synchronization module 7211 initiates a data re-run or re-recording request to the second central data synchronization module 7221 of the enterprise public cloud platform 722.

[0279] Step 6: The second central data synchronization module 7221 receives the above request and reads the data to be synchronized in the object storage cloud service 723.

[0280] Step 7: The second central data synchronization module 7221 verifies the consistency of the data according to the task notification details.

[0281] Step 8: If the data verification is successful, the second central data synchronization module 7221 writes the data into the hive storage unit 7222 and updates the task status to the management platform 724 as successful.

[0282] Step 9 (not shown in the figure), if the data verification fails, the second central data synchronization module 7221 cleans up the data and tries again; if the retry is successful, the task status is updated to success; if the number of retries reaches the upper limit and the verification still fails, the task status is updated to failure, and the management platform 724 initiates a system notification to notify the business party and platform administrator to intervene manually.

[0283] To summarize, when realizing data synchronization of the first data between the data synchronization sending platform and the data synchronization receiving platform, the first data is temporarily stored through the intermediate storage end, and the task status of the data synchronization task of the first data during the data synchronization process is recorded and updated through the management platform, that is, after the data synchronization receiving platform completes the data synchronization of the first data, the data synchronization receiving platform updates the task status recorded in the management platform to a completed status through a first status update message, so that the data synchronization sending platform can know whether the data synchronization task of the first data is completed at any time through the management platform, thereby coordinating the data synchronization sending platform and the data synchronization receiving platform through the management platform, so that the data synchronization process of the first data maintains consistent task status in a cross-platform environment, so that in the cross-platform data synchronization process, by unifying the information recognition of the task status between different platforms, the data synchronization efficiency of data during cross-platform synchronization is guaranteed. At the same time, the intermediate storage end, as an intermediate storage of data, opens up the data transmission channel between different platforms. Since the management platform is set as the intermediate storage of the task status, after the data synchronization sending platform sends the first data in the data synchronization task to the intermediate storage end, the data synchronization sending platform only needs to pay attention to the task status in the management platform. Even if the data synchronization receiving platform fails to synchronize, it can also clear the data and re-acquire the first data from the intermediate storage end for data synchronization. Different platforms no longer need to re-acquire the first data through retransmission instructions after data synchronization fails, which reduces the data interaction requirements between platforms and avoids the repeated cross-platform transmission of the first data when it is heterogeneous data. It avoids the high failure rate of heterogeneous data when it is transmitted across platforms, and improves the data synchronization efficiency when the data is synchronized across platforms.

[0284] For example, in combination with the message synchronization system provided in the above embodiment, the hardware environment of the message synchronization system is schematically described. Figure 8 As shown, it shows a schematic diagram of the overall architecture 800 of a message synchronization system provided by an exemplary embodiment of the present application.

[0285] The overall architecture 800 is an environment for data synchronization between platforms of a first enterprise and a second enterprise, or data synchronization between platforms within the second enterprise. The overall architecture 800 includes a computing node 811 in the first enterprise platform of the first enterprise. The computing node 811 is used to provide hardware support when sending data to be synchronized and corresponding data synchronization task notifications to the second enterprise platform of the second enterprise.

[0286] The overall architecture 800 also includes the hardware of the second enterprise platform of the second enterprise, and the second enterprise platform includes the enterprise intranet platform, the enterprise public cloud platform and the operation and maintenance platform. Among them, the enterprise intranet platform includes the external dependency module 821, the first heterogeneous data source 822, and the first central data synchronization module 823; the enterprise public cloud platform includes the second heterogeneous data source 831 and the second central data synchronization module 832; the operation and maintenance platform includes the management module 841, the third heterogeneous data source 842, and the task status database 843.

[0287] The overall architecture 800 also includes an object storage cloud service 850 .

[0288] Among them, both the first enterprise platform and the second enterprise platform can read and write data with the object storage cloud service 850. The first enterprise platform communicates with the enterprise intranet platform through the computing node 811 and the external dependency module 821 in the enterprise intranet platform. The first central data synchronization module 823 and the second central data synchronization module 832 interact with the management module 841 to maintain the consistency of the task status of the data synchronization task.

[0289] Please refer to Fig. 9 , which shows a flowchart of a data synchronization method provided by an exemplary embodiment of the present application. In the embodiment of the present application, taking the method executed by the management platform as an example, the method includes steps 910 to 920.

[0290] Step 910: Obtain a data synchronization task corresponding to the first data.

[0291] In an embodiment of the present application, the management platform is used to manage the task status of the data synchronization task. Schematically, the first data is the data to be synchronized to the data synchronization receiving platform, and the first data is the data written to the intermediate storage end by the data synchronization sending platform.

[0292] In an embodiment of the present application, at the beginning stage of a data synchronization task, the management platform records that the task status of the data synchronization task is in a pending synchronization state.

[0293] In some embodiments, a data synchronization task for synchronizing data stored on the data synchronization sending platform and data stored on the data synchronization receiving platform is preconfigured in the management platform.

[0294] In some embodiments, the management platform provides a visual interface to the user for creating a new data synchronization task. Schematically, a terminal equipped with the management platform displays a task creation interface, which is used to create a data processing task, wherein the task creation interface includes multiple task type candidates, and the multiple task type candidates include a data synchronization type; a first selection operation for a data source type is received, and a second selection operation for a data target type is received, wherein the data source type is used to indicate the data type of the first data, and the data target type is used to indicate the data type of the first data after the data processing task is completed; in the case of receiving a selection of a data synchronization type from multiple task type candidates, a data synchronization sending platform is determined based on the data source type, and a data synchronization receiving platform is determined based on the data target type; a data synchronization task is created based on the data synchronization sending platform and the data synchronization receiving platform.

[0295] In one example, if Fig.10 As shown, it shows a schematic diagram of a visual interface provided to a user by a management platform provided by an exemplary embodiment of the present application. The management platform provides a management interface 1000 to the user, in which the user can create and manage data synchronization tasks, wherein a task creation control 1010 is displayed in the management interface 1000. In response to the task creation control 1010 receiving a trigger operation, a task creation interface 1020 is displayed. The task creation interface 1020 includes multiple task type candidates 1030, and the multiple task type candidates 1030 include a data synchronization type 1031. After selecting the data synchronization type 1031, the user enters / selects a data source type 1041 in the data source type input box provided in the task creation interface 1020, and enters / selects a data target type 1042 in the data target type input box. When the confirmation control 1050 receives the trigger operation, a data synchronization task is generated with the data synchronization sending platform corresponding to the data source type 1041 as the data sender and the data synchronization receiving platform corresponding to the data target type 1042 as the data receiver.

[0296] That is, the management platform provides users with a visual operation interface, which reduces the cost of business learning and improves the configuration efficiency of data synchronization task configuration.

[0297] Step 920: When a first status update message sent by the data synchronization receiving platform is received, the task status of the data synchronization task is switched to a completed status.

[0298] Among them, the first status update message is sent by the data synchronization receiving platform in response to the received data synchronization request, reading the first data from the intermediate storage end, performing data synchronization processing on the first data, and after the first data synchronization is completed. The data synchronization request is sent after the data synchronization task is started.

[0299] In an embodiment of the present application, the data synchronization receiving platform is used to read the first data from the intermediate storage end based on the received data synchronization request, and perform data synchronization processing on the first data.

[0300] Optionally, the data synchronization process includes at least one of the following operations: a storage operation, a verification operation, a conversion operation, an update operation, a log recording operation, and a permission setting operation.

[0301] In an embodiment of the present application, when the data synchronization receiving platform determines that the data synchronization processing performed on the first data has been successfully executed, it determines that the synchronization of the first data is completed, generates a first status update message, and sends the first status update message to the management platform. The first status update message is used to indicate to the management platform that the data synchronization task of the first data has been completed.

[0302] In an embodiment of the present application, the task status of the data synchronization task managed in the management platform is used to keep the task status of the data synchronization sending platform and the data synchronization receiving platform consistent when executing the data synchronization task.

[0303] In some embodiments, the data synchronization sending platform cyclically detects the task status of the data synchronization task stored in the management platform, and determines that the data synchronization task is completed when it is detected that the task status is a success status.

[0304] To summarize, when realizing data synchronization of the first data between the data synchronization sending platform and the data synchronization receiving platform, the first data is temporarily stored through the intermediate storage end, and the task status of the data synchronization task of the first data during the data synchronization process is recorded and updated through the management platform, that is, after the data synchronization receiving platform completes the data synchronization of the first data, the data synchronization receiving platform updates the task status recorded in the management platform to a completed status through a first status update message, so that the data synchronization sending platform can know whether the data synchronization task of the first data is completed at any time through the management platform, thereby coordinating the data synchronization sending platform and the data synchronization receiving platform through the management platform, so that the data synchronization process of the first data maintains consistent task status in a cross-platform environment, so that in the cross-platform data synchronization process, by unifying the information recognition of the task status between different platforms, the data synchronization efficiency of data during cross-platform synchronization is guaranteed.

[0305] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0306] Please refer to Fig.11 , which shows a structural block diagram of a data synchronization device provided by an exemplary embodiment of the present application, the device includes the following modules:

[0307] An acquisition module 1110 is used to acquire a data synchronization task corresponding to first data, wherein the task status of the data synchronization task is in a to-be-synchronized state, the first data is data written by a data synchronization sending platform to an intermediate storage terminal, and the first data is data to be synchronized to a data synchronization receiving platform;

[0308] The switching module 1120 is used to switch the task status of the data synchronization task to the completion status when receiving the first status update message sent by the data synchronization receiving platform; the first status update message is the response of the data synchronization receiving platform to the received data synchronization request, reading the first data from the intermediate storage end, performing data synchronization processing on the first data, and sent after the first data synchronization is completed, and the data synchronization request is sent after the data synchronization task is started.

[0309] In some optional embodiments, such as Fig.12 As shown, the device also includes:

[0310] Display module 1130, used to display a task creation interface, the task creation interface is used to create a data processing task, the task creation interface includes a plurality of task type candidates, the plurality of task type candidates include a data synchronization type;

[0311] The receiving module 1140 is configured to receive a first selection operation for a data source type and a second selection operation for a data target type, wherein the data source type is used to indicate a data type of the first data to be synchronized, and the data target type is used to indicate a data type of the first data after the data processing task is completed;

[0312] A determination module 1150 is configured to, upon receiving a request to select the data synchronization type from the plurality of task type candidates, determine the data synchronization sending platform based on the data source type, and determine the data synchronization receiving platform based on the data target type;

[0313] The creation module 1160 is used to create the data synchronization task based on the data synchronization sending platform and the data synchronization receiving platform.

[0314] In some optional embodiments, the acquisition module 1110 is further used to receive a data synchronization task notification sent by the first data platform; wherein the first data platform is used to acquire and send a data synchronization task notification corresponding to the first data to the management platform, and the data synchronization task notification is used to indicate that the task state of the data synchronization task is in the pending synchronization state;

[0315] The switching module 1120 is further configured to, in response to the data synchronization task notification, record that the task status of the data synchronization task is in the pending synchronization status.

[0316] In some optional embodiments, the first data platform is further used to cyclically detect the task status of the data synchronization task stored in the management platform, and when it is detected that the task status is a success status, determine that the data synchronization task is completed.

[0317] In some optional embodiments, the first data platform is further configured to send the data synchronization request to the data synchronization receiving platform when a task start condition of the data synchronization task is met.

[0318] In some optional embodiments, the first data platform is the data synchronization sending platform, and the first data platform includes a first data synchronization module and a first storage unit; the first data synchronization module is used to read the first data from the first storage unit, write the first data to the intermediate storage end, and generate the data synchronization task notification corresponding to the first data.

[0319] In some optional embodiments, the data synchronization sending platform is a second data platform, and the second data platform is used to write the first data to the intermediate storage end, generate the data synchronization task notification corresponding to the first data, and send the data synchronization task notification to the first data platform.

[0320] In some optional embodiments, the data synchronization receiving platform further includes a second data synchronization module and a second storage unit;

[0321] The second data synchronization module is also used to obtain first attribute information of the first data before data synchronization, and when reading the first data from the intermediate storage end, detect second attribute information corresponding to the first data, and when the first attribute information and the second attribute information match, write the first data into the second storage unit, and when it is determined that the first data has been written into the second storage unit, determine that the synchronization of the first data is completed.

[0322] In some optional embodiments, the first attribute information includes at least one of the following:

[0323] The data volume of the first data;

[0324] a data type of the first data;

[0325] a data format of the first data;

[0326] index information of the first data;

[0327] A hash value of the first data.

[0328] In some optional embodiments, the acquisition module 1110 is further configured to acquire a data synchronization task notification corresponding to the first data, wherein the data synchronization task notification includes the first attribute information;

[0329] The device also includes:

[0330] The sending module 1170 is used to send the first attribute information to the second data synchronization module based on the data synchronization task notification.

[0331] In some optional embodiments, the data synchronization receiving platform further includes a second data synchronization module and a second storage unit;

[0332] The second data synchronization module is used to perform data conversion processing on the first data to obtain second data when the first data and the data stored in the second storage unit are heterogeneous data, and write the second data into the second storage unit. When it is determined that the second data has been written into the second storage unit, it is determined that the synchronization of the first data is completed.

[0333] In some optional embodiments, the data synchronization receiving platform is also used to clear the acquired first data, re-read the first data from the intermediate storage end, and perform data synchronization processing on the re-read first data when it is determined that the synchronization of the first data has failed.

[0334] In some optional embodiments, the acquisition module 1110 is further configured to receive a second status update message sent by the data synchronization receiving platform when the number of first data synchronization failures reaches a first threshold;

[0335] The switching module 1120 is further used to switch the task state of the data synchronization task to a failure state;

[0336] The acquisition module 1110 is further used to generate a data synchronization failure notification, where the data synchronization failure notification is used to notify manual intervention in the data synchronization task.

[0337] In some optional embodiments, the first data platform is further used to cyclically detect the task status of the data synchronization task stored in the management platform, and stop the cyclic detection of the task status when it is detected that the task status is a failure status.

[0338] It should be noted that the data synchronization device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data synchronization device provided in the above embodiment belongs to the same concept as the data synchronization method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0339] Fig.13 The schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown, which specifically includes the following structure.

[0340] The server 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The server 1300 also includes a mass storage device 1306 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0341] The mass storage device 1306 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1306 and its associated computer readable media provide non-volatile storage for the server 1300. That is, the mass storage device 1306 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0342] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1304 and mass storage device 1306 can be collectively referred to as memory.

[0343] According to various embodiments of the present application, the server 1300 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1300 can be connected to the network 1312 through the network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote computer systems (not shown).

[0344] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0345] The embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the model training method provided by the above-mentioned method embodiments. Optionally, the computer device can be a terminal or a server.

[0346] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the model training method provided by the above-mentioned method embodiments.

[0347] The embodiments of the present application also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the model training method described in any of the above embodiments.

[0348] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0349] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0350] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data synchronization system, characterized in that: The system includes a management platform, an intermediate storage terminal and a data synchronization receiving platform; The intermediate storage end is used to obtain first data from the data synchronization sending platform, where the first data is data to be synchronized to the data synchronization receiving platform; The management platform is used to obtain a data synchronization task corresponding to the first data, wherein the task status of the data synchronization task is in a pending synchronization status; The data synchronization receiving platform is used to receive a data synchronization request; in response to the data synchronization request, read the first data from the intermediate storage end; Performing data synchronization processing on the first data; when the first data synchronization is completed, sending a first status update message to the management platform; the data synchronization request is sent after the data synchronization task is started; The management platform is used to switch the task state of the data synchronization task to a completed state after receiving the first status update message.

2. The system according to claim 1, characterized in that The system also includes a first data platform; The first data platform is used to cyclically detect the task status of the data synchronization task stored in the management platform, and determine that the data synchronization task is completed when it is detected that the task status is a success status.

3. The system according to claim 2, characterized in that The first data platform is further configured to send the data synchronization request to the data synchronization receiving platform when a task starting condition of the data synchronization task is met.

4. The system according to claim 2 or 3, characterized in that: The first data platform is further used to obtain a data synchronization task notification corresponding to the first data, wherein the data synchronization task notification is used to indicate that the task status of the data synchronization task is in the pending synchronization status; and send the data synchronization task notification to the management platform; The management platform is further configured to, upon receiving the data synchronization task notification sent by the first data platform, record that the task status of the data synchronization task is in the pending synchronization status.

5. The system according to claim 4, characterized in that The first data platform is the data synchronization sending platform, and the first data platform includes a first data synchronization module and a first storage unit; The first data synchronization module is used to read the first data from the first storage unit and write the first data into the intermediate storage end; Generate the data synchronization task notification corresponding to the first data.

6. The system according to claim 4, characterized in that The system further includes a second data platform, which is the data synchronization sending platform; The second data platform is used to write the first data into the intermediate storage end; Generate the data synchronization task notification corresponding to the first data; Sending the data synchronization task notification to the first data platform.

7. The system according to any one of claims 1 to 3, characterized in that: The data synchronization receiving platform also includes a second data synchronization module and a second storage unit; The second data synchronization module is used to obtain the first attribute information of the first data before data synchronization; when the first data is read from the intermediate storage end, detect the second attribute information corresponding to the first data; when the first attribute information and the second attribute information match, write the first data into the second storage unit; when it is determined that the first data has been written into the second storage unit, determine that the synchronization of the first data is completed.

8. The system according to claim 7, characterized in that The first attribute information includes at least one of the following: The data volume of the first data; a data type of the first data; a data format of the first data; index information of the first data; A hash value of the first data.

9. The system according to claim 7, characterized in that The management platform is used to obtain a data synchronization task notification corresponding to the first data, where the data synchronization task notification includes the first attribute information; and send the first attribute information to the second data synchronization module based on the data synchronization task notification.

10. The system according to any one of claims 1 to 3, characterized in that: The data synchronization receiving platform also includes a second data synchronization module and a second storage unit; The second data synchronization module is used to perform data conversion processing on the first data to obtain second data when the first data and the data stored in the second storage unit are heterogeneous data; write the second data into the second storage unit; and determine that the synchronization of the first data is completed when it is determined that the second data has been written into the second storage unit.

11. The system according to any one of claims 1 to 3, characterized in that: The data synchronization receiving platform is further used to clear the acquired first data when it is determined that the synchronization of the first data fails; and re-read the first data from the intermediate storage end; A data synchronization process is performed on the re-read first data.

12. The system according to claim 11, characterized in that The data synchronization receiving platform is further configured to send a second status update message to the management platform when the number of first data synchronization failures reaches a first threshold; The management platform is further used to switch the task status of the data synchronization task to a failure status when receiving the second status update message; and generate a data synchronization failure notification, wherein the data synchronization failure notification is used to notify manual intervention in the data synchronization task.

13. The system according to claim 12, characterized in that The system also includes a first data platform; The first data platform is further used to cyclically detect the task status of the data synchronization task stored in the management platform, and stop the cyclic detection of the task status when it is detected that the task status is a failure status.

14. A data synchronization method, characterized in that: The method is executed by a management platform, and includes: Acquire a data synchronization task corresponding to first data, wherein the task status of the data synchronization task is in a pending synchronization state, the first data is data written to an intermediate storage end by a data synchronization sending platform, and the first data is data to be synchronized to a data synchronization receiving platform; When the first status update message sent by the data synchronization receiving platform is received, the task status of the data synchronization task is switched to the completion status; the first status update message is sent by the data synchronization receiving platform in response to the received data synchronization request, reading the first data from the intermediate storage end, and performing data synchronization processing on the first data after the first data synchronization is completed, and the data synchronization request is sent after the data synchronization task is started.

15. The method according to claim 14, characterized in that The acquiring of the data synchronization task corresponding to the first data includes: Displaying a task creation interface, wherein the task creation interface is used to create a data processing task, wherein the task creation interface includes a plurality of task type candidates, wherein the plurality of task type candidates include a data synchronization type; receiving a first selection operation for a data source type, and receiving a second selection operation for a data target type, wherein the data source type is used to indicate a data type of the first data to be synchronized, and the data target type is used to indicate a data type of the first data after the data processing task is completed; In case of receiving the data synchronization type selected from the plurality of task type candidates, determining the data synchronization sending platform based on the data source type, and determining the data synchronization receiving platform based on the data target type; The data synchronization task is created based on the data synchronization sending platform and the data synchronization receiving platform.

16. A data synchronization device, characterized in that: The device comprises: An acquisition module is used to acquire a data synchronization task corresponding to first data, wherein the task status of the data synchronization task is in a to-be-synchronized state, the first data is data written to an intermediate storage end by a data synchronization sending platform, and the first data is data to be synchronized to a data synchronization receiving platform; A switching module is used to switch the task status of the data synchronization task to a completion status when a first status update message sent by the data synchronization receiving platform is received; the first status update message is a response by the data synchronization receiving platform to a received data synchronization request, the first data is read from the intermediate storage end, and data synchronization processing is performed on the first data, and the message is sent after the first data synchronization is completed, and the data synchronization request is sent after the data synchronization task is started.

17. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the data synchronization method as described in claim 14 or 15.

18. A computer-readable storage medium, characterized in that: At least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by the processor to implement the data synchronization method as described in claim 14 or 15.

19. A computer program product, characterized in that It comprises a computer program or an instruction, which, when executed by a processor, implements the data synchronization method as claimed in claim 14 or 15.

Images