Data synchronization method and device, equipment, storage medium and product

Abstract

The invention provides a data synchronization method and device, equipment, a storage medium and a product, and relates to the field of data synchronization. The method comprises the following steps: obtaining full data and performing packaging processing to obtain a plurality of data packets, performing encryption processing on the data packets, generating a token for each data packet to obtain a plurality of encrypted data packets and tokens corresponding to the encrypted data packets, obtaining a data synchronization request sent by an application of a downstream system, and according to authentication information in the data synchronization request, performing data synchronization on the encrypted data packets according to the authentication information in the data synchronization request. Authenticating an application of a downstream system, determining a transmission rule, determining target transmission data according to request information in the data synchronization request, and performing data processing on the plurality of encrypted data packets and the token corresponding to each encrypted data packet according to the target transmission data and the transmission rule to obtain to-be-synchronized data, and issuing the to-be-synchronized data to an application of a downstream system. According to the method, the data synchronization efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data synchronization, and more particularly to a data synchronization method, apparatus, device, storage medium, and product. Background Technology

[0002] With the development of information technology, multi-dimensional and multi-source data has become the norm, posing a technical challenge to traditional data synchronization methods in terms of how to synchronize data efficiently and securely. Meanwhile, authentication mechanisms, as an important technology in the security field, protect the security of data access by verifying the identity of users or systems.

[0003] Most data synchronization is direct database-to-database synchronization, lacking hierarchical and layered data processing during data transfer. Furthermore, after opening interfaces or transmitting data, the authentication mechanism only authenticates whether data access is permitted or only verifies the identity of the data recipient, lacking authentication management for individual data items.

[0004] The main technical problem with existing technologies is the low efficiency of data synchronization. Summary of the Invention

[0005] This application provides a data synchronization method, apparatus, device, storage medium, and product to solve the problem of low efficiency in data synchronization in the prior art.

[0006] Firstly, this application provides a data synchronization method, including:

[0007] Obtain the full data and encapsulate it into multiple data packets;

[0008] The data packets are encrypted, and a token is generated for each data packet to obtain multiple encrypted data packets and a token corresponding to each encrypted data packet;

[0009] Obtain data synchronization requests sent by applications in downstream systems, where the data synchronization requests include authentication information and request information;

[0010] Based on the authentication information, the downstream system applications are authenticated to determine the transmission rules.

[0011] Based on the request information, determine the target data to be transmitted;

[0012] Based on the target transmission data and transmission rules, data processing is performed on multiple encrypted data packets and the token corresponding to each encrypted data packet to obtain the data to be synchronized;

[0013] Applications that send data to be synchronized to downstream systems.

[0014] In one possible design, the authentication information includes at least one of the following: the system identifier of the downstream system, the application identifier of the application, and the user identifier corresponding to the downstream system.

[0015] Accordingly, based on the authentication information, the applications in the downstream system are authenticated, including:

[0016] Obtain pre-configured authentication rules and authentication information to authenticate applications in downstream systems. The authentication rules include at least one of the following: the correspondence between system identifier and transmission rules, the correspondence between application identifier and transmission rules, and the correspondence between user identifier and transmission rules.

[0017] The transmission rules include full transmission, partial transmission, encrypted transmission, and no transmission allowed.

[0018] In one possible design, before obtaining pre-configured authentication rules and information to authenticate applications in downstream systems, the following steps are also included:

[0019] Display the rule configuration window;

[0020] In response to the user's configuration operation in the rule configuration window, determine the pre-configured authentication rules.

[0021] In one possible design, applications that distribute the data to be synchronized to downstream systems include:

[0022] Obtain the target task queue corresponding to the application in the downstream system;

[0023] Add the data to be synchronized to the target task queue, so that the data to be synchronized can be distributed to downstream system applications based on the target task queue.

[0024] In one possible design, the application of adding data to be synchronized to a target task queue, and then distributing the data to downstream systems based on the target task queue, includes:

[0025] Determine the queued tasks of all task queues in the upstream system, where the task queues include the target task queue; pop the queued tasks of the task queues in order according to their queue order.

[0026] In one possible design, after adding the data to be synchronized to the target task queue, the following is also included:

[0027] Obtain the data synchronization status of applications in downstream systems;

[0028] If the data synchronization status is "synchronization failed", an alarm will be issued.

[0029] In one possible design, after determining the target data to be transmitted based on the request information, the following is also included:

[0030] According to the transmission rules, the target transmission data is written to disk.

[0031] Secondly, this application provides a data synchronization device, comprising:

[0032] The first acquisition module is used to acquire the full data and encapsulate the full data into multiple data packets;

[0033] The first processing module is used to encrypt the data packets and generate a token for each data packet to obtain multiple encrypted data packets and a token corresponding to each encrypted data packet.

[0034] The second acquisition module is used to acquire data synchronization requests sent by applications in the downstream system, wherein the data synchronization requests include authentication information and request information.

[0035] The second processing module is used to authenticate the applications of the downstream system based on the authentication information in order to determine the transmission rules.

[0036] The third processing module is used to determine the target data to be transmitted based on the request information;

[0037] The fourth processing module is used to process multiple encrypted data packets and the token corresponding to each encrypted data packet according to the target transmission data and transmission rules, so as to obtain the data to be synchronized;

[0038] The fifth processing module is used to send the data to be synchronized to downstream system applications.

[0039] Thirdly, this application provides a data synchronization device, comprising:

[0040] At least one processor;

[0041] and memory that is communicatively connected to at least one processor;

[0042] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the data synchronization method as described in any of the first aspects.

[0043] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data synchronization method as described in any of the first aspects.

[0044] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the data synchronization method as described in any of the first aspects.

[0045] This application provides a data synchronization method, apparatus, device, storage medium, and product. The method involves acquiring all data and encapsulating it into multiple data packets. These data packets are then encrypted, and a token is generated for each packet, resulting in multiple encrypted data packets and their corresponding tokens. The method also involves acquiring data synchronization requests from downstream system applications, authenticating the applications based on authentication information in the data synchronization requests, determining transmission rules, identifying target data for transmission based on request information, and processing the multiple encrypted data packets and their corresponding tokens according to the target data and transmission rules to obtain data to be synchronized. This data is then distributed to the downstream system applications. This application achieves multi-dimensional data synchronization based on an authentication mechanism, solving the aforementioned technical problems, improving data synchronization security, ensuring the consistency, integrity, flexibility, and relevance of transmitted data, and ultimately improving data synchronization efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This application provides a schematic diagram of a data synchronization system architecture.

[0048] Figure 2 A flowchart illustrating a data synchronization method provided in this application embodiment. Figure 1 ;

[0049] Figure 3 A flowchart illustrating a data synchronization method provided in this application embodiment. Figure 2 ;

[0050] Figure 4 A flowchart illustrating the data synchronization mechanism provided in this application embodiment;

[0051] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of a data synchronization device provided in an embodiment of this application.

[0053] Figure label:

[0054] 11-Data acquisition equipment; 12-Processing equipment; 13-Display equipment;

[0055] 51-First Acquisition Module; 52-First Processing Module; 53-Second Acquisition Module; 54-Second Processing Module; 55-Third Processing Module; 56-Fourth Processing Module; 57-Fifth Processing Module;

[0056] 60 - Electronic device; 61 - Processor; 62 - Memory; 63 - Communication component; 64 - Bus. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0058] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0059] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more.

[0060] It should be noted that the phrase "at...time" in the embodiments of this application can refer to the instant at which a certain situation occurs, or to a period of time after the occurrence of a certain situation; the embodiments of this application do not specifically limit this. Furthermore, the data synchronization method provided in the embodiments of this application is merely an example, and data synchronization methods may include more or fewer elements.

[0061] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0062] Application Programming Interface (API): An API is a set of predefined functions designed to provide applications and developers with the ability to access a set of routines based on certain software or hardware, without needing access to the source code or understanding the details of the internal workings. It is a set of functions that can be used to manipulate components, applications, or operating systems. An API itself is abstract; it only defines an interface and does not involve the specific operations performed by the application in its actual implementation.

[0063] Software Development Kit (AK) – In computer communication and network protocols, it represents an acknowledgment mechanism. When one party (such as the sender) sends data to another party (such as the receiver), the receiver sends an ACK signal in response to acknowledge successful receipt of the data. This is a fundamental communication protocol mechanism used to ensure reliable data transmission.

[0064] A Software Development Kit (SDK) is a set of tools provided by the manufacturer of a hardware platform, operating system, or programming language to assist software developers in creating applications for a specific platform, system, or programming language. It typically includes various tools and resources needed to develop applications, such as compilers, debuggers, APIs, documentation, libraries, runtime / development environments, testing / analysis tools, and network protocols. These tools and resources can greatly simplify the development process and improve development efficiency.

[0065] With the development of information technology, multi-dimensional and multi-source data has become the norm, making efficient and secure data synchronization a crucial technical challenge. Traditional data synchronization is mostly direct database-to-database synchronization, such as from MySQL to ClickHouse or from Oracle to a data warehouse. Specific implementation methods include message queues (MQ), CDC, and even database-based binlog methods. However, during data transmission, there is a lack of data layering and hierarchical classification; furthermore, after opening interfaces or transmitting data, authentication mechanisms only authenticate whether data can be accessed or only authenticate the identity of the data recipient, lacking authentication management of individual data items.

[0066] In this process, a unified interface is used for data security and application mechanisms. After full integration, the data application party filters and uses the data. There is a lack of comprehensive data security protection, and sensitive data cannot be effectively transmitted.

[0067] Therefore, traditional data synchronization methods face several challenges: how to ensure the security and integrity of data transmission during data synchronization; how to guarantee data consistency during the synchronization of multi-source data; how to avoid system errors caused by data conflicts; and how to minimize latency during data synchronization to ensure the real-time responsiveness of the system.

[0068] Existing authentication mechanisms, as a crucial technology in the security field, protect the security of data access by verifying the identity of users or systems. Combining authentication mechanisms with data synchronization technology can not only improve the security of data synchronization but also simplify the synchronization process to some extent, thereby enhancing the overall performance of the system.

[0069] Based on this, in order to address the limitations of existing data synchronization methods in terms of data synchronization security, data transmission consistency, integrity, flexibility, and specificity, which leads to low data synchronization efficiency, this application provides a data synchronization method, apparatus, device, storage medium, and product that can be used in the field of data synchronization.

[0070] The inventive concept of this application is as follows: during the data synchronization process, each data item is authenticated to enable one-to-one access between the user, application and specific data item; after the data synchronization is completed, the data target is authenticated to ensure that the data can only be used by the authorized target; at the same time, the authentication policy is updated regularly according to business needs and security policies to ensure the security of data synchronization, thereby improving the efficiency of data synchronization.

[0071] Optional, Figure 1 This is a schematic diagram of a data synchronization system architecture provided in an embodiment of this application. The application data processing system is a computer device. Figure 1 In the above architecture, at least one of data acquisition device 11, processing device 12 and display device 13 is included.

[0072] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of the application data processing system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0073] In the specific implementation process, the data acquisition device 11 may include an input / output interface or a communication interface, and the data acquisition device 11 can be connected to the processing device through the input / output interface or the communication interface.

[0074] The processing device 12 can authenticate each data item during the data synchronization process, enabling one-to-one access between users, applications, and specific data items; after data synchronization is completed, it can authenticate the data target to ensure that the data can only be used by authorized targets; at the same time, it can regularly update the authentication policy according to business needs and security policies to ensure the security of data synchronization and thus improve the efficiency of data synchronization.

[0075] Display device 13 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0076] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0077] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0078] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0079] Figure 2 A flowchart illustrating a data synchronization method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method includes:

[0080] S201. Obtain the full data and encapsulate it into multiple data packets.

[0081] In this embodiment, the upstream system passes the data object to be distributed to the SDK. The SDK obtains the full data and performs packet processing on the full data to obtain multiple data packets.

[0082] S202. Encrypt the data packets and generate a token for each data packet to obtain multiple encrypted data packets and a token corresponding to each encrypted data packet.

[0083] In this embodiment, the SDK encrypts the data packets and generates a token for each data packet to obtain multiple encrypted data packets and a token corresponding to each encrypted data packet.

[0084] S203. Obtain the data synchronization request sent by the application of the downstream system, wherein the data synchronization request includes authentication information and request information.

[0085] In this embodiment, authentication information and request information sent by applications in the downstream system are obtained.

[0086] S204. Based on the authentication information, authenticate the applications in the downstream system to determine the transmission rules.

[0087] The authentication information includes at least one of the following: the system identifier of the downstream system, the application identifier of the application, and the user identifier corresponding to the downstream system.

[0088] Specifically, pre-configured authentication rules and authentication information are obtained to authenticate applications in downstream systems. The authentication rules include at least one of the following: the correspondence between system identifiers and transmission rules, the correspondence between application identifiers and transmission rules, and the correspondence between user identifiers and transmission rules.

[0089] The pre-configured authentication rules and authentication information can be determined according to the actual situation, and this application embodiment does not impose specific restrictions on them.

[0090] More specifically, the transmission rules include full transmission, partial transmission, encrypted transmission, and no transmission allowed.

[0091] For example, authentication rules can implement a series of personalized processing measures, such as distributing the same data to system A but not to system B; distributing 10 fields of the same data to system A and 5 fields to system B; and distributing a certain field of the same data in plaintext to system A and in de-identified ciphertext to system B.

[0092] It should be noted that before obtaining the pre-configured authentication rules and information to authenticate applications in downstream systems, the following steps are also included:

[0093] Optional, display the rules configuration window.

[0094] Further optionally, in response to the user's configuration operation in the rule configuration window, the pre-configured authentication rules are determined.

[0095] For example, rules can be configured in the engine's management interface through functions such as data interface registration, data permission application, rule configuration, and de-identification rule configuration.

[0096] Specifically, the authentication rules are highly scalable and can be extended to include new rules based on actual business needs. By integrating the authentication rules into the entire data processing chain, new business scenarios can be quickly met.

[0097] It should also be noted that when metadata changes, it is tracked according to the mapping rules and the data field status. If the field changes, the downstream configuration personnel need to redefine the data mapping relationship and configure the authentication rules in the upstream system.

[0098] S205. Determine the target data to be transmitted based on the request information.

[0099] In this embodiment, the target transmission data is determined based on the request information. The target transmission data includes the data to be transmitted and its transmission object.

[0100] It should be noted that after step S205, the following steps are also included:

[0101] According to the transmission rules, the target transmission data is written to disk.

[0102] In this embodiment, the target transmitted data is written to disk based on the identity and role information in the request information to achieve data granularity confirmation.

[0103] Specifically, data persistence on disk can ensure data durability and reliability, and prevent data loss due to power outages or other emergencies.

[0104] S206. Based on the target transmission data and transmission rules, perform data processing on multiple encrypted data packets and the token corresponding to each encrypted data packet to obtain the data to be synchronized.

[0105] Specifically, based on the target data transmission and transmission rules, field management and data anonymization are performed on multiple encrypted data packets and the token corresponding to each encrypted data packet to obtain the data to be synchronized.

[0106] S207, Applications that send data to be synchronized to downstream systems.

[0107] In this embodiment, the application that sends the data to be synchronized to the downstream system of the SDK receives the data based on the SDK. The receiving process requires identity processing, calling authentication, data decryption and unpacking, and receiving the data.

[0108] This embodiment provides a data synchronization method that acquires all data and performs packet processing to obtain multiple data packets. These data packets are then encrypted, and a token is generated for each data packet, resulting in multiple encrypted data packets and their corresponding tokens. A data synchronization request sent by a downstream system application is obtained. Based on the authentication information in the data synchronization request, the downstream system application is authenticated to determine transmission rules. Based on the request information in the data synchronization request, the target data to be transmitted is determined. Based on the target data and transmission rules, the multiple encrypted data packets and their corresponding tokens are processed to obtain the data to be synchronized. This data to be synchronized is then sent to the downstream system application. This application achieves multi-dimensional data synchronization based on an authentication mechanism, solving the aforementioned technical problems, improving the security of data synchronization, ensuring the consistency, integrity, flexibility, and specificity of transmitted data, and thus improving the efficiency of data synchronization.

[0109] Figure 3 A flowchart illustrating a data synchronization method provided in this application embodiment. Figure 2 ,exist Figure 2 Based on the embodiments, such as Figure 3 As shown, the specific implementation steps of S207 above include:

[0110] S301. Obtain the target task queue corresponding to the application in the downstream system.

[0111] In this embodiment, a data task corresponding to the application of the downstream system is generated based on the request information, and a target task queue is generated based on the data task. The target task queue includes two queue forms: an MQ queue and a memory queue.

[0112] S302. Add the data to be synchronized to the target task queue, so that the data to be synchronized can be sent to the downstream system application based on the target task queue.

[0113] Specifically, in the upstream system, all task queues are defined, including the target task queue.

[0114] In this embodiment, based on the task assignments from the downstream system's applications, the task assignment queues for all task queues in the upstream system are determined, and each task queue is added to its corresponding task queue.

[0115] More specifically, tasks are popped from the task queue in the order they appear in the queue.

[0116] In this embodiment, tasks in each queue are processed by the task manager according to the queue order of the task queue, and the queue tasks of the task queue are popped out in sequence.

[0117] The task order can be preset or determined based on the task issuance time, task generation time, and task importance.

[0118] S303, Obtain the data synchronization status of the downstream system's applications.

[0119] In this embodiment, after the downstream system application completes data reception, it will return the data reception status, generate an ACK to confirm reception, and record and distribute logs based on the data synchronization status of the downstream system application.

[0120] Among these measures, logs are cleaned up regularly to reduce junk data.

[0121] S304. If the data synchronization status is "synchronization failed", an alarm message will be issued.

[0122] In this embodiment, if the data transmission fails, the task queue to which the data belongs is determined to be a failure queue, and a retransmission strategy is implemented for the failure queue. If multiple failures occur, the data synchronization status is determined to be unsuccessful, the task queue to which the data belongs is determined to be a dead-letter queue, and an alarm is issued for the dead-letter queue.

[0123] In this embodiment, task data is distributed based on the task queue, and data synchronization is achieved based on data processing methods such as failure retry mechanism, cleanup strategy and monitoring alarm, taking into account the consistency and integrity of the data, thereby ensuring the efficiency of data synchronization.

[0124] Figure 4 This is a flowchart illustrating the data synchronization mechanism provided in the embodiments of this application. Figure 2 and Figure 3 Based on the embodiments, such as Figure 4 As shown, the specific implementation steps of the data synchronization mechanism include:

[0125] S401. Perform identity classification for upstream and downstream systems.

[0126] The identity classification includes application type and data permissions.

[0127] S402. Set authentication rules on the management side, including data rules and authentication strategies.

[0128] Specifically, downstream application data permissions and data mapping rules are set and established based on identity, clarifying the data requirements of downstream applications to minimize the layering of upstream data content and achieve multi-dimensional mapping process management. Different data mapping rules can be set for different downstream applications based on the same API.

[0129] S403. The upstream system performs full data encapsulation, encrypts the encapsulated data, generates encrypted data packets, performs identity processing, generates a token corresponding to the encrypted data packets, thereby obtaining the data to be distributed, and sends the data to be distributed to the SDK.

[0130] S404. Downstream applications submit data authentication requests, request identity information, and perform permission authentication to confirm their data request permissions.

[0131] S405. Confirm the request information and write the data to disk according to the identity and role, so as to realize the confirmation of data granularity.

[0132] S406. The data engine performs data anonymization based on the established data rules, generates data tasks based on the downstream application's request content, processes the data tasks according to the distribution rules, and generates distribution tasks according to the authorization rules.

[0133] Among them, the distribution rules, also known as field management, can meet the different format requirements of different data receiving platforms, that is, process the distribution fields according to the requirements.

[0134] Authorization rules are a set of rules or policies that define and control access permissions to system resources (such as data, files, services, etc.). In the context of data anonymization and processing, authorization rules focus particularly on controlling access, use, and distribution permissions for anonymized data.

[0135] S407: The data engine adds the generated distribution tasks to the corresponding queues, processes the tasks in each queue through the task manager, and pops up the data distribution tasks in sequence.

[0136] The queues include message queues (mq) and memory queues.

[0137] S408 and Task Manager send data to the SDK based on a queue, and downstream applications receive data based on the SDK. The SDK receiving process requires identity processing, including authentication calls, and decryption and unpacking of encrypted data packets to achieve data reception.

[0138] After S409 and SDK complete data reception, they return the data reception status to the data engine. The data engine generates an ACK to confirm the reception. Based on the data reception status, the data engine records and distributes logs. The data engine periodically cleans up the logs to reduce redundant data.

[0139] S410: The Task Manager implements a failure resend policy for the failure queue. If the number of failures is too high, it is identified as a dead letter queue, and monitoring, alarms, or manual execution are performed.

[0140] S411. When metadata changes, it is tracked according to the mapping rules and the data field. If the field changes, the downstream configuration personnel need to redefine the data mapping relationship in the upstream system.

[0141] In this embodiment, by authenticating the data source, data synchronization process, and data target, unauthorized users or applications are effectively prevented from accessing the data. It considers not only data consistency and integrity but also data access permissions, achieving multi-dimensional data synchronization. By periodically updating the authentication policy, the scope and method of data synchronization can be flexibly adjusted according to business needs and security policies. This improves the flexibility, security, consistency, and targeting of data synchronization, thereby increasing its efficiency.

[0142] This application also provides a preferred embodiment, which implements multi-dimensional data distribution from the authentication center based on application authentication rules.

[0143] Step 1: Set the distribution rules, confirm the application of the distribution rules, and confirm the data distribution content.

[0144] Specifically, taking some user attributes as an example:

[0145] Name <name>,Phone number <phonenum>,gender <sex>,post <post>.

[0146] Step 2: Register the application at the certification center. After registration, set the rules for receiving data.

[0147] Specifically:

[0148] Application fields <== Authentication center fields.

[0149] Name <xingming>Name <name>.

[0150] gender <gender>Gender <sex>.

[0151] Role <role><==Job Position <post>.

[0152] Step 3: Data delivery. The user center packages the user attribute data and sends it to the data engine to confirm that data needs to be sent to the application.

[0153] Step 4: Perform authentication management for the application and confirm the application data settings.

[0154] Step 5: Regarding the authentication situation, confirm the data required by the application, including the name, gender, and position of the authentication center, and write the corresponding data to disk.

[0155] Step Six: Integrate and map data according to data requirement rules, and perform targeted data processing.

[0156] Step 7: Add the completed data generation tasks to the corresponding queue.

[0157] Step 8: Send the data to the downstream application for corresponding unpacking and processing, and the downstream data has completed data reception.

[0158] Step 9: Based on the data reception status, the downstream system returns data and completes the log recording.

[0159] In this embodiment, authentication is used to ensure multi-dimensional data synchronization process; authentication management ensures the stability and security of the data synchronization process, effectively avoiding data security issues caused by excessive data transmission, thereby improving the efficiency of data synchronization.

[0160] Figure 5 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of this application, as shown below. Figure 5 As shown, the device includes: a first acquisition module 51, a first processing module 52, a second acquisition module 53, a second processing module 54, a third processing module 55, a fourth processing module 56, and a fifth processing module 57.

[0161] The first acquisition module 51 is used to acquire the full data and encapsulate the full data to obtain multiple data packets.

[0162] The first processing module 52 is used to encrypt data packets and generate tokens for each data packet to obtain multiple encrypted data packets and tokens corresponding to each encrypted data packet.

[0163] The second acquisition module 53 is used to acquire data synchronization requests sent by applications in the downstream system, wherein the data synchronization requests include authentication information and request information.

[0164] The second processing module 54 is used to authenticate the applications of the downstream system based on the authentication information in order to determine the transmission rules.

[0165] The third processing module 55 is used to determine the target data to be transmitted based on the request information.

[0166] The fourth processing module 56 is used to process multiple encrypted data packets and the token corresponding to each encrypted data packet according to the target transmission data and transmission rules, so as to obtain the data to be synchronized.

[0167] The fifth processing module 57 is used to send the data to be synchronized to downstream system applications.

[0168] In one possible design, the authentication information includes at least one of the following: the system identifier of the downstream system, the application identifier of the application, and the user identifier corresponding to the downstream system.

[0169] Accordingly, the second processing module 54 is also used to obtain pre-configured authentication rules and authentication information, and to authenticate the applications of the downstream system. The authentication rules include at least one of the following: the correspondence between system identifier and transmission rules, the correspondence between application identifier and transmission rules, and the correspondence between user identifier and transmission rules.

[0170] The transmission rules include full transmission, partial transmission, encrypted transmission, and no transmission allowed.

[0171] In one possible design, the second processing module 54 is also used to display the rule configuration window;

[0172] In response to the user's configuration operation in the rule configuration window, determine the pre-configured authentication rules.

[0173] In one possible design, the fifth processing module 57 is also used to obtain the target task queue corresponding to the application of the downstream system;

[0174] Add the data to be synchronized to the target task queue, so that the data to be synchronized can be distributed to downstream system applications based on the target task queue.

[0175] In one possible design, the fifth processing module 57 is also used to determine the queued tasks of all task queues in the upstream system, wherein the task queues include the target task queue.

[0176] Pop tasks from the task queue in the order they appear in the queue.

[0177] In one possible design, the fifth processing module 57 is also used to obtain the data synchronization status of the application in the downstream system;

[0178] If the data synchronization status is "synchronization failed", an alarm will be issued.

[0179] In one possible design, the third processing module 55 is also used to perform data write-to-disk processing on the target transmission data according to the transmission rules.

[0180] The data synchronization device provided in this embodiment can execute a data synchronization method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0181] In a specific implementation of the aforementioned data synchronization method, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, thereby enabling the processor to execute the aforementioned data synchronization method.

[0182] Figure 6 This is a schematic diagram of a data synchronization device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory 62. The electronic device 60 also includes a communication component 63. The processor 61, memory 62, and communication component 63 are connected via a bus 64.

[0183] In a specific implementation, at least one processor 61 executes computer execution instructions stored in memory 62, causing at least one processor 61 to execute a data synchronization method according to claims 1-7 as executed by the data synchronization device side.

[0184] The specific implementation process of processor 61 can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here. In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0185] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0186] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0187] The above description of the functions implemented by the data synchronization device and the master control device illustrates the solutions provided by the embodiments of the present invention. It is understood that, in order to achieve the above functions, the data synchronization device or the master control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0188] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data synchronization method described above.

[0189] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0190] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0191] This application also provides a computer program product, including a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.

[0192] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0193] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / post> < / role> < / sex> < / gender> < / name> < / xingming> < / post> < / sex> < / phonenum> < / name>

Claims

1. A method of data synchronization, the method comprising: The method comprises the following steps: obtaining full data, and performing packet processing on the full data to obtain a plurality of data packets; performing encryption processing on the data packets, and generating a token for each data packet to obtain a plurality of encrypted data packets and a token corresponding to each encrypted data packet; obtaining a data synchronization request sent by an application of a downstream system, wherein the data synchronization request comprises authentication information and request information; authenticating the application of the downstream system according to the authentication information to determine a transmission rule; determining target transmission data according to the request information; performing data processing on the plurality of encrypted data packets and the token corresponding to each encrypted data packet according to the target transmission data and the transmission rule to obtain to-be-synchronized data; downloading the to-be-synchronized data to the application of the downstream system.

2. The method of claim 1, wherein, The authentication information comprises at least one of a system identifier of the downstream system, an application identifier of the application, and a user identifier corresponding to the downstream system. Accordingly, the authentication of the application of the downstream system according to the authentication information comprises: obtaining a preconfigured authentication rule and the authentication information, and authenticating the application of the downstream system, wherein the authentication rule comprises at least one of a corresponding relationship between a system identifier and a transmission rule, a corresponding relationship between an application identifier and a transmission rule, and a corresponding relationship between a user identifier and a transmission rule; wherein the transmission rule comprises full transmission, partial transmission, encrypted transmission, and disallowed transmission.

3. The method of claim 2, wherein, Before the authentication of the application of the downstream system according to the preconfigured authentication rule and the authentication information, the method further comprises: displaying a rule configuration window; determining a preconfigured authentication rule in response to a configuration operation of a user on the rule configuration window.

4. The method according to any one of claims 1 to 3, characterized in that, The downloading of the to-be-synchronized data to the application of the downstream system comprises: obtaining a target task queue corresponding to the application of the downstream system; adding the to-be-synchronized data to the target task queue to download the to-be-synchronized data to the application of the downstream system based on the target task queue.

5. The method of claim 4, wherein, The adding of the to-be-synchronized data to the target task queue to download the to-be-synchronized data to the application of the downstream system based on the target task queue comprises: determining a queue task of all task queues in an upstream system, wherein the task queues comprise the target task queue; sequentially popping the queue task of the task queue according to a queue order of the task queue.

6. The method of claim 5, wherein, After the adding of the to-be-synchronized data to the target task queue, the method further comprises: obtaining a data synchronization state of the application of the downstream system; if the data synchronization state is unsuccessful, issuing an alarm prompt.

7. The method according to any one of claims 1 to 3, characterized in that, After the determination of the target transmission data according to the request information, the method further comprises: performing data disk processing on the target transmission data according to the transmission rule.

8. A data synchronization apparatus, characterized by comprising: The method comprises the following steps: a first obtaining module is configured to obtain full data, and perform packet processing on the full data to obtain a plurality of data packets; The first processing module is configured to encrypt the data packets and generate a token for each data packet to obtain a plurality of encrypted data packets and a token corresponding to each encrypted data packet; The second acquisition module is configured to acquire a data synchronization request sent by an application of a downstream system, wherein the data synchronization request comprises authentication information and request information; The second processing module is configured to authenticate the application of the downstream system according to the authentication information to determine a transmission rule; The third processing module is configured to determine target transmission data according to the request information; The fourth processing module is configured to perform data processing on the plurality of encrypted data packets and the token corresponding to each encrypted data packet according to the target transmission data and the transmission rule to obtain to-be-synchronized data; The fifth processing module is configured to deliver the to-be-synchronized data to the application of the downstream system.

9. A data synchronization device, characterized by comprise: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data synchronization method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the data synchronization method of any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the data synchronization method of any one of claims 1 to 7.

Images