Data synchronization method, apparatus, device, storage medium, and computer program product

Abstract

Embodiments of the present application provide a data synchronization method and device, electronic equipment, computer readable storage medium and computer program product, comprising: in response to a data synchronization instruction, starting at least one data read thread and at least one data write thread, and establishing a communication connection between at least one data read thread and a data source; based on the communication connection, reading at least one piece of to-be-synchronized data from the data source through at least one data read thread, and storing the read to-be-synchronized data in a cache space; wherein the cache space is used to cache the to-be-synchronized data read by the data read thread; when the cache space stores the to-be-synchronized data, writing the to-be-synchronized data cached in the cache space into a local database through at least one data write thread. Through the embodiments of the present application, the efficiency and stability of data synchronization can be improved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data synchronization method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Most data synchronization solutions in related technologies adopt a single process to handle the full synchronization steps of data reading, filtering, mapping, fusion, transformation and writing to the local data source. However, using a single process to handle the entire synchronization process of a single piece of data will cause the entire user data synchronization process to need to constantly interact with the upstream data source. If each piece of data fails during the synchronization process, it is necessary to reread the data from the upstream data source, resulting in low stability. At the same time, when the data volume is large, it will also lead to a long life cycle of the process from creation to release, resulting in low data synchronization efficiency. Summary of the Invention

[0003] This application provides a data synchronization method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the efficiency and stability of data synchronization.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data synchronization method, including:

[0006] In response to a data synchronization command, at least one data read thread and at least one data write thread are started, and a communication connection is established between the at least one data read thread and the data source.

[0007] Based on the communication connection, at least one piece of data to be synchronized is read from the data source through the at least one data read thread, and the read data to be synchronized is stored in the cache space;

[0008] The cache space is used to cache the data to be synchronized read by the data read thread;

[0009] When the cache space contains the data to be synchronized, the cached data to be synchronized in the cache space is written to the local database through the at least one data write thread.

[0010] This application provides a data synchronization device, including:

[0011] The startup module is used to start at least one data read thread and at least one data write thread in response to a data synchronization command, and to establish a communication connection between the at least one data read thread and the data source.

[0012] A reading module is configured to read at least one piece of data to be synchronized from the data source through the at least one data reading thread based on the communication connection, and store the read data to be synchronized into a cache space; wherein, the cache space is used to cache the data to be synchronized read by the data reading thread;

[0013] The write module is used to write the data to be synchronized, which is cached in the cache space, to the local database through the at least one data write thread when the cache space stores the data to be synchronized.

[0014] In the above scheme, the device further includes a first detection module, which is used to detect the cache space based on the data write thread to obtain a first detection result; the write module is further used to write the cached data to be synchronized in the cache space into a local database through the at least one data write thread when the first detection result indicates that the cache space stores the data to be synchronized.

[0015] In the above scheme, the device further includes a first comparison module, which is used to obtain the total amount of the data to be synchronized in the data source and the amount of the data to be synchronized read from the data source; compare the total amount and the amount of read to obtain a first comparison result; when the first comparison result indicates that there is no data to be synchronized in the data source, close the communication connection between the at least one data reading thread and the data source, and close the at least one data reading thread.

[0016] In the above scheme, the data to be synchronized cached in the cache space is unprocessed data read from the data source. The writing module is further configured to filter the data to be synchronized based on the data object of the data to be synchronized to obtain target synchronization data; obtain a pre-set attribute mapping relationship, and perform attribute mapping on the target synchronization data based on the attribute mapping relationship to obtain general data corresponding to the data attributes of the local database; and write the general data into the local database through the at least one data writing thread.

[0017] In the above scheme, the device further includes a verification module, which is used to perform format verification on the target synchronization data and obtain a verification result; when the verification result indicates that the target synchronization data passes the verification, the process proceeds to the step of obtaining the pre-set attribute mapping relationship; when the verification result indicates that the target synchronization data fails the verification, the target synchronization data is recorded to an error log.

[0018] In the above scheme, the number of data writing threads is at least two, the cache space is a message queue, and the writing module is further used to have the at least two data writing threads sequentially extract the cached data to be synchronized from the message queue in a parallel processing manner; and write the extracted data to be synchronized into the local database.

[0019] In the above scheme, the device further includes a second detection module, which is used to detect the writing result of the data to be synchronized and obtain a second detection result; when the second detection result indicates that the data to be synchronized has been written to the local database, the device obtains the total cached amount of the data to be synchronized in the cache space and the amount of data to be synchronized written from the cache space to the local database; the device compares the total cached amount with the amount of data written to obtain a second comparison result; when the second comparison result indicates that the data to be synchronized does not exist in the cache space, the device closes the at least one data writing thread.

[0020] In the above scheme, the device further includes a third detection module, which is used to detect the number of times the data to be synchronized has been written when the second detection result indicates that the data to be synchronized has not been written to the local database, and obtain a third detection result; when the third detection result indicates that the number of times the data to be synchronized has been written has not reached the write count threshold, the data to be synchronized is written to the local database through the at least one data write thread.

[0021] In the above scheme, the third detection module is further configured to stop writing to the data to be synchronized when the third detection result indicates that the number of times the data to be synchronized has been written has reached the write count threshold, and to record the data to be synchronized to the error log.

[0022] In the above scheme, the device further includes a recording module, which is used to shut down the at least one data write thread; obtain the start time of the at least one data read thread and the at least one data write thread, and the shutdown time of the at least one data read thread and the at least one data write thread; and record the start time and the shutdown time.

[0023] In the above scheme, the device further includes an incremental data synchronization module, which is used to respond to an incremental data synchronization command, acquire the recorded start time and shutdown time; determine the incremental data to be synchronized based on the start time and shutdown time, and perform data synchronization on the incremental data to be synchronized.

[0024] This application provides an electronic device, including:

[0025] Memory, used to store executable instructions;

[0026] The processor, when executing executable instructions stored in the memory, implements the data synchronization method provided in the embodiments of this application.

[0027] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the data synchronization method provided in this application.

[0028] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data synchronization method provided in this application.

[0029] The embodiments of this application have the following beneficial effects:

[0030] Upon receiving a data synchronization command, a data read thread and a data write thread are started. The data read thread caches the data to be synchronized read from the data source into a cache space, allowing the data write thread to retrieve the cached data to be synchronized from the cache space and write it to the local database. In this way, the data synchronization process is split into two asynchronous threads: data reading and data writing. This reduces dependence on the data source, decouples the data source from the local database, and improves the stability of the data synchronization process. Furthermore, the introduction of a cache space allows the data read and data write threads to run concurrently, thereby maximizing data synchronization efficiency. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the architecture of the data synchronization system 100 provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0033] Figure 3 This is a flowchart illustrating the data synchronization method provided in an embodiment of this application;

[0034] Figure 4 This is the human-computer interaction interface for data synchronization provided in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram illustrating a specific application scenario provided in the embodiments of this application;

[0036] Figure 6This is a schematic diagram illustrating a specific application scenario provided in the embodiments of this application;

[0037] Figure 7 This is a schematic diagram illustrating the deployment of data synchronization provided in the embodiments of this application;

[0038] Figure 8 This is a flowchart illustrating the data synchronization method provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the user data synchronization process provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram illustrating the asynchronous parallelism of read and write processes provided in an embodiment of this application;

[0041] Figure 11 This is a flowchart illustrating the data synchronization process provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0047] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0048] 1) Client: Also known as user terminal, it refers to the program that provides local services to users in contrast to the server. Except for some applications that can only run locally, it is generally installed on ordinary client machines and needs to cooperate with the server to run. That is, there needs to be a corresponding server and service program in the network to provide the corresponding services. Thus, a specific communication connection needs to be established between the client and the server to ensure the normal operation of the application.

[0049] 2) Data source (upstream data source): This refers to the data provider in the data synchronization process, often an existing user directory service of the enterprise, such as Active Directory or OpenLDAP. The data source provides data query interfaces based on standard protocols such as LDAP and APIs, allowing external systems to retrieve all user data through these interfaces.

[0050] 3) Local data source (local database) refers to the database where the final data is stored in the user data synchronization process. This patent is applied to general directory services and uses MongoDB as the database type.

[0051] 4) Raw Data refers to user data entries that are retrieved through the data query interface provided by the upstream user source and have not undergone any processing.

[0052] 5) Active Directory is a centralized directory management service for large network environments in an architecture. It acts as a centralized repository and stores all data related to objects such as Active Directory users, computers, servers, and other resources within the organization.

[0053] 6) OpenLDAP (Open Lightweight Directory Access Protocol) is a commonly used directory service that provides all the functions of a directory service, including directory search, authentication, secure channels, and filters.

[0054] 7) Message Queue (MQ): A queue that stores data to be synchronized in a first-in, first-out manner. MQ is used to balance the differences in processing speed at each stage of the message transmission path in order to provide stable and reliable message transmission processing.

[0055] 8) Lightweight Directory Access Protocol (LDAP) is an open, neutral, industry-standard application protocol that provides access control and maintains distributed directory information via the IP protocol.

[0056] 9) An application programming interface (API) is a set of predefined functions or conventions for connecting different components of a software system to enable communication and data exchange between two separate software systems.

[0057] 10) ForgeRock Directory Service is a solution for full synchronization of user account data. The core technology lies in the use of the self-developed LDAP-type local database OpenDJ. It matches the most common user data source types in enterprises as much as possible in terms of transmission protocol, data format, data structure, data schema, and read / write rules, such as Active Directory and OpenLDAP.

[0058] 11) Keep-alive is a built-in component in Vue that allows contained components to retain their state and avoid being re-rendered.

[0059] 12) Vue, a progressive front-end framework used to improve front-end development efficiency.

[0060] The inventors discovered that data synchronization technologies in related fields, such as ForgeRock Directory Service, aim to reduce the data processing workload required for user data synchronization by ensuring that the local data source and the upstream data source use the same or similar data source types. This significantly reduces the processing workload required by ForgeRock Directory Service for the read raw data. Consequently, when the upstream data source uses a standard configuration, each piece of data can be directly stored in the local data source after simple attribute mapping. In other words, ForgeRock Directory Service primarily optimizes the workload during data processing and writing, shortening the total time required for data synchronization.

[0061] However, the above data synchronization technology has the following problems: First, it uses a single process to handle the entire synchronization process of a single piece of data—reading, filtering, mapping, merging, transforming, and writing. That is, data reading and writing are treated as a single process. Thus, during the entire user data synchronization process, ForgeRock Directory... The service needs to continuously interact with upstream data sources and constantly read data, which places high demands on the stability of the network and upstream data sources. When the user data volume is large and the user synchronization cycle is long, synchronization failures are likely to occur, resulting in multiple retries or only partial data synchronization. Second, if an error occurs during the synchronization of each piece of user data, data needs to be read from the upstream data source again, increasing the burden on the network and upstream data sources. Third, using a single process to handle the entire synchronization process for a single piece of data results in a long process lifecycle from creation to release, reducing the efficiency of multi-threaded parallelism. Fourth, using a single process to handle the entire synchronization process for a single piece of data results in a long process lifecycle from creation to release. When reading data, if the connection established with the upstream data source cannot be closed in time, it will cause waste of connection pool resources. Fifth, there are many restrictions on the type of local data source. When the local data source and the upstream data source use different types of databases, or when there is data from multiple upstream data sources, the synchronization efficiency drops significantly. Sixth, when the amount of user data to be synchronized is large, it is necessary to close the network for maintenance of all supporting systems to ensure that there are no data changes or network interruptions during the synchronization period, resulting in low flexibility.

[0062] Based on this, embodiments of this application provide a data synchronization method, apparatus, electronic device, computer-readable storage medium, and computer program product, which optimizes traditional data synchronization methods by splitting the data synchronization process into two asynchronous processes: data reading and data writing, which are performed simultaneously. Furthermore, paginated reading is used for upstream data sources of different types and performance levels. To optimize reading efficiency, the raw data read from the upstream data source is not processed but instead stored in a message queue (MQ) through a caching mechanism. Then, the raw data in the MQ is filtered, mapped, merged, and transformed during the data writing process before being written to the local database. In this way, by utilizing high-performance caching and the MQ mechanism, asynchronous parallelism between data reading, data processing, and writing is achieved. While ensuring that the overall synchronization time remains relatively unchanged, the interaction with the upstream data source is minimized, thereby reducing dependence on and impact on the external environment and system, reducing retries and error rates, and improving user data synchronization efficiency.

[0063] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the data synchronization system 100 provided in this application embodiment. To realize the application scenario of data synchronization (for example, the application scenario of data synchronization may be that after the user opens the client for data synchronization, the user determines the data to be synchronized based on the presented human-computer interaction interface, and triggers the data synchronization instruction for the data to be synchronized, so that the server can synchronize the data selected by the user upon receiving the data synchronization instruction), the terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 may be a wide area network or a local area network, or a combination of both. The terminal 400 is used for the user to use the client 401 and display on the display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other through a wired or wireless network.

[0064] The terminal 400 is used to generate a data synchronization instruction in response to a synchronization confirmation operation for the data to be synchronized, and send the data synchronization instruction to the server 200.

[0065] Server 200 is used to receive data synchronization instructions sent by terminal 400; in response to the data synchronization instructions, start at least one data read thread and at least one data write thread, and establish a communication connection between at least one data read thread and the data source; based on the communication connection, read at least one piece of data to be synchronized from the data source through at least one data read thread, and store the read data to be synchronized into a cache space; wherein, the cache space is used to cache the data to be synchronized read by the data read thread; when the cache space stores data to be synchronized, write the cached data to be synchronized in the cache space into the local database through at least one data write thread.

[0066] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, set-top box, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, smart speakers, and smartwatches), but is not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0067] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. In practical applications, the electronic device can be... Figure 1 The server 200 or terminal 400 shown are described in the following document. Figure 2 , Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0068] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0069] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0070] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0071] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0072] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0073] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0074] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0075] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430.

[0076] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0077] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data synchronization device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a startup module 4551, a reading module 4552, and a writing module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0078] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the data synchronization apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data synchronization method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0079] In some embodiments, the terminal or server can implement the data synchronization method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0080] Based on the above description of the data synchronization system and electronic device provided in the embodiments of this application, the data synchronization method provided in the embodiments of this application is described below. In actual implementation, the data synchronization method provided in the embodiments of this application can be implemented by a terminal or a server alone, or by a terminal and a server working together, so that... Figure 1 The following description uses the example of server 200 executing the data synchronization method provided in this embodiment of the application independently. See also... Figure 3 , Figure 3 This is a flowchart illustrating the data synchronization method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0081] Step 101: In response to the data synchronization command, the server starts at least one data read thread and at least one data write thread, and establishes a communication connection between at least one data read thread and the data source.

[0082] In practice, the server first detects the data synchronization command sent by the terminal. When the server detects a data synchronization command for the data to be synchronized, it responds by starting at least one data read thread and at least one data write thread.

[0083] It should be noted that the data synchronization command can be triggered by the user. Specifically, the terminal has a client installed for data synchronization, and in response to a trigger operation on that client, a human-computer interaction interface for implementing data synchronization is presented. For example, see [link to example]. Figure 4 , Figure 4 This application provides a human-computer interaction interface for data synchronization, based on... Figure 4The dashed box 401 contains the data source's link address, username, and corresponding password. The dashed box 402 contains the function to determine which data source to synchronize. The terminal responds to input operations in the input fields of dashed box 401, receiving the data source's link address, username, and corresponding password, and verifying the username and password. If the username and password pass verification, the data source to be synchronized is determined. Then, in response to trigger operations such as clicks on the function items in dashed box 401, a data synchronization command is generated and sent to the server. It should be noted that the data synchronization type (e.g., full synchronization, incremental synchronization), data synchronization frequency (e.g., per minute, hour, day, week, month), and custom data mapping and processing logic can be determined based on the human-computer interaction interface for data synchronization, and can also be customized to adapt to specific data to be synchronized.

[0084] As an example, see Figure 5 , Figure 5 This is a schematic diagram illustrating a specific application scenario provided in the embodiments of this application, based on Figure 5 When enterprises need to gradually synchronize data, during the migration (synchronization) period, some data from migrated applications has been written to the local database, while data from non-migrated applications remains in upstream data sources such as Active Directory. During this transition period, enterprises achieve efficient data synchronization by splitting the data synchronization process into two asynchronous threads: data reading and data writing. This ensures consistency between the existing and new data sources, preventing application unavailability due to inconsistencies.

[0085] As another example, see Figure 6 , Figure 6 This is a schematic diagram illustrating a specific application scenario provided in the embodiments of this application, based on Figure 6 When the data to be synchronized is related to risk control products, it is necessary to periodically synchronize data from the enterprise's data source to support the product's business functions, such as... Figure 6 As shown, the system first integrates multiple existing data sources of the enterprise, such as AD clusters and LDAP clusters, and then synchronizes the data from the integrated multiple data sources to the local database. Based on the local database, it supports terminal authentication for multiple operating systems, such as Windows and MacOS.

[0086] In practice, upon receiving a data synchronization command, at least one data read thread and at least one data write thread are started. When there are multiple data read threads, a parallel processing approach is used to establish a communication connection between at least one data read thread and the data source. It should be noted that the communication connection between at least one data read thread and the data source is kept-alive, i.e., multi-threaded reuse. This prevents frequent opening and closing of communication connections with the data source, which could lead to connection pool overflow or unnecessary resource consumption.

[0087] Step 102: Based on the communication connection, at least one piece of data to be synchronized is read from the data source through at least one data read thread, and the read data to be synchronized is stored in the cache space; wherein, the cache space is used to cache the data to be synchronized read by the data read thread.

[0088] In practical implementation, when parallel processing is used for each data read thread, after establishing a communication connection between at least one data read thread and the data source, the process of reading at least one piece of data to be synchronized from the data source through at least one data read thread based on the communication connection specifically includes: based on the communication connection, each data read thread reads at least one piece of data to be synchronized page by page from the data source, and then stores the read data to be synchronized into the cache space. It should be noted that the data to be synchronized page by page here can be raw data, and the cache space can be a message queue. Therefore, for the process of reading at least one piece of data to be synchronized page by page from the data source, there is no need to perform format checking, data processing, or other operations on the read raw data; the read raw data can be directly stored into the message queue.

[0089] It's worth noting that caching can take various forms; here, a message queue is used to leverage its multi-device read / write and scalability features. Other high-performance caches, such as Redis, can also be employed.

[0090] In some embodiments, after the data reading process is completed, the communication connection with the data source can be actively closed. Specifically, the total amount of data to be synchronized in the data source and the amount of data to be synchronized read from the data source are obtained; the total amount and the amount of data read are compared to obtain a first comparison result; when the first comparison result indicates that there is no data to be synchronized in the data source, the communication connection between at least one data reading thread and the data source is closed, and at least one data reading thread is closed. In this way, the connection pool resources of the data source are released to reduce the impact of data synchronization on the data source business.

[0091] Step 103: When there is data to be synchronized stored in the cache space, the cached data to be synchronized in the cache space is written to the local database through at least one data write thread.

[0092] In practical implementation, before writing the cached data to be synchronized to the local database through at least one data write thread, it is first checked whether the cached space contains the data to be synchronized. If the cached space contains the data to be synchronized, then at least one data write thread writes the cached data to the local database. The process of checking whether the cached space contains the data to be synchronized specifically includes: based on the data write thread, checking the cached space to obtain a first detection result; and when the first detection result indicates that the cached space contains the data to be synchronized, then at least one data write thread writes the cached data to the local database. It should be noted that this detection can be performed in real-time or periodically, and the detection period can be set according to actual needs, such as 2 seconds.

[0093] In some embodiments, when the number of data write threads is at least two and the cache space is a message queue, the process of writing the cached data to be synchronized in the cache space to the local database by at least one data write thread specifically includes: at least two data write threads sequentially extracting the cached data to be synchronized from the message queue in parallel processing; and writing the extracted data to be synchronized to the local database.

[0094] In practice, the process of writing the data to be synchronized cached in the cache space to the local database using at least one data write thread includes: transforming the data to be synchronized to obtain general data; and writing the general data to the local database using at least one data write thread. Here, the data to be synchronized cached in the cache space is unprocessed data read from the data source.

[0095] In practice, the process of transforming the data to be synchronized into general data includes: filtering the data to be synchronized based on the data object to obtain the target data to be synchronized; obtaining the pre-set attribute mapping relationship; and performing attribute mapping on the target data to obtain general data corresponding to the data attributes of the local database.

[0096] It should be noted that the process of filtering the data to be synchronized based on the data objects to obtain the target synchronized data can be based on pre-defined filters, such as users, user groups, or organizations. For example, when synchronizing data associated with a specific organization, the pre-defined filters can be used to select the target organization's corresponding data to be synchronized from multiple organizations or user groups. Conversely, the process of mapping the target synchronized data to attributes based on attribute mapping relationships to obtain general data corresponding to the local database's data attributes can be based on the target synchronized data. This can be done by reading the attribute mapping relationships corresponding to different objects, and then mapping the data object's attributes based on these pre-defined relationships. For example, when the target synchronized data is "SamAccount," the pre-defined attribute mapping relationships can be used to map the data object's attributes to obtain general data corresponding to the local database's data attributes, i.e., "UserName," thus determining that the target synchronized data is the corresponding username.

[0097] In some embodiments, after filtering the data to be synchronized and obtaining the target synchronization data, the target synchronization data can also be format-validated to obtain a validation result. When the validation result indicates that the target synchronization data passes the validation, the process proceeds to the step of obtaining a pre-set attribute mapping relationship. When the validation result indicates that the target synchronization data fails the validation, the target synchronization data is recorded in the error log.

[0098] It's important to note that the process of validating the format of the target synchronized data and obtaining the validation result is crucial. Different data sources correspond to different data formats; for example, the local database might use XML format while the data source might use JSON format. Therefore, validating the target synchronized data prevents the local database from encountering data corruption issues due to its inability to read the synchronized data in the correct format. Here, if the validation result indicates that the target synchronized data has failed validation, the synchronization of that data entry is skipped, and the data entry is recorded in the error log.

[0099] In practical implementation, the process of writing general data to the local database using at least one data write thread specifically includes: establishing a communication connection between at least one data write thread and the local database; and, based on the communication connection, writing the general data to the data table of the local database using at least one data write thread. Here, the communication connection between at least one data write thread and the local database is kept-alive, i.e., multi-threaded reuse. This prevents frequent opening and closing of the communication connection with the local database, which could cause connection pool overflow or unnecessary resource consumption.

[0100] In some embodiments, after writing the data to be synchronized cached in the cache space to the local database using at least one data write thread, the write result of the data to be synchronized can be checked to obtain a second detection result. When the second detection result indicates that the data to be synchronized has been written to the local database, the total cached amount of the data to be synchronized cached in the cache space and the amount of data to be synchronized written from the cache space to the local database are obtained. The total cached amount and the amount of data written are compared to obtain a second comparison result. When the second comparison result indicates that there is no data to be synchronized in the cache space, at least one data write thread is closed. When the second comparison result indicates that there is data to be synchronized in the cache space, the data to be synchronized cached in the cache space is written to the local database using at least one data write thread. In this way, after the data processing and writing process is completed, the communication connection established with the local database due to data synchronization is actively closed, and the connection pool resources of the local database are released to reduce the impact of data synchronization on product business.

[0101] In practice, when the second detection result indicates that the data to be synchronized has not been written to the local database, the number of write attempts for the data to be synchronized is checked to obtain a third detection result. When the third detection result indicates that the number of write attempts for the data to be synchronized has not reached the write attempt threshold, the data to be synchronized is written to the local database through at least one data write thread. When the third detection result indicates that the number of write attempts for the data to be synchronized has reached the write attempt threshold, the writing of the data to be synchronized is stopped, and the data to be synchronized is recorded in the error log. It should be noted that the write attempt threshold here is preset, such as 3 times, 5 times, etc., so that when the number of write attempts for the data to be synchronized has not reached the write attempt threshold, the writing of the data to be synchronized is retried.

[0102] In some embodiments, after writing the cached data to be synchronized to the local database using at least one data write thread, at least one data write thread can be shut down; the start time and shutdown time of at least one data read thread and at least one data write thread can be obtained; and the start time and shutdown time can be recorded. After recording the start time and shutdown time, these times are used in subsequent incremental data synchronization. Specifically, in response to an incremental data synchronization command, the recorded start time and shutdown time are obtained; based on the start time and shutdown time, the incremental data to be synchronized is determined, and data synchronization is performed on the incremental data to be synchronized.

[0103] In some embodiments, the process of detecting the cache space can also be performed by a data detection thread. Specifically, in response to a data synchronization instruction, at least one data read thread and a data detection thread are started, and a communication connection is established between the at least one data read thread and the data source. Based on the communication connection, at least one piece of data to be synchronized is read from the data source through the at least one data read thread, and the read data to be synchronized is stored in the cache space. Based on the data detection thread, the cache space is detected to obtain the corresponding detection results. When it is determined that there is data to be synchronized stored in the cache space based on the corresponding detection results, at least one data write thread is started, and the data to be synchronized cached in the cache space is written to the local database through the at least one data write thread.

[0104] It should be noted that, regarding the deployment model of data synchronization, from a network and deployment perspective, data synchronization falls under the operations and maintenance plane and consumes significant system resources. Therefore, it is deployed on separate nodes for the business and management planes. See [link / reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the deployment form of data synchronization provided in the embodiments of this application, based on Figure 7 Data synchronization nodes have independent computing resources, thereby reducing the impact on business and management functions, and also preventing the collapse of the entire system due to resource overflow.

[0105] The data synchronization method provided in the embodiments of this application will be further described below. See also... Figure 8 , Figure 8 This is a flowchart illustrating the data synchronization method provided in the embodiments of this application, based on... Figure 8 The data synchronization method provided in this application embodiment is implemented collaboratively by the client and the server.

[0106] Step 201: In response to the trigger operation for the data synchronization determination function item, the client sends a data synchronization command to the server.

[0107] In practical implementation, the client can be a data synchronization client set up on the terminal. Specifically, the terminal has a client installed for data synchronization. In response to a trigger operation on the client, a human-computer interaction interface for data synchronization is presented. Here, the human-computer interaction interface presents input fields for inputting the link address of the data source, username, and corresponding user password. Thus, the data synchronization command can be automatically generated by the client based on certain trigger conditions. For example, after the client obtains the link address of the data source, username, and corresponding user password input by the user, it automatically generates a data synchronization command for the corresponding data source. Alternatively, it can be generated by the user triggering a corresponding confirmation function item based on the client's human-computer interaction interface. For example, the human-computer interaction interface also presents a data synchronization confirmation function item for confirming the synchronization of the corresponding data source. Thus, the user can select the data source to be synchronized based on the client's human-computer interaction interface, and then generate a data synchronization command in response to the trigger operation of the data synchronization confirmation function item. Alternatively, it can be sent to the client by other devices that are connected to the terminal.

[0108] Step 202: In response to the data synchronization command, the server starts at least one data read thread and at least one data write thread, and establishes a communication connection between at least one data read thread and the data source.

[0109] Step 203: Based on the communication connection, at least one piece of data to be synchronized is read from the data source through at least one data read thread, and the read data to be synchronized is stored in the cache space.

[0110] Step 204: Based on the data write thread, the cache space is detected to obtain the first detection result. When the first detection result indicates that there is data to be synchronized stored in the cache space, the data to be synchronized is filtered based on the data object of the data to be synchronized to obtain the target synchronization data.

[0111] Step 205: Perform format verification on the target synchronization data, obtain the verification result, and when the verification result indicates that the target synchronization data has passed the verification, obtain the pre-set attribute mapping relationship.

[0112] Step 206: Based on the attribute mapping relationship, perform attribute mapping on the target synchronization data to obtain general data.

[0113] Step 207: Write general data into a data table in the local database using at least one data write thread.

[0114] Step 208: The server generates a notification message indicating successful synchronization.

[0115] Step 209: The server sends a notification message to the client.

[0116] Step 210: The client receives the prompt message sent by the server and displays the prompt message.

[0117] In practice, the client can display prompt messages in its human-computer interaction interface, save the prompt messages locally on the terminal, or send the prompt messages to other devices that are connected to the terminal.

[0118] By applying the embodiments described above, upon receiving a data synchronization command, a data read thread and a data write thread are started. The data read thread caches the data to be synchronized read from the data source into a cache space, allowing the data write thread to retrieve the cached data to be synchronized from the cache space and write the retrieved data to be synchronized into the local database. In this way, the data synchronization process is split into two asynchronous threads: data reading and data writing. This reduces dependence on the data source, decouples the data source from the local database, and improves the stability of the data synchronization process. Furthermore, the introduction of a cache space allows the data read thread and the data write thread to run concurrently, thereby maximizing data synchronization efficiency.

[0119] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0120] Most data synchronization solutions in related technologies employ a single process to handle the full user data synchronization steps, including reading, filtering, mapping, merging, transforming, and writing to a local data source (local database). This approach is inefficient in large enterprises (with over 100,000 employees) and scenarios with complex upstream data sources (data sources), typically requiring network closure and maintenance to complete full user data synchronization. Furthermore, these solutions use the same or similar local data source type as the upstream data source to reduce the data processing workload during user data synchronization. However, upstream data sources are often Active Directory or OpenLDAP types, which have limitations in security, scalability, flexibility, and stability. If the product uses a local database similar to the upstream data source for storage, it will restrict and impact other business functions of the product.

[0121] Based on this, embodiments of this application provide a data synchronization method, apparatus, electronic device, computer-readable storage medium, and computer program product. First, the user data (data to be synchronized) is split into two asynchronous processes, which shortens the life cycle of each process, allows computing resources to be released in a timely manner, and reduces errors caused by the local system not responding to processes. Simultaneously, after completing the read and write operations, the connections (communication connections) established with the upstream data source and the local database can be closed promptly, quickly releasing connection pool resources and reducing the impact of user data synchronization on business operations. Secondly, by introducing a message queue (caching space) as an intermediate cache, the general directory service (local database) can be extended to support unified filtering, mapping, fusion, and transformation of raw data from multiple data sources, effectively improving the synchronization efficiency of multiple data sources. At the same time, by centralizing computer resources to complete the reading of user data and storing it in the message queue, in subsequent data processing and writing processes (data write thread), if errors occur, there is no need for frequent interaction and retries with the upstream data source, further reducing the dependence on the upstream data source and the enterprise network. Moreover, it greatly shortens the interaction time with the upstream data source in the user data synchronization process, reduces retries and resource consumption caused by network and upstream data source instability, and improves the efficiency of user data synchronization.

[0122] Next, this technical solution will be described in detail. The following section explains the technical solution from a product perspective. This technical solution can be applied to user directory service products, in scenarios where the integration, upgrading, and migration of an enterprise's existing user data sources are required, such as... Figure 5 As shown, to ensure the smooth operation of enterprise business during the transition and migration process, it is necessary to periodically synchronize user data between the existing and new user directory services. Specifically, enterprises need to use a general directory service product to gradually replace their existing Active Directory, that is, to synchronize Active Directory data to the local database. During the migration, some migrated applications use the general directory service product as their user data source, while some non-migrated applications continue to use Active Directory. During this transition period, enterprises need efficient user data synchronization capabilities to ensure the consistency of the existing and new user data sources; otherwise, enterprise applications will become unavailable. Additionally, as... Figure 6 As shown, this technical solution can also be applied to business and application-based risk control products, thereby periodically synchronizing data from the enterprise's existing user data sources to support the product's business functions. For example, it can first integrate multiple existing data sources within the enterprise, forming a unified, universal directory service to support endpoint authentication for platforms including Windows and macOS.

[0123] In actual implementation, when data synchronization is required, the system administrator can open the synchronization management interface through the management console to define the parameters in the data synchronization process, such as the connection method of the upstream data source, the data synchronization type such as full synchronization or incremental synchronization, the data synchronization frequency cycle such as per minute, per hour, per day, per week, per month, etc., as well as custom data mapping and processing logic to adapt to the enterprise's unique user data.

[0124] The technical solution of this application will now be described from a technical perspective. (See also...) Figure 9 , Figure 9 This is a schematic diagram of the user data synchronization process provided in the embodiments of this application, based on Figure 9 User data bidirectional synchronization mainly involves four functions: data initialization, full synchronization, incremental synchronization, and real-time writing. For the full data synchronization process, when the amount of data to be synchronized is large, the dependence of user data synchronization on the network and upstream data sources is reduced, thereby improving synchronization efficiency.

[0125] It should be noted that the full user data synchronization technology in related technologies requires stable interaction with the upstream data source throughout the entire synchronization process, placing high demands on the stability of the upstream data source and the network. Therefore, see [link to relevant documentation]. Figure 10 , Figure 10 This is a schematic diagram of asynchronous parallel read / write processes provided in an embodiment of this application, based on... Figure 10 This application optimizes the bottleneck in the full user data synchronization process by implementing asynchronous read-write separation, thereby shortening the interaction time with upstream data sources and improving synchronization efficiency. For the specific implementation process of the user data synchronization technology that asynchronously separates data reading from processing and writing, please refer to [link to relevant documentation]. Figure 11 , Figure 11 This is a flowchart illustrating the data synchronization process provided in an embodiment of this application, based on... Figure 11The data synchronization method provided in this application embodiment is implemented by steps 1101 to 1108 and steps 1201 to 1212. Here, steps 1101 to 1108 and steps 1201 to 1212 are two asynchronous processes. First, when the data synchronization event is triggered, the user reading process (data read thread) is started. Here, the maximum number of user data entries read is 1000 (based on the default maximum number of returned entries of a standard LDAP data source). Then, a connection is established with the upstream data source to read the user data in pages. Then, the read raw data is... Data (data to be synchronized) is directly stored in the message queue (cache space). Here, there is no need to perform format checks or data processing on the read RawData. After the data reading process is completed, the connection with the upstream data source is actively closed, releasing the connection pool resources of the upstream data source to reduce the impact of data synchronization on the upstream data source business. At the same time, when a user synchronization event is triggered, the message queue detection process is started. When the message queue is not empty, the data processing and writing process (data write thread) is started, thus enabling multi-threaded parallelism. Here, the maximum number of parallel threads is dynamically adjusted based on the system performance configuration. Each process sequentially reads one line of Raw Data from the message queue, processes it, and writes it to the local database.

[0126] In practical implementation, the data processing procedure involves several steps. First, data is filtered based on a pre-defined filter. Common user data objects include users, user groups, and organizations. Then, based on the filtering results, the attributes of the data objects are validated for their data formats. If the data format is incorrect, the synchronization of that data is skipped and recorded in the ErrorLog. When the filtering results pass validation, the attribute mappings (attribute mapping relationships) corresponding to different objects are read. Based on pre-defined mapping rules, the attributes of the data objects are transformed, thus completing the data processing. It should be noted that if the mapping rules include attributes that require mapping based on preconditions of the synchronized data, the synchronization of that attribute is skipped until the precondition data is synchronized.

[0127] In actual implementation, for the data writing process, specifically, after the data transformation is completed, a connection to the local database is initiated, and the data is written to the table of the corresponding data object. The connection to the local database adopts Keep Alive, that is, multi-threaded reuse, so as to prevent frequent opening and closing of the connection with the local database, which would cause connection pool overflow or unnecessary resource consumption. When an error occurs during data writing, if the data writing cannot be completed after 3 retries, the data is skipped and recorded in the Error Log. After the data processing and writing process is completed, the connection with the local database established due to data synchronization is actively closed to release the connection pool resources of the local database, so as to reduce the impact of data synchronization on product business.

[0128] Finally, when the asynchronous data reading, processing, and writing are all completed, the start and end times of data synchronization are recorded for use in subsequent incremental synchronization. Then, the synchronization log is output to complete the user synchronization.

[0129] It's important to note that the core of this solution lies in separating data reading and data processing / writing into two asynchronous processes, which are then parallelized through an intermediate cache. This intermediate cache can take various forms; here, a message queue is used to leverage its multi-device read / write and scalability features. Alternatively, other high-performance caches, such as Redis, can also be employed.

[0130] In practical implementation, due to different product deployment models and varying patterns of multi-threaded reading and writing of user data, distributed multi-threading can be used to distribute the workload across multiple nodes in the cluster. Alternatively, a dedicated user synchronization node can be used to centralize multi-threading on a single node specifically for user synchronization. Figure 7 As shown.

[0131] It should be noted that user data synchronization should support common upstream data sources and retrieve user data through standard protocols. The user data synchronization protocol used in this solution can be the LDAP protocol, or it can use multi-threaded calls to the upstream data source API to retrieve user data, and use pagination queries that comply with the number of parallel threads and the number of returned data entries supported by the upstream data source API.

[0132] By applying the embodiments described above, upon receiving a data synchronization command, a data read thread and a data write thread are started. The data read thread caches the data to be synchronized read from the data source into a cache space, allowing the data write thread to retrieve the cached data to be synchronized from the cache space and write the retrieved data to be synchronized into the local database. In this way, the data synchronization process is split into two asynchronous threads: data reading and data writing. This reduces dependence on the data source, decouples the data source from the local database, and improves the stability of the data synchronization process. Furthermore, the introduction of a cache space allows the data read thread and the data write thread to run concurrently, thereby maximizing data synchronization efficiency.

[0133] The following description continues to illustrate the exemplary structure of the data synchronization device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software module stored in the data synchronization device 455 of the memory 440 may include:

[0134] The startup module 4551 is used to start at least one data read thread and at least one data write thread in response to a data synchronization command, and to establish a communication connection between the at least one data read thread and the data source.

[0135] The reading module 4552 is used to read at least one piece of data to be synchronized from the data source through the at least one data reading thread based on the communication connection, and to store the read data to be synchronized into a cache space; wherein, the cache space is used to cache the data to be synchronized read by the data reading thread;

[0136] The writing module 4553 is used to write the data to be synchronized cached in the cache space to the local database through the at least one data writing thread when the cache space stores the data to be synchronized.

[0137] In some embodiments, the apparatus further includes a first detection module, which is configured to detect the cache space based on the data write thread to obtain a first detection result; the write module 4553 is further configured to write the cached data to be synchronized in the cache space into a local database through the at least one data write thread when the first detection result indicates that the cache space stores the data to be synchronized.

[0138] In some embodiments, the apparatus further includes a first comparison module, which is configured to obtain the total amount of the data to be synchronized in the data source and the amount of the data to be synchronized read from the data source; compare the total amount and the amount of read to obtain a first comparison result; when the first comparison result indicates that the data to be synchronized does not exist in the data source, close the communication connection between the at least one data read thread and the data source, and close the at least one data read thread.

[0139] In some embodiments, the data to be synchronized cached in the cache space is unprocessed data read from the data source. The writing module 4553 is further configured to filter the data to be synchronized based on the data object of the data to be synchronized to obtain target synchronization data; obtain a pre-set attribute mapping relationship, and perform attribute mapping on the target synchronization data based on the attribute mapping relationship to obtain general data corresponding to the data attributes of the local database; and write the general data into the local database through the at least one data writing thread.

[0140] In some embodiments, the apparatus further includes a verification module, which is used to perform format verification on the target synchronization data to obtain a verification result; when the verification result indicates that the target synchronization data passes the verification, the process proceeds to the step of obtaining the pre-set attribute mapping relationship; when the verification result indicates that the target synchronization data fails the verification, the target synchronization data is recorded in an error log.

[0141] In some embodiments, the number of data writing threads is at least two, the cache space is a message queue, and the writing module 4553 is further configured to use the at least two data writing threads to sequentially extract the cached data to be synchronized from the message queue in a parallel processing manner; and write the extracted data to be synchronized into the local database.

[0142] In some embodiments, the apparatus further includes a second detection module, which is configured to detect the write result of the data to be synchronized and obtain a second detection result; when the second detection result indicates that the data to be synchronized has been written to the local database, the second detection module obtains the total cached amount of the data to be synchronized in the cache space and the write amount of the data to be synchronized written from the cache space to the local database; compares the total cached amount with the write amount to obtain a second comparison result; when the second comparison result indicates that the data to be synchronized does not exist in the cache space, the at least one data write thread is closed.

[0143] In some embodiments, the apparatus further includes a third detection module, which is configured to detect the number of times the data to be synchronized has been written to the local database when the second detection result indicates that the data to be synchronized has not been written to the local database, and obtain a third detection result; and when the third detection result indicates that the number of times the data to be synchronized has been written to the local database through the at least one data write thread, write the data to be synchronized to the local database.

[0144] In some embodiments, the third detection module is further configured to stop writing to the data to be synchronized and record the data to be synchronized to an error log when the third detection result indicates that the number of writes to the data to be synchronized has reached the write count threshold.

[0145] In some embodiments, the apparatus further includes a recording module, which is configured to shut down the at least one data write thread; acquire the start time of the at least one data read thread and the at least one data write thread, and the shutdown time of the at least one data read thread and the at least one data write thread; and record the start time and the shutdown time.

[0146] In some embodiments, the apparatus further includes an incremental data synchronization module, which is configured to, in response to an incremental data synchronization command, acquire the recorded start time and shutdown time; determine the incremental data to be synchronized based on the start time and shutdown time, and perform data synchronization on the incremental data to be synchronized.

[0147] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data synchronization method described in this application.

[0148] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the data synchronization method provided in this application. For example, ... Figure 3 The data synchronization method is shown.

[0149] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0150] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0151] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0152] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0153] In summary, the embodiments of this application have the following beneficial effects:

[0154] (1) The data synchronization process is split into two asynchronous threads: data reading and data writing. This reduces the dependence on the data source, decouples the data source from the local database, and improves the stability of the data synchronization process. At the same time, a cache space is introduced so that the data reading thread and the data writing thread can run simultaneously, thereby maximizing the efficiency of data synchronization.

[0155] (2) The synchronization of data to be synchronized is split into two asynchronous processes, which shortens the life cycle of each process, allows computing resources to be released in a timely manner, and reduces errors caused by unresponsive processes in the local system. At the same time, after completing the read and write operations, the communication connection established with the data source and the local database can be closed in a timely manner, quickly releasing connection pool resources and reducing the impact of user data synchronization on business.

[0156] (3) Centralize computer resources to complete the reading of user data and store it in the message queue. In subsequent data writing threads, if an error occurs, there is no need to frequently interact and retry with the upstream data source, further reducing the dependence on the upstream data source and the enterprise network.

[0157] (4) It greatly shortens the interaction time with the data source in the data synchronization process, reduces retries and resource consumption caused by network and data source instability, and improves the efficiency of user data synchronization.

[0158] (5) At least one data read thread uses Keep-alive as the communication connection method with the data source, which is multi-threaded reuse. This prevents frequent opening and closing of the communication connection with the data source, which could cause connection pool overflow or unnecessary resource consumption.

[0159] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data synchronization method, characterized in that, The method includes: In response to a data synchronization command, at least one data read thread and at least one data write thread are started, and a communication connection is established between the at least one data read thread and the data source, wherein the communication connection remains open after being established so that it can be reused by multiple threads. Based on the communication connection, at least one piece of data to be synchronized is read from the data source through the at least one data read thread. The data to be synchronized is unprocessed data from the data source, and the read data to be synchronized is stored in the cache space. The cache space is used to cache the data to be synchronized read by the data read thread; When the cache space stores the data to be synchronized, the data to be synchronized is filtered based on the data object of the data to be synchronized to obtain the target synchronization data; a pre-set attribute mapping relationship is obtained, and the target synchronization data is mapped based on the attribute mapping relationship to obtain general data corresponding to the data attributes of the local database; the general data is written to the local database through the at least one data write thread.

2. The method as described in claim 1, characterized in that, After responding to a data synchronization command and starting at least one data read thread and at least one data write thread, the method further includes: Based on the data write thread, the cache space is detected to obtain a first detection result; When the cache space stores the data to be synchronized, the at least one data write thread writes the cached data to be synchronized in the cache space to the local database, including: When the first detection result indicates that the data to be synchronized is stored in the cache space, the data to be synchronized cached in the cache space is written to the local database through the at least one data write thread.

3. The method as described in claim 1, characterized in that, After storing the read data to be synchronized into the cache space, the method further includes: Obtain the total amount of data to be synchronized in the data source, and the amount of data to be synchronized read from the data source; The total amount and the number of reads are compared to obtain the first comparison result; When the first comparison result indicates that the data to be synchronized does not exist in the data source, the communication connection between the at least one data read thread and the data source is closed, and the at least one data read thread is shut down.

4. The method as described in claim 1, characterized in that, After filtering the data to be synchronized to obtain the target synchronization data, the method further includes: The target synchronization data is format-validated to obtain the validation result; When the verification result indicates that the target synchronization data passes the verification, the process proceeds to the step of obtaining the pre-set attribute mapping relationship; When the verification result indicates that the target synchronization data has failed the verification, the target synchronization data is recorded in the error log.

5. The method as described in claim 1, characterized in that, The number of data write threads is at least two, the cache space is a message queue, and the step of writing the data to be synchronized cached in the cache space to the local database through the at least one data write thread includes: The at least two data writing threads use a parallel processing method to sequentially retrieve the cached data to be synchronized from the message queue; The extracted data to be synchronized is written into the local database.

6. The method as described in claim 1, characterized in that, After the method involves writing the data to be synchronized cached in the cache space to the local database using at least one data write thread, the method further includes: The write result of the data to be synchronized is detected to obtain a second detection result; When the second detection result indicates that the data to be synchronized has been written to the local database, the total amount of the data to be synchronized cached in the cache space and the amount of data to be synchronized written from the cache space to the local database are obtained. The total cache size is compared with the total write size to obtain a second comparison result; When the second comparison result indicates that the data to be synchronized does not exist in the cache space, the at least one data write thread is shut down.

7. The method as described in claim 6, characterized in that, The method further includes: When the second detection result indicates that the data to be synchronized has not been written to the local database, the number of times the data to be synchronized has been written is detected to obtain a third detection result; When the third detection result indicates that the number of writes of the data to be synchronized has not reached the write count threshold, the data to be synchronized is written to the local database through the at least one data write thread.

8. The method as described in claim 7, characterized in that, The method further includes: When the third detection result indicates that the number of writes to the data to be synchronized has reached the write count threshold, the writing of the data to be synchronized is stopped, and the data to be synchronized is recorded in the error log.

9. The method as described in claim 1, characterized in that, After the method involves writing the data to be synchronized cached in the cache space to the local database using at least one data write thread, the method further includes: Close at least one data write thread; Obtain the start time of the at least one data read thread and the at least one data write thread, as well as the shutdown time of the at least one data read thread and the at least one data write thread; The startup time and the shutdown time are recorded.

10. The method as described in claim 9, characterized in that, After recording the startup time and the shutdown time, the method further includes: In response to the incremental data synchronization command, the recorded start time and stop time are obtained; Based on the startup time and shutdown time, the incremental data to be synchronized is determined, and the incremental data to be synchronized is synchronized.

11. A data synchronization device, characterized in that, The device includes: A startup module is used to respond to a data synchronization command, start at least one data read thread and at least one data write thread, and establish a communication connection between the at least one data read thread and the data source, wherein the communication connection remains open after being established so that it can be reused by multiple threads. A reading module is configured to, based on the communication connection, read at least one piece of data to be synchronized from the data source through at least one data reading thread, wherein the data to be synchronized is unprocessed data from the data source, and store the read data to be synchronized into a cache space; wherein the cache space is used to cache the data to be synchronized read by the data reading thread; The write module is used to, when the cache space stores the data to be synchronized, filter the data to be synchronized based on the data object of the data to be synchronized to obtain target synchronization data; obtain a pre-set attribute mapping relationship, and perform attribute mapping on the target synchronization data based on the attribute mapping relationship to obtain general data corresponding to the data attributes of the local database; and write the general data into the local database through the at least one data write thread.

12. The apparatus as claimed in claim 11, characterized in that, The device further includes: The first detection module is used to detect the cache space based on the data write thread and obtain a first detection result; the writing module is further used to write the data to be synchronized cached in the cache space into the local database through the at least one data write thread when the first detection result indicates that the cache space stores the data to be synchronized.

13. The apparatus as claimed in claim 11, characterized in that, The device further includes: The first comparison module is used to obtain the total amount of the data to be synchronized in the data source and the amount of the data to be synchronized read from the data source; compare the total amount and the amount of read to obtain a first comparison result; when the first comparison result indicates that there is no data to be synchronized in the data source, close the communication connection between the at least one data read thread and the data source, and close the at least one data read thread.

14. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data synchronization method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the data synchronization method according to any one of claims 1 to 10.

16. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the data synchronization method according to any one of claims 1 to 10 is implemented.

Images