Synchronous data processing method, device, computer equipment and storage medium

By receiving data synchronization instructions in the data synchronization tool, using the data synchronization component to acquire data and load it into a lock-free circular buffer for processing, the problem of low data synchronization efficiency in the prior art is solved, and efficient data synchronization and processing is achieved.

CN113342836BActive Publication Date: 2024-11-05BEIJING SENHE HENGXING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110698956.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-23
Publication Date
2024-11-05
Estimated Expiration
2041-06-23

AI Technical Summary

Technical Problem

Existing data synchronization tools are inefficient when processing massive data, making it difficult to synchronize data directly from the data source, and have a single function, resulting in limited synchronization speed and affecting data processing efficiency.

Method used

By receiving data synchronization instructions, the data synchronization component is used to obtain the data to be synchronized, and load it into the lock-free circular buffer by filling the cache line, and processing it using preset data processing threads to achieve efficient one-stop data processing.

Benefits of technology

It improves the efficiency of data synchronization processing, is compatible with multiple platforms and data reception methods, realizes the coordinated work of each node, reduces resource overhead, and improves concurrent processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113342836B_ABST
    Figure CN113342836B_ABST
Patent Text Reader

Abstract

The embodiments of the present application belong to the field of big data, and relate to a synchronous data processing method, device, computer equipment and storage medium, the method comprising: receiving a data synchronization instruction; according to the data synchronization instruction, obtaining the data to be synchronized through the data synchronization component, the data to be synchronized comes from the data source or the abnormal cluster node; loading the data to be synchronized into the lock-free circular buffer in the form of filling the cache line; processing the data to be synchronized in the lock-free circular buffer through a preset data processing thread to complete the synchronous data processing. In addition, the present application also relates to blockchain technology, and the processed data to be synchronized can also be stored in the blockchain. The present application improves the efficiency of synchronous data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a synchronous data processing method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology, big data related applications are becoming more and more widespread. Data synchronization is a very common part of big data applications. In order to better synchronize massive data, some data synchronization tools have emerged.

[0003] However, in actual applications, due to the complexity of data source channels, data synchronization tools sometimes have difficulty synchronizing data directly from data sources, which affects the synchronization speed. Moreover, existing data synchronization tools have some shortcomings in use. For example, data synchronization tools can only achieve data synchronization, with a single function, and require extra time for data processing when using synchronized data; in addition, the synchronization speed of existing data synchronization tools is limited, and the data synchronization efficiency is low when the data volume is large. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a synchronous data processing method, apparatus, computer equipment and storage medium to solve the problem of low synchronous data processing efficiency.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a synchronous data processing method, which adopts the following technical solution:

[0006] Receive data synchronization instructions;

[0007] According to the data synchronization instruction, the data to be synchronized is acquired through the data synchronization component, where the data to be synchronized comes from a data source or an abnormal cluster node;

[0008] Loading the data to be synchronized into a lock-free circular buffer in a manner of filling a cache line;

[0009] The data to be synchronized in the lock-free circular buffer is processed by a preset data processing thread to complete synchronization data processing.

[0010] In order to solve the above technical problems, the embodiment of the present application further provides a synchronous data processing device, which adopts the following technical solution:

[0011] An instruction receiving module, used for receiving a data synchronization instruction;

[0012] A data acquisition module, configured to acquire the data to be synchronized through the data synchronization component according to the data synchronization instruction, wherein the data to be synchronized comes from a data source or an abnormal cluster node;

[0013] A data loading module, used for loading the to-be-synchronized data into a lock-free circular buffer in a manner of filling a cache line;

[0014] The data processing module is used to process the data to be synchronized in the lock-free circular buffer through a preset data processing thread to complete synchronous data processing.

[0015] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0016] Receive data synchronization instructions;

[0017] According to the data synchronization instruction, the data to be synchronized is acquired through the data synchronization component, where the data to be synchronized comes from a data source or an abnormal cluster node;

[0018] Loading the data to be synchronized into a lock-free circular buffer in a manner of filling a cache line;

[0019] The data to be synchronized in the lock-free circular buffer is processed by a preset data processing thread to complete synchronization data processing.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0021] Receive data synchronization instructions;

[0022] According to the data synchronization instruction, the data to be synchronized is acquired through the data synchronization component, where the data to be synchronized comes from a data source or an abnormal cluster node;

[0023] Loading the data to be synchronized into a lock-free circular buffer in a manner of filling a cache line;

[0024] The data to be synchronized in the lock-free circular buffer is processed by a preset data processing thread to complete synchronization data processing.

[0025] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: after receiving the data synchronization instruction, the data to be synchronized can be obtained from the data source through various data synchronization components, which is compatible with various platform channels and data receiving methods, ensuring that data synchronization is directly realized from various data sources. In addition, obtaining the data to be synchronized from the abnormal cluster node realizes the collaborative work of each node; the acquired data to be synchronized is loaded into the lock-free circular buffer in the form of filling cache lines; and through a plurality of preset data processing threads, efficient one-stop processing is realized for the data to be synchronized in the lock-free circular buffer according to the preset processing method. The use of filling cache lines and lock-free circular buffers can reduce resource overhead, improve concurrent processing efficiency, and thus improve the synchronization data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0028] Figure 2 is a flow chart of an embodiment of a synchronous data processing method according to the present application;

[0029] Figure 3 is a structural schematic diagram of an embodiment of a synchronous data processing device according to the present application;

[0030] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0034] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, a data providing server 105, and servers 106, 107. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103, the data providing server 105, and the servers 106, 107. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with servers 106, 107 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0037] The data providing server 105 may be a server that provides various services, can generate or store data to be synchronized, and may include a business system or database, etc. The servers 106 and 107 are nodes in a server cluster, and may be servers that provide various services, such as a backend server that provides support for pages displayed on the terminal devices 101, 102, and 103. The data providing server 105 may interact with the servers 106 and 107 through a network.

[0038] It should be noted that the synchronous data processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the synchronous data processing device is generally set in the server.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0040] Continue to refer Figure 2 , shows a flow chart of an embodiment of a synchronous data processing method according to the present application. The synchronous data processing method comprises the following steps:

[0041] Step S201, receiving a data synchronization instruction.

[0042] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0043] The data synchronization instruction may be an instruction for instructing a data synchronization tool to perform data synchronization.

[0044] Specifically, the data synchronization tool is set in the server. The data synchronization tool can perform data synchronization according to the received data synchronization instruction. The data synchronization instruction can be triggered by the terminal and sent by the terminal to the data synchronization tool in the server; it can also be triggered by the server.

[0045] Furthermore, before the above-mentioned step S201, the synchronous data processing method may also include: triggering a data synchronization instruction according to a data synchronization request sent by a data source; or, obtaining working status information of each node in the cluster; determining abnormal cluster nodes according to the working status information, the abnormal cluster nodes including overloaded nodes and faulty nodes; generating a data synchronization instruction according to the abnormal cluster nodes.

[0046] The data source may generate data to be synchronized, for example, the data source may be an upstream business system, and the business system is used to process business transactions. The working status information is information representing the working status of nodes in the server cluster.

[0047] Specifically, the data source can generate or publish data. When the generated or published data needs to be synchronized, the data source can send a data synchronization request to the server. The server triggers a data synchronization instruction according to the data synchronization request, and the data synchronization tool starts data synchronization according to the data synchronization instruction.

[0048] The server can also periodically obtain working status information from each node in the server cluster. The working status information represents the working status of each node. For example, it can represent whether the node is currently in an overloaded state or whether the node is abnormal. For example, when the data to be synchronized that a node in the cluster needs to process exceeds the processing capacity of the node, the node is in an overloaded state; when the data synchronization tool on a node in the cluster has a functional defect, the node is in an abnormal state. Among them, nodes in an overloaded state or an abnormal state will be determined as abnormal cluster nodes.

[0049] For nodes that are in an overloaded or abnormal state, the data allocated to these nodes needs to be transferred out to achieve normal synchronous data processing. The server can generate data synchronization instructions based on the node identifiers of these abnormal cluster nodes.

[0050] When the data synchronization tool detects a triggered data synchronization instruction, it starts to perform synchronization data processing according to the data synchronization instruction.

[0051] In this embodiment, the data synchronization instruction can be triggered according to the data synchronization request of the data source, and can also be triggered according to the working status information of each node in the cluster, which meets various application scenarios.

[0052] Step S202: According to the data synchronization instruction, the data to be synchronized is obtained through the data synchronization component, and the data to be synchronized comes from the data source or the abnormal cluster node.

[0053] The data synchronization component may be a functional component in a data synchronization tool that obtains data to be synchronized from a data source or an abnormal cluster node.

[0054] Specifically, the source environment of the data to be synchronized is diverse, for example, it can be a data source, such as a database, a business system, etc., or it can be an abnormal cluster node in a server cluster. After receiving the data synchronization instruction, different types of data synchronization components can be used to obtain the data to be synchronized according to the different source scenarios of the data to be synchronized. The data synchronization component can be adaptively designed according to the source scenario of the data.

[0055] Step S203: Load the data to be synchronized into the lock-free circular buffer in a manner of filling cache lines.

[0056] Among them, the lock-free circular buffer (RingQueue) is a bounded array based on a ring data structure, which is used to buffer data between multiple threads. The data synchronization tool can load the data to be synchronized into the lock-free circular buffer. When continuously writing data to the lock-free circular buffer, the written data will wrap around to the head of the array when it reaches the end of the array, starting a new round of wrapping. The lock-free circular buffer maintains three variables: the position of the next available data object in the buffer, the position of the next unread data object, and the position of the end of the array. In the lock-free circular buffer, there is no need to delete the data when reading it, and the data will be retained until it is overwritten. The processor does not need to load the next element in the lock-free circular buffer from the main memory every time. When an element is loaded into a cache line, several adjacent elements of the element are loaded into the same cache line together.

[0057] Filling cache lines is a data cache method. Data is stored in the form of cache lines in the CPU (central processing unit) cache. Cache lines are usually 64 bytes. Long type variables in Java are 8 bytes, and one cache line can store 8 long type variables. When a value in a long array is loaded into the cache line, the other seven values ​​will be loaded into the cache line together. This loading feature has a disadvantage: if the first position (head) in the array is being written by the producer, and the eighth position (tail) is being written by the consumer, the producer and the consumer run in threads of different kernels and use different caches. When the producer updates the head, the value of the head in the memory will be updated. The update of the head value in the memory will invalidate all cache lines that cache the head value. Consumers who consume tail need to read the entire cache line from the memory again, and the consumption process is delayed. This sharing of cache lines will affect the efficiency of data processing in concurrent scenarios. For this reason, the data synchronization tool in this application uses the method of filling cache lines to exchange space for time, and 7 unused long type variables are used to fill the data. When filling a cache line, the data to be synchronized is written at a fixed position in the cache line, and the remaining positions can be filled with long type variables with fixed values.

[0058] Furthermore, the above step S203 may include: querying the available data objects in the lock-free circular buffer; and loading the data to be synchronized into the available data objects in a manner of filling cache lines.

[0059] Specifically, the size of the lock-free circular buffer is fixed when it is created, and memory is pre-allocated. The lock-free circular buffer is pre-divided into several areas, each of which is a data object. The corresponding data object for each position in the lock-free circular buffer is determined in advance (initially an empty object). The data objects that can currently be modified or overwritten by the data synchronization component are available data objects.

[0060] When the producer (i.e., data synchronization component) loads data, it obtains the available data object in the lock-free circular buffer, and then loads the data to be synchronized into the data object by filling the cache line, thereby avoiding object recycling caused by frequent creation and destruction of objects, reducing the performance overhead caused by GC (garbage collection) operations, and reducing the garbage collection time.

[0061] The data synchronization tool in this application does not use the blocking queue commonly used in concurrent frameworks to fully utilize the CPU cache. When using a queue, the three variables of the queue head, tail, and size are prone to write contention. For example, taking data out of the queue and removing data are both write operations. When more than one thread performs a write operation, the queue needs to be locked, causing switching between user mode and kernel mode, consuming a lot of system resources, and losing data in the cache.

[0062] In this embodiment, the data to be synchronized is loaded into the available data object of the lock-free circular buffer in a manner of filling cache lines, thereby avoiding the use of a blocking queue and greatly improving data processing efficiency.

[0063] Step S204, processing the data to be synchronized in the lock-free circular buffer by a preset data processing thread to complete the synchronization data processing.

[0064] The data processing thread may be a thread in a data synchronization tool that processes data to be synchronized.

[0065] Specifically, multiple data processing threads Workers are set up in the data synchronization tool. Different data processing threads can process the data to be synchronized differently. The data processing thread reads data from the lock-free circular buffer and processes the data. Each data processing thread maintains the object sequence number of the next data object it needs to access. The data processing thread is equivalent to a consumer in the data processing tool, consuming the data to be synchronized in a preset order. The preset data processing thread processes the data to be synchronized, realizing a one-stop processing flow from synchronization to processing of the data to be synchronized.

[0066] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned processed data to be synchronized, the above-mentioned processed data to be synchronized can also be stored in a node of a blockchain.

[0067] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0068] In this embodiment, after receiving the data synchronization instruction, the data to be synchronized can be obtained from the data source through various data synchronization components, which is compatible with various platform channels and data receiving methods, ensuring that data synchronization is directly realized from various data sources. In addition, the data to be synchronized is obtained from the abnormal cluster node to realize the collaborative work of each node; the obtained data to be synchronized is loaded into the lock-free circular buffer in the form of filling cache lines; and through a plurality of preset data processing threads, efficient one-stop processing is realized for the data to be synchronized in the lock-free circular buffer in accordance with the preset processing method. The use of filling cache lines and lock-free circular buffers can reduce resource overhead, improve concurrent processing efficiency, and thus improve the synchronization data processing efficiency.

[0069] Furthermore, the above step S202 may include:

[0070] Step S2021, obtaining a data source identifier from a data synchronization instruction.

[0071] Specifically, the data source identifier can be the identifier of the source of the data to be synchronized. For example, when the data to be synchronized comes from a database, the data source identifier can be the name or address of the database; when the data to be synchronized comes from an upstream business system, the data source identifier can be the name or address of the business system; when the data to be synchronized comes from other nodes in the server cluster, the data source identifier can be a node identifier or address. The data synchronization instruction can include the data source identifier, and the server can extract the data source identifier from the data synchronization instruction.

[0072] Step S2022, determining the data synchronization component according to the data source identifier, the data synchronization component includes a database synchronization component, a middleware synchronization component, an interface synchronization component and a composite synchronization component.

[0073] Specifically, the data source identifier indicates the source of the data to be synchronized, and the data synchronization component to be used can be determined based on the data source identifier.

[0074] The data synchronization component (Receiver) in the data synchronization tool connects to the data source or the node in the server cluster, and is responsible for extracting or receiving the data to be synchronized. There are many types of data synchronization components, including database synchronization component (DBReceiver), middleware synchronization component (MsgReceiver), interface synchronization component (ApiReceiver) and composite synchronization component (Poly Receiver).

[0075] Among them, the database synchronization component is used to connect to the database and can realize the synchronization of large quantities of historical data; the middleware synchronization component is used to connect to the message middleware, which can play a buffering role to achieve peak shaving and valley filling and load balancing; the interface synchronization component provides an API interface (Application Programming Interface) to the outside world, with a simple access method, suitable for the synchronization of a single or small amount of data; the composite synchronization component is a mixture of database synchronization components, middleware synchronization components and interface synchronization components, with high compatibility, which can meet the data synchronization of any data source. For data synchronization tools, data synchronization components can be regarded as producers of data, so composite synchronization components need to be queued and locked when in use. Data synchronization components outside the composite synchronization component usually work in single producer mode to avoid write contention caused by multiple producers.

[0076] Step S2023: Acquire the data to be synchronized through the determined data synchronization component.

[0077] Specifically, the data synchronization tool may read the data to be synchronized from the data source or the abnormal cluster node according to the determined data synchronization component, and load the data to be synchronized to the node where the data synchronization tool is located.

[0078] In this embodiment, the data synchronization component is determined according to the data source identifier. The data synchronization component includes a database synchronization component, a middleware synchronization component, an interface synchronization component and a composite synchronization component, and is compatible with multiple platform channels and data receiving methods.

[0079] Furthermore, when the data to be synchronized comes from an abnormal cluster node, the above step S2023 may include: determining the unread data objects in the lock-free circular buffer of the abnormal cluster node; reading the unread data objects forward from the tail of the lock-free circular buffer; and obtaining the data to be synchronized in the read unread data objects through the data synchronization component.

[0080] The unread data object may be a data object in a lock-free circular buffer that has not yet been processed by a data processing thread.

[0081] Specifically, when the nodes in the server cluster are under high load, the number of nodes in the cluster can be increased to relieve the working pressure of the nodes through machine expansion. After adding a node, the new node obtains the working status information of other nodes in the cluster to determine the overloaded node, so that the new node can obtain the data to be synchronized from the overloaded node to achieve load balancing; or, determine the faulty node based on the working status information, obtain the data to be synchronized from the faulty node, and replace the faulty node to achieve data synchronization processing.

[0082] When obtaining the data to be synchronized from the lock-free circular buffer of the data synchronization tool of the abnormal cluster node, you can first determine the unread data objects in the lock-free circular buffer, read the unread data objects from the tail of the lock-free circular buffer forward, and synchronize the data to be synchronized in the unread data objects to its own lock-free circular buffer through the data synchronization component. Reading the unread data objects from the tail forward can avoid data contention between different data synchronization tools.

[0083] In one embodiment, multiple data synchronization tools can be deployed on a node, and data can be transferred between data synchronization tools on the same node. When transferring data, the unread data objects in the lock-free circular buffer are first determined, and then the data to be synchronized in the unread data objects are transferred in order from the end to the front.

[0084] In this embodiment, when transferring data to be synchronized between different nodes, the unread data objects in the lock-free circular buffer are first determined, and then the unread data objects are read forward from the tail of the lock-free circular buffer, thereby avoiding data contention between different data synchronization tools and achieving smooth transfer of data to be synchronized in the unread data objects.

[0085] Furthermore, the above step S204 may include: verifying the digital signature of the data to be synchronized in the lock-free circular buffer through a data verification thread; when the digital signature verification is passed, determining the duplicate data in the data to be synchronized through a deduplication thread, and performing deduplication processing on the duplicate data; obtaining the data identifier of the data to be synchronized after deduplication processing; performing standardization processing on the data to be synchronized through a data standardization thread corresponding to the data identifier to obtain standard data; sending the standard data to a preset data application thread to complete the synchronization data processing; or, storing the standard data to complete the synchronization data processing.

[0086] Specifically, in the data synchronization tool, the default data processing threads include data verification thread (Interpreter), log output thread (Logger), deduplication thread (Presecutor), persistent storage thread (Flusher), data matching thread (Allocator) and data standardization thread (Creator).

[0087] The data to be synchronized in the lock-free circular buffer has a digital signature, and the data verification thread can verify the digital signature of the data to be synchronized to determine whether the source of the data to be synchronized is reliable and whether the data is complete.

[0088] In one embodiment, after the digital signature is verified, the data verification thread instantiates the data to be synchronized into an event object. The data synchronization tool can run on a JVM (Java Virtual Machine), and the event object is a Java object.

[0089] When the digital signature verification is passed, the idempotence check is performed on the data to be synchronized through the deduplication thread to determine the duplicate data in the data to be synchronized, and the duplicate data is removed through the deduplication thread.

[0090] The data to be synchronized includes a data identifier, which is used to identify the data, for example, to identify the source of the data, the project that generates the data, etc. The data matching thread can obtain the data identifier of the data to be synchronized after deduplication processing.

[0091] The data to be synchronized with different data identifiers have different data processing logics. The data standardization thread for standardizing the data to be synchronized can be determined based on the data identifier. The data matching thread can assign the data to be synchronized to the corresponding data standardization thread based on the data identifier. The data standardization thread performs standardization processing on the data to be synchronized according to the preset data processing logic, such as performing ETL processing on the data to be synchronized, including data cleaning, extraction, aggregation, calculation, and mapping operations, to generate standard data.

[0092] In one embodiment, the data synchronization tool provides a front-end configuration page, which can determine the data standardization thread corresponding to the data to be synchronized based on factors such as data type, data identification and data usage, so as to realize online configuration modification.

[0093] After the standard data is generated, the standard data can be sent to the preset data application thread, and the data application thread actually applies the standard data. For example, when the data to be synchronized is used for machine learning, the data processing thread can also include a data packaging thread (Wrapper) and a data sending thread (Driver). The data packaging thread is used to package the standard data in batches, and then split the packaged standard data into training data and test data. The data sending thread then uploads the training data and test data to the data application thread (machine learning model) for machine learning. In addition, the standard data can also be stored.

[0094] The log output thread in the data processing thread can generate a data processing log, which includes information such as data identification, data processing start time, data processing end time, synchronization time, digital signature, data type, data source identification, data usage, etc.

[0095] The data synchronization tool can manage the dependencies of data processing threads when processing data to be synchronized through the thread manager Reference, and inform the data processing thread of the maximum sequence number of the data object that can currently be accessed. When the next accessible sequence number of the data processing thread is greater than the sequence number given by the thread manager, the data processing thread enters a waiting state until it receives notification from the thread manager of the newly accessible data object.

[0096] In this embodiment, the data to be synchronized is automatically processed in one stop through a preset data processing thread, including operations such as verification, deduplication, standardization, sending and storage, thereby improving the processing efficiency of the data to be synchronized.

[0097] Furthermore, after the above-mentioned step of determining the duplicate data in the data to be synchronized through the deduplication thread and performing deduplication processing on the duplicate data when the digital signature verification is passed, it may also include: obtaining the synchronization time of the data to be synchronized after the deduplication processing through the persistent storage thread; determining the hard disk write order of the data to be synchronized according to the synchronization time; and persisting the data to be synchronized to the hard disk according to the hard disk write order to complete data backup.

[0098] Specifically, the data synchronization tool in this application runs based on memory, and the use of a memory-based model requires solving problems such as service restart, service crash, and server downtime. The above problems can be solved through the concept of "events". All events processed by the data synchronization tool are persisted and saved, so that the original processing status can be obtained through event replay.

[0099] The persistent storage thread obtains the synchronization time of the data to be synchronized. The synchronization time can be the time when the data to be synchronized is synchronized to the data synchronization tool. The event objects of the data to be synchronized can be sorted in chronological order to obtain the hard disk write order, and then the event objects of the data to be synchronized are persisted to the hard disk according to the hard disk write order. When persisting, events are written sequentially and can be accessed continuously when reading. The efficiency of continuous access to the hard disk is almost the same as the memory reading speed, thereby ensuring the speed of event replay.

[0100] The data synchronization tool can be run in a server cluster. The data synchronization tool writes event snapshots to the hard disk at a certain frequency to achieve quasi-real-time backup. The backup in the hard disk will be synchronized in the cluster to prevent single point failure.

[0101] Considering that service restart scenarios account for the vast majority of cases, in order to cope with service restart, a script is used to monitor the normal shutdown of the JVM. The shutdown hook is registered to the JVM through the Runtime package addShutdownHook. When the JVM is shut down normally, the Hook thread records a snapshot of the Datapush memory site and writes the unfinished event snapshot in the memory to the hard disk in the form of an event stream. When the service is restarted, the ApplicationRunner interface provided by SpringBoot is extended to read the event stream from the hard disk and replay the event in the memory.

[0102] In this embodiment, the data to be synchronized is sequentially written to the hard disk according to the synchronization time, thereby realizing data backup and avoiding the situation where data loss is caused by server errors. At the same time, sequential writing ensures the reading speed of the hard disk.

[0103] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0105] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0106] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a synchronous data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 3 As shown, the synchronous data processing device 300 described in this embodiment includes: an instruction receiving module 301, a data acquisition module 302, a data loading module 303 and a data processing module 304, wherein:

[0108] The instruction receiving module 301 is used to receive a data synchronization instruction.

[0109] The data acquisition module 302 is used to acquire the data to be synchronized through the data synchronization component according to the data synchronization instruction, wherein the data to be synchronized comes from the data source or the abnormal cluster node.

[0110] The data loading module 303 is used to load the data to be synchronized into the lock-free circular buffer in a manner of filling cache lines.

[0111] The data processing module 304 is used to process the data to be synchronized in the lock-free circular buffer through a preset data processing thread to complete the synchronous data processing.

[0112] In this embodiment, after receiving the data synchronization instruction, the data to be synchronized can be obtained from the data source through various data synchronization components, which is compatible with various platform channels and data receiving methods, ensuring that data synchronization is directly realized from various data sources. In addition, the data to be synchronized is obtained from the abnormal cluster node to realize the collaborative work of each node; the obtained data to be synchronized is loaded into the lock-free circular buffer in the form of filling cache lines; and through a plurality of preset data processing threads, efficient one-stop processing is realized for the data to be synchronized in the lock-free circular buffer in accordance with the preset processing method. The use of filling cache lines and lock-free circular buffers can reduce resource overhead, improve concurrent processing efficiency, and thus improve the synchronization data processing efficiency.

[0113] In some optional implementations of this embodiment, the synchronous data processing device 300 may further include: a request triggering module, an information acquisition module, an abnormality determination module, and an instruction generation module, wherein:

[0114] The request trigger module is used to trigger the data synchronization instruction according to the data synchronization request sent by the data source.

[0115] The information acquisition module is used to obtain the working status information of each node in the cluster.

[0116] The abnormality determination module is used to determine abnormal cluster nodes according to the working status information. The abnormal cluster nodes include overloaded nodes and faulty nodes.

[0117] The instruction generation module is used to generate data synchronization instructions according to abnormal cluster nodes.

[0118] In this embodiment, the data synchronization instruction can be triggered according to the data synchronization request of the data source, and can also be triggered according to the working status information of each node in the cluster, which meets various application scenarios.

[0119] In some optional implementations of this embodiment, the data acquisition module 302 may include: a source acquisition submodule, a component determination submodule and a data acquisition submodule, wherein:

[0120] The source acquisition submodule is used to obtain the data source identifier from the data synchronization instruction.

[0121] The component determination submodule is used to determine the data synchronization component according to the data source identifier. The data synchronization component includes a database synchronization component, a middleware synchronization component, an interface synchronization component and a composite synchronization component.

[0122] The data acquisition submodule is used to acquire the data to be synchronized through a determined data synchronization component.

[0123] In this embodiment, the data synchronization component is determined according to the data source identifier. The data synchronization component includes a database synchronization component, a middleware synchronization component, an interface synchronization component and a composite synchronization component, and is compatible with multiple platform channels and data receiving methods.

[0124] In some optional implementations of this embodiment, when the data to be synchronized comes from an abnormal cluster node, the data acquisition submodule may include: an object determination unit, an object reading unit, and a data reading unit, wherein:

[0125] The object determination unit is used to determine the unread data objects in the lock-free circular buffer of the abnormal cluster node.

[0126] The object reading unit is used to read unread data objects from the tail of the lock-free circular buffer.

[0127] The data reading unit is used to obtain the data to be synchronized in the unread data object read through the data synchronization component.

[0128] In this embodiment, when transferring data to be synchronized between different nodes, the unread data objects in the lock-free circular buffer are first determined, and then the unread data objects are read forward from the tail of the lock-free circular buffer, thereby avoiding data contention between different data synchronization tools and achieving smooth transfer of data to be synchronized in the unread data objects.

[0129] In some optional implementations of this embodiment, the data loading module 303 may include: an object query submodule and a data loading submodule, wherein:

[0130] The object query submodule is used to query the available data objects in the lock-free circular buffer.

[0131] The data loading submodule is used to load the data to be synchronized into the available data object in a manner of filling the cache line.

[0132] In this embodiment, the data to be synchronized is loaded into the available data object of the lock-free circular buffer in a manner of filling cache lines, thereby avoiding the use of a blocking queue and greatly improving data processing efficiency.

[0133] In some optional implementations of this embodiment, the data processing module 304 may include: a signature verification submodule, a duplication determination submodule, an identification acquisition submodule, a standard processing submodule, a data transmission submodule, and a data storage submodule, wherein:

[0134] The signature verification submodule is used to verify the digital signature of the data to be synchronized in the lock-free circular buffer through the data verification thread.

[0135] The duplicate determination submodule is used to determine the duplicate data in the data to be synchronized through the deduplication thread when the digital signature verification is passed, and to deduplicate the duplicate data.

[0136] The identifier acquisition submodule is used to obtain the data identifier of the data to be synchronized after deduplication processing.

[0137] The standard processing submodule is used to perform standardization processing on the data to be synchronized through the data standardization thread corresponding to the data identifier to obtain standard data.

[0138] The data sending submodule is used to send standard data to the preset data application thread to complete synchronous data processing.

[0139] The data storage submodule is used to store standard data and complete synchronous data processing.

[0140] In this embodiment, the data to be synchronized is automatically processed in one stop through a preset data processing thread, including operations such as verification, deduplication, standardization, sending and storage, thereby improving the processing efficiency of the data to be synchronized.

[0141] In some optional implementations of this embodiment, the data processing module 304 may further include a data persistence submodule, which may include: a time acquisition unit, a sequence determination unit, and a data persistence unit, wherein:

[0142] The time acquisition unit is used to obtain the synchronization time of the data to be synchronized after deduplication processing through the persistent storage thread.

[0143] The sequence determination unit is used to determine the hard disk writing sequence of the data to be synchronized according to the synchronization time.

[0144] The data persistence unit is used to persist the data to be synchronized to the hard disk in the order in which the hard disk is written to complete data backup.

[0145] In this embodiment, the data to be synchronized is sequentially written to the hard disk according to the synchronization time, thereby realizing data backup and avoiding the situation where data loss is caused by server errors. At the same time, sequential writing ensures the reading speed of the hard disk.

[0146] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0148] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0149] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the synchronous data processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0150] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the synchronous data processing method.

[0151] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0152] The computer device provided in this embodiment can execute the above-mentioned synchronous data processing method. Here, the synchronous data processing method can be the synchronous data processing method of each of the above-mentioned embodiments.

[0153] In this embodiment, after receiving the data synchronization instruction, the data to be synchronized can be obtained from the data source through various data synchronization components, which is compatible with various platform channels and data receiving methods, ensuring that data synchronization is directly realized from various data sources. In addition, the data to be synchronized is obtained from the abnormal cluster node to realize the collaborative work of each node; the obtained data to be synchronized is loaded into the lock-free circular buffer in the form of filling cache lines; and through a plurality of preset data processing threads, efficient one-stop processing is realized for the data to be synchronized in the lock-free circular buffer in accordance with the preset processing method. The use of filling cache lines and lock-free circular buffers can reduce resource overhead, improve concurrent processing efficiency, and thus improve the synchronization data processing efficiency.

[0154] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the synchronous data processing method as described above.

[0155] In this embodiment, after receiving the data synchronization instruction, the data to be synchronized can be obtained from the data source through various data synchronization components, which is compatible with various platform channels and data receiving methods, ensuring that data synchronization is directly realized from various data sources. In addition, the data to be synchronized is obtained from the abnormal cluster node to realize the collaborative work of each node; the obtained data to be synchronized is loaded into the lock-free circular buffer in the form of filling cache lines; and through a plurality of preset data processing threads, efficient one-stop processing is realized for the data to be synchronized in the lock-free circular buffer in accordance with the preset processing method. The use of filling cache lines and lock-free circular buffers can reduce resource overhead, improve concurrent processing efficiency, and thus improve the synchronization data processing efficiency.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0157] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A synchronous data processing method, characterized in that: The steps include: Receive data synchronization instructions; According to the data synchronization instruction, the data to be synchronized is acquired through the data synchronization component, where the data to be synchronized comes from a data source or an abnormal cluster node; The data to be synchronized is loaded into a lock-free circular buffer in a manner of filling a cache line, wherein the data is not deleted when being read in the lock-free circular buffer, and the data is retained until being overwritten; Processing the data to be synchronized in the lock-free circular buffer by a preset data processing thread to complete synchronization data processing; The step of processing the to-be-synchronized data in the lock-free circular buffer by a preset data processing thread to complete the synchronization data processing comprises: Verifying the digital signature of the data to be synchronized in the lock-free circular buffer by a data verification thread; When the digital signature is verified, duplicate data in the data to be synchronized is determined through a deduplication thread, and deduplication processing is performed on the duplicate data; Obtaining the data identifier of the data to be synchronized after deduplication processing; The data to be synchronized is standardized through a data standardization thread corresponding to the data identifier to obtain standard data; Send the standard data to a preset data application thread to complete synchronous data processing; or, The standard data is stored to complete synchronous data processing.

2. The synchronous data processing method according to claim 1, characterized in that: Before the step of receiving the data synchronization instruction, the method further includes: According to the data synchronization request sent by the data source, trigger the data synchronization instruction; or, Get the working status information of each node in the cluster; Determine an abnormal cluster node according to the working status information, wherein the abnormal cluster node includes an overloaded node and a faulty node; A data synchronization instruction is generated according to the abnormal cluster node.

3. The synchronous data processing method according to claim 1, characterized in that: The step of acquiring the data to be synchronized through the data synchronization component according to the data synchronization instruction includes: Obtaining a data source identifier from the data synchronization instruction; According to the data source identifier, a data synchronization component is determined, wherein the data synchronization component includes a database synchronization component, a middleware synchronization component, an interface synchronization component and a composite synchronization component; The data to be synchronized is obtained through a determined data synchronization component.

4. The synchronous data processing method according to claim 3, characterized in that: When the data to be synchronized comes from an abnormal cluster node, the step of obtaining the data to be synchronized through the determined data synchronization component includes: Determine unread data objects in the lock-free circular buffer of the abnormal cluster node; Reading unread data objects forward from the tail of the lock-free circular buffer; The data synchronization component is used to obtain the data to be synchronized in the read unread data object.

5. The synchronous data processing method according to claim 1, characterized in that: The step of loading the data to be synchronized into the lock-free circular buffer in a manner of filling cache lines comprises: Query the available data objects in the lock-free circular buffer; The to-be-synchronized data is loaded into the available data object in a manner of filling a cache line.

6. The synchronous data processing method according to claim 1, characterized in that: After the step of determining duplicate data in the data to be synchronized by using a deduplication thread and deduplicating the duplicate data when the digital signature is verified, the method further includes: Obtaining the synchronization time of the data to be synchronized after deduplication processing through a persistent storage thread; Determine the hard disk writing order of the data to be synchronized according to the synchronization time; The data to be synchronized is persisted to the hard disk in the order in which the hard disk is written, so as to complete the data backup.

7. A synchronous data processing device, characterized in that: include: An instruction receiving module, used for receiving a data synchronization instruction; A data acquisition module, configured to acquire the data to be synchronized through the data synchronization component according to the data synchronization instruction, wherein the data to be synchronized comes from a data source or an abnormal cluster node; A data loading module, used for loading the data to be synchronized into a lock-free circular buffer in a manner of filling a cache line, wherein the data is not deleted when being read in the lock-free circular buffer, and the data is retained until being overwritten; A data processing module, used for processing the data to be synchronized in the lock-free circular buffer through a preset data processing thread to complete synchronization data processing; The data processing module is also used to verify the digital signature of the data to be synchronized in the lock-free circular buffer through a data verification thread; when the digital signature verification is passed, the duplicate data in the data to be synchronized is determined through a deduplication thread, and the duplicate data is deduplicated; the data identifier of the data to be synchronized after the deduplication processing is obtained; the data to be synchronized is standardized through a data standardization thread corresponding to the data identifier to obtain standard data; the standard data is sent to a preset data application thread to complete the synchronization data processing; or, the standard data is stored to complete the synchronization data processing.

8. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the synchronous data processing method according to any one of claims 1 to 6 when executing the computer-readable instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the synchronous data processing method according to any one of claims 1 to 6 are implemented.