Data processing method, device, electronic device, system and storage medium

Abstract

The embodiment of the present invention discloses a data processing method, device, electronic device, system and storage medium. The data processing method comprises: obtaining source data from a source server through a primary thread and storing the source data in a primary cache; obtaining source data from a primary cache through a secondary thread, deserializing the source data to obtain target data, and storing the target data in a secondary cache; obtaining target data from a secondary cache through a third thread, and writing the target data to a target server. The embodiment of the present invention can reduce the requirements for technical personnel and reduce operation and maintenance costs.

Description

Technical Field

[0001] The present invention relates to data synchronization technology, and in particular to a data processing method, device, electronic equipment, system and storage medium. Background Art

[0002] Kafka is an open source stream processing platform with the characteristics of high throughput, low latency, and high performance. It is currently widely used by major Internet companies. Major Internet companies use Kafka as the source of real-time data and synchronize data to their own servers in real time. In the process of implementing the present invention, the inventors found that the current Kafka data synchronization solutions are mostly based on real-time computing frameworks such as Flink, Spark streaming, and Flume. These frameworks have high entry barriers and high requirements for technical personnel. In addition, these frameworks rely on external components such as zookeeper and Hadoop to provide services, and the operation and maintenance costs are high. Summary of the invention

[0003] The embodiments of the present invention provide a data processing method, device, electronic device, system and storage medium, which can reduce the requirements for technical personnel and reduce operation and maintenance costs.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0005] Obtain source data from a source server through a primary thread, and store the source data into a primary cache;

[0006] Obtaining the source data from the first-level cache through a second-level thread, deserializing the source data to obtain target data, and storing the target data in the second-level cache;

[0007] The target data is obtained from the secondary cache through a third-level thread, and the target data is written to a target server.

[0008] In a second aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0009] A first processing module, configured to obtain source data from a source server through a primary thread, and store the source data into a primary cache;

[0010] A second processing module is used to obtain the source data from the first-level cache through a second-level thread, deserialize the source data to obtain target data, and store the target data in the second-level cache;

[0011] The third processing module is used to obtain the target data from the secondary cache through a third-level thread, and write the target data into a target server.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method as described in any one of the embodiments of the present invention is implemented.

[0013] In a fourth aspect, an embodiment of the present invention further provides a data processing system, including a source server, a target server, and an electronic device for executing any of the data processing methods described in the embodiments of the present invention.

[0014] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the data processing method as described in any one of the embodiments of the present invention is implemented.

[0015] In an embodiment of the present invention, source data can be obtained from a source server through a first-level thread and stored in a first-level cache. Source data can be obtained from the first-level cache through a second-level thread, and the source data can be deserialized to obtain target data, which can be stored in a second-level cache. Target data can be obtained from the second-level cache through a third-level thread, and the target data can be written to a target server. That is, in an embodiment of the present invention, data in a source server can be synchronized to a target server by calling each thread. Therefore, it is only necessary to configure the operating environment of a thread framework composed of each thread, which reduces requirements for technical personnel, does not rely on external components to provide services, and reduces operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention.

[0017] Figure 2 It is another flowchart of the data processing method provided by an embodiment of the present invention.

[0018] Figure 3 It is a thread framework schematic diagram provided by an embodiment of the present invention.

[0019] Figure 4 It is another flowchart of the data processing method provided by an embodiment of the present invention.

[0020] Figure 5 It is a structural schematic diagram of a data processing device provided by an embodiment of the present invention.

[0021] Figure 6 It is a structural schematic diagram of a data processing system provided by an embodiment of the present invention.

[0022] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] Figure 1 A flow chart of a data processing method provided in an embodiment of the present invention is provided. The method can be executed by a data processing device provided in an embodiment of the present invention, and the device can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated in a server. The following embodiments will be described by taking the device integrated in a server as an example.

[0025] In a specific implementation, the server may be a synchronization server for synchronizing data from a source server to a target server, wherein the source server may be a Kafka server, and the target server may be any one of a distributed file system (Hadoop Distributed File System, HDFS) server, an HBase server, a Clickhouse server, a Hive server, and an Elasticsearch server. A Kubernetes (K8s for short) environment may be configured in the synchronization server. K8s is an open source container orchestration engine from Google, which is mainly responsible for the operation of containers on a host, as well as the scheduling and orchestration of containers. Specifically, in an embodiment of the present invention, a thread framework for implementing a data processing method according to an embodiment of the present invention may be deployed in a K8s environment in the form of a container for operation.

[0026] Please refer to Figure 1 The data processing method provided by the embodiment of the present invention may specifically include the following steps:

[0027] Step 101, obtaining source data from a source server through a primary thread, and storing the source data in a primary cache.

[0028] For example, the source server can create a topic queue based on the data type and send the produced source data to the topic queue of the corresponding type. The synchronization server can subscribe to the topic queue created by the source server, and then obtain the source data from the subscribed topic queue through the first-level thread, and store the obtained source data in the first-level cache.

[0029] In the embodiment of the present invention, when the source data is obtained from the subscribed topic queue through the primary thread, the source client can be created through the primary thread first, and the source client can be used as a consumer of the data in the topic queue to obtain the source data from the subscribed topic queue using the source client. For example, when the source server is a Kafka server, a Kafka client can be created through the primary thread, and the Kafka client can be used to obtain the source data from the corresponding topic queue in the Kafka server.

[0030] Specifically, the acquired source data may be a batch of data represented in the form of key-value pairs. After the source data is acquired, the source data may be encapsulated one by one. In a specific embodiment, there may be one or more first-level caches. When there is one first-level cache, all source data may be directly stored in the first-level cache; when there are multiple first-level caches, the first-level cache where each source data should be stored may be determined according to the hash modulus value of each source data, and each source data may be stored in the corresponding first-level cache. That is, when there are multiple first-level caches, for any source data, a hash modulus operation may be performed on the source data according to the number of first-level caches to obtain a first modulus value, and a first-level target cache may be determined from the first-level cache according to the first modulus value, and the source data may be stored in the first-level target cache; and so on, a first-level target cache may be determined for each source data, and each source data may be stored in the corresponding first-level target cache. Wherein, when performing a hash modulus operation on each source data, a hash modulus operation may be performed on the key in each source data.

[0031] Step 102, obtain source data from the first-level cache through the second-level thread, deserialize the source data to obtain target data, and store the target data in the second-level cache.

[0032] For example, there may be one or more secondary threads, and the number of primary caches may be set according to the number of secondary threads (i.e., the number of upstream caches may be set according to the number of downstream threads), such as setting the number of primary caches to be the same as the number of secondary threads. When there are multiple secondary threads and multiple primary caches, a primary correspondence may be pre-set, and the primary correspondence includes a correspondence between the secondary threads and the primary caches, such as binding the identification information of each secondary thread (such as thread name, number, etc.) with the identification information of the corresponding primary cache (such as cache address, number, etc.) in the primary correspondence.

[0033] Specifically, when there is only one first-level cache and one second-level thread, the second-level thread can be directly used to obtain source data from the first-level cache; when there are multiple first-level caches and multiple second-level threads, the first-level cache corresponding to each second-level thread can be determined according to the first-level correspondence relationship, and each second-level thread can obtain source data from its corresponding first-level cache.

[0034] After obtaining the source data, the secondary thread can be used to deserialize the obtained source data. For example, the secondary thread can be used to convert the source data of the binary stream or text stream structure into a data structure that is easy to process and read. Furthermore, the data obtained after the deserialization process can be cleaned, encrypted, etc. to obtain the target data.

[0035] After obtaining the target data, the target data can be encapsulated, and the encapsulated target data can be stored in the secondary cache through the secondary thread. In a specific embodiment, there can also be one or more secondary caches. When there is one secondary cache, all target data can be directly stored in the secondary cache; when there are multiple secondary caches, the secondary cache where each target data should be stored can be determined according to the hash modulus value of each target data, and each target data can be stored in the corresponding secondary cache. That is, when there are multiple secondary caches, for any target data, a hash modulus operation can be performed on the target data according to the number of secondary caches to obtain a second modulus value, and a secondary target cache can be determined from the secondary cache according to the second modulus value, and the target data can be stored in the secondary target cache; and so on, a secondary target cache is determined for each target data, and each target data is stored in the corresponding secondary target cache. Among them, when performing a hash modulus operation on each target data, a hash modulus operation can be performed on the key in each target data.

[0036] Step 103, obtaining target data from the secondary cache through the third-level thread, and writing the target data to the target server.

[0037] For example, there may be one or more level 3 threads, and the number of level 2 caches may be set according to the number of level 3 threads, such as setting the number of level 2 caches to be the same as the number of level 3 threads. When there are multiple level 3 threads and level 2 caches, a level 2 correspondence may be pre-set, and the level 2 correspondence includes a correspondence between level 3 threads and level 2 caches, such as binding the identification information (such as thread name, number, etc.) of each level 3 thread with the identification information (such as cache address, number, etc.) of the corresponding level 2 cache in the level 2 correspondence.

[0038] Specifically, when there is only one L2 cache and L3 thread, the L3 thread can be used directly to obtain target data from the L2 cache; when there are multiple L2 caches and L3 threads, the L2 cache corresponding to each L3 thread can be determined according to the L2 correspondence relationship, and each L3 thread can obtain target data from its corresponding L2 cache.

[0039] In the embodiment of the present invention, both the first-level cache and the second-level cache may be common caches, such as a ring cache (Ringbuff), a blocking queue (Array Blocking Queue) cache, etc., which are not specifically limited here.

[0040] After obtaining the target data, the target data can be written to the target server using the third-level thread. In a specific implementation, when writing the target data to the target server using the third-level thread, the target client can be created through the third-level thread first. The target client can be used as a consumer of the target data and the target data can be written to the target server using the target client. For example, when the target server is an HBase server, an HBase client can be created through the third-level thread and the target data can be written to the HBase server using the HBase client. For example, when the target server is an HDFS server, an HDFS client can be created through the third-level thread and the target data can be written to the HDFS server using the HDFS client.

[0041] In a specific implementation, when using a target client to write target data to a target server, the storage format supported by the target server can be determined first, and the target data can be written to the target server in batches according to the storage format through the target client. For example, when the target server is an HDFS server, the storage format can be SequenceFile or MapFile format; when the target server is a Hive server, the storage format can be TextFile format; when the target server is a Clickhouse server, the storage format can be an array format, etc.

[0042] The above technical solution can obtain source data from the source server through a first-level thread and store the source data into a first-level cache, obtain source data from the first-level cache through a second-level thread, deserialize the source data to obtain target data, store the target data into a second-level cache, obtain target data from the second-level cache through a third-level thread, and write the target data to the target server. That is, in an embodiment of the present invention, the data in the source server can be synchronized to the target server by calling each thread, so it is only necessary to configure the operating environment of the thread framework composed of each thread (for example, configure the thread framework composed of each thread in the form of a container to run in a K8s environment), which reduces the requirements for technical personnel, does not rely on external components to provide services, and reduces operation and maintenance costs.

[0043] In the embodiment of the present invention, the primary thread is used to connect to the source server, so there can be only one primary thread, and there can be multiple secondary threads and tertiary threads. Multiple secondary threads and tertiary threads can process data in parallel, thereby improving data processing efficiency.

[0044] The following takes the case where there are multiple secondary threads and tertiary threads as an example to illustrate the data processing method provided by the embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0045] Step 201: Create a source client through a primary thread, and use the source client to obtain source data from a source server.

[0046] Specifically, the source client can be a consumer of data in the topic queue of the source server, and use the source client to obtain source data from the subscribed topic queue. For example, when the source server is a Kafka server, a Kafka client can be created through a first-level thread, and the Kafka client can be used to obtain source data from the corresponding topic queue in the Kafka server.

[0047] Step 202: Perform a hash modulus operation on the source data according to the number of first-level caches to obtain a first modulus value, determine a first-level target cache from the first-level cache according to the first modulus value, and store the source data in the first-level target cache.

[0048] Specifically, the acquired source data may be a batch of data represented in the form of key-value pairs. After acquiring the source data, the source data may be encapsulated one by one, and then the first-level cache where each source data should be stored is determined according to the hash modulus value of each source data, and each source data is stored in the corresponding first-level cache. That is, for any source data, a hash modulus operation may be performed on the source data according to the number of first-level caches to obtain a first modulus value, and a first-level target cache is determined from the first-level cache according to the first modulus value, and the source data is stored in the first-level target cache; and so on, a first-level target cache is determined for each source data, and each source data is stored in the corresponding first-level target cache. Wherein, when performing a hash modulus operation on each source data, a hash modulus operation may be performed on the key in each source data.

[0049] Step 203, determining the level 1 cache corresponding to the level 2 thread according to the level 1 correspondence, obtaining source data from the level 1 cache corresponding to the level 2 thread through the level 2 thread, and deserializing the source data to obtain target data.

[0050] The first-level correspondence relationship can be preset, and the first-level correspondence relationship includes the correspondence relationship between the second-level thread and the first-level cache. For example, the identification information (such as thread name, number, etc.) of each second-level thread can be bound to the identification information (such as cache address, number, etc.) of the corresponding first-level cache in the first-level correspondence relationship. The first-level cache corresponding to each second-level thread can be determined according to the first-level correspondence relationship, and source data can be obtained from the corresponding first-level cache through each second-level thread.

[0051] After obtaining the source data, the secondary thread can be used to deserialize the obtained source data. For example, the secondary thread can be used to convert the source data of the binary stream or text stream structure into a data structure that is easy to process and read. Furthermore, the data obtained after the deserialization process can be cleaned, encrypted, etc. to obtain the target data.

[0052] Step 204 , performing a hash modulo operation on the target data according to the number of the secondary caches to obtain a second modulo value, determining a secondary target cache from the secondary cache according to the second modulo value, and storing the target data in the secondary target cache.

[0053] After obtaining the target data, the target data can be encapsulated, and then a hash modulo operation can be performed on each target data after encapsulation to determine the secondary cache where each target data should be stored. For any target data, a hash modulo operation can be performed on the target data according to the number of secondary caches to obtain a second modulo value, and a secondary target cache can be determined from the secondary cache according to the second modulo value, and the target data can be stored in the secondary target cache; and so on, a secondary target cache is determined for each target data, and each target data is stored in the corresponding secondary target cache. Among them, when performing a hash modulo operation on each target data, a hash modulo operation can be performed on the key in each target data.

[0054] Step 205 , determining the L2 cache corresponding to the L3 thread according to the L2 correspondence, and obtaining target data from the L2 cache corresponding to the L3 thread through the L3 thread.

[0055] The second-level correspondence relationship may be preset, and the second-level correspondence relationship includes the correspondence relationship between the third-level thread and the second-level cache. For example, the identification information (such as thread name, number, etc.) of each third-level thread may be bound to the identification information (such as cache address, number, etc.) of the corresponding second-level cache in the second-level correspondence relationship. The second-level cache corresponding to each third-level thread may be determined according to the second-level correspondence relationship, and target data may be obtained from the second-level cache corresponding to the third-level thread through each third-level thread.

[0056] Step 206: Determine the storage format supported by the target server.

[0057] For example, when the target server is an HDFS server, the storage format may be SequenceFile or MapFile format; when the target server is a Hive server, the storage format may be TextFile format; when the target server is a Clickhouse server, the storage format may be an array format, etc.

[0058] Step 207: write the target data into the target server according to the storage format through the third-level thread.

[0059] Specifically, when using the third-level thread to write the target data to the target server, you can first create a target client through the third-level thread. The target client can be used as a consumer of the target data and the target data can be written to the target server using the target client. For example, when the target server is an HBase server, you can create an HBase client through the third-level thread and use the HBase client to write the target data to the HBase server. For example, when the target server is an HDFS server, you can create an HDFS client through the third-level thread and use the HDFS client to write the target data to the HDFS server.

[0060] In a specific embodiment, the first-level thread can be implemented by using a source operator, the second-level thread can be implemented by using a parse operator, and the third-level thread can be implemented by using a sink operator.

[0061] In a specific embodiment, for example, if there are three level-2 threads and three level-3 threads, the thread framework of the embodiment of the present invention can be as follows: Figure 3 As shown, the first-level thread is used to obtain source data from the source server, and determines the first-level cache where each source data should be stored through calculation, and then stores the source data into the determined first-level cache; each second-level thread is used to determine its corresponding first-level cache according to the first-level correspondence, obtain the source data from its corresponding first-level cache, and deserialize the obtained source data to obtain the target data, determine the second-level cache where each target data should be stored through calculation, and then store the target data into the determined second-level cache; each third-level thread is used to determine its corresponding second-level cache according to the second-level correspondence, obtain the target data from its corresponding second-level cache, and write the target data to the target server.

[0062] It should be noted that Figure 3 In the thread framework shown, only the example of there being multiple secondary threads and tertiary threads, and the number of secondary threads is the same as the number of tertiary threads, is used for illustration. In a specific implementation, the number of secondary threads and tertiary threads may also be different, which is not specifically limited here.

[0063] In an embodiment of the present invention, data in the source server can be synchronized to the target server by calling each thread. Therefore, it is only necessary to configure the operating environment of the thread framework composed of each thread (for example, the thread framework composed of each thread is configured in the form of a container to run in the K8s environment), which reduces the requirements for technical personnel and does not rely on external components to provide services, thereby reducing operation and maintenance costs. In addition, multiple secondary threads and tertiary threads can be set, and multiple secondary threads and tertiary threads can process data in parallel, thereby improving data processing efficiency.

[0064] In a specific embodiment, data consumption locations can also be recorded to avoid data duplication and data loss. The specific implementation method can be as follows: Figure 4 As shown, the following steps are included:

[0065] Step 301: Create a data consumption record of the source client through a primary thread, and send the data consumption record to a secondary thread.

[0066] For example, after the primary thread creates the source client, it can periodically create a snapshot of the source client's consumption position for the data in the subscription topic queue, encapsulate the consumption position snapshot into a data consumption record, and broadcast the data consumption record to the downstream secondary thread through the primary thread.

[0067] Step 302: Send the data consumption record to the target client created by the third-level thread through the second-level thread.

[0068] After receiving the data consumption record, each secondary thread may send the data consumption record to the downstream tertiary thread in the form of broadcast.

[0069] Step 303: Use the target client to write the target data in batches to the target server according to the data consumption record.

[0070] In a specific implementation, after obtaining the target data, each third-level thread can wait for the broadcasted data consumption record, and after receiving the data consumption record, the target client can be used to write the target data in batches to the target server according to the data consumption record. For example, the current consumption position can be determined according to the data consumption record, and the target client can be used to write the target data before the current consumption position in batches to the target server.

[0071] Step 304: Send the data consumption record to the source server through the fourth-level thread.

[0072] Specifically, after the target client is used to write the target data before the current consumption position in batches to the target server, the data consumption record can be sent to the source server through the fourth-level thread. In this way, after the thread framework on the synchronization server is restarted, when the source data is obtained from the source server again, the source server can provide data to the synchronization server according to the data consumption record, thereby avoiding data duplication and data loss, and improving the writing efficiency through the batch writing writing method.

[0073] Relying on the data processing method provided by the thread framework of the embodiment of the present invention, it is possible to synchronize the data in the Kafka server (server cluster) to the HDFS server, HBase server, Clickhouse server, Hive server, Elasticsearch server, etc. in real time. Taking the real-time synchronization of data to the HDFS server as an example, under the premise that the number of Kafka topic partitions is 1 and the container resource specification is 4 cores and 12G, the real-time synchronization of 12GB of data can be completed within 1 minute, and the data synchronization performance can be horizontally expanded according to the number of Kafka topic partitions, that is, the real-time synchronization data size / minute = 12GB*number of topic partitions.

[0074] Figure 5 1 is a structural diagram of a data processing device provided by an embodiment of the present invention, and the device is suitable for executing the data processing method provided by an embodiment of the present invention. Figure 5 As shown, the device may specifically include:

[0075] The first processing module 501 is used to obtain source data from the source server through a primary thread and store the source data into a primary cache;

[0076] The second processing module 502 is used to obtain the source data from the first-level cache through a second-level thread, deserialize the source data to obtain target data, and store the target data in the second-level cache;

[0077] The third processing module 503 is used to obtain the target data from the second-level cache through a third-level thread, and write the target data into a target server.

[0078] In one embodiment, the first processing module 501 obtains source data from the source server through a primary thread, including: creating a source client through the primary thread, and obtaining the source data from the source server using the source client;

[0079] The third processing module 503 writes the target data into the target server through the third-level thread, including: creating a target client through the third-level thread, and using the target client to write the target data into the target server.

[0080] In one embodiment, the first processing module 501 stores the source data into the first-level cache, including: performing a hash modulus operation on the source data according to the number of the first-level cache to obtain a first modulus value, determining a first-level target cache from the first-level cache according to the first modulus value, and storing the source data into the first-level target cache;

[0081] The second processing module 502 stores the target data into the secondary cache, including: performing a hash modulus operation on the target data according to the number of the secondary cache to obtain a second modulus value, determining a secondary target cache from the secondary cache according to the second modulus value, and storing the target data into the secondary target cache.

[0082] In one embodiment, the second processing module 502 obtains the source data from the first-level cache through the second-level thread, including: determining the first-level cache corresponding to the second-level thread according to the first-level corresponding relationship, and obtaining the source data from the first-level cache corresponding to the second-level thread through the second-level thread;

[0083] The third processing module 503 obtains the target data from the secondary cache through the third-level thread, including: determining the secondary cache corresponding to the third-level thread according to the secondary corresponding relationship, and obtaining the target data from the secondary cache corresponding to the third-level thread through the third-level thread.

[0084] In one embodiment, the first processing module 501 is further used to create a data consumption record of the source client through the first-level thread, and send the data consumption record to the target client created by the third-level thread through the second-level thread;

[0085] The third processing module 503 uses the target client to write the target data into the target server, including:

[0086] The target data is written in batches to the target server using the target client according to the data consumption record.

[0087] In one embodiment, the device further comprises:

[0088] The fourth processing module is used to send the data consumption record to the source server through a fourth-level thread.

[0089] In one embodiment, the third processing module 503 writes the target data to the target server through a three-level thread, including:

[0090] Determining a storage format supported by the target server;

[0091] The target data is written into the target server according to the storage format through the third-level thread.

[0092] In one embodiment, the source server includes a Kafka server, and the target server includes any one of a distributed file system HDFS server, an HBase server, a Clickhouse server, a Hive server, and an Elasticsearch server.

[0093] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0094] The device of the embodiment of the present invention can obtain source data from the source server through a first-level thread and store the source data into a first-level cache, obtain source data from the first-level cache through a second-level thread, deserialize the source data to obtain target data, store the target data into a second-level cache, obtain target data from the second-level cache through a third-level thread, and write the target data to the target server. That is, in the embodiment of the present invention, data in the source server can be synchronized to the target server by calling each thread, so it is only necessary to configure the operating environment of the thread framework composed of each thread, which reduces the requirements for technical personnel, does not rely on external components to provide services, and reduces operation and maintenance costs.

[0095] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data processing method provided in any of the above embodiments is implemented.

[0096] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the data processing method provided by any of the above embodiments is implemented.

[0097] Figure 6 An exemplary system architecture is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0098] like Figure 6 As shown, the system architecture may include an electronic device 601, a source server 602 and a target server 603, wherein the source server 602 may be a Kafka server, and the target server 603 may be any one of an HDFS server, an HBase server, a Clickhouse server, a Hive server, and an Elasticsearch server. The electronic device 601 may be a synchronization server for synchronizing the data of the source server 602 to the target server 603, and a K8s environment may be configured in the electronic device 602. Specifically, in an embodiment of the present invention, a thread framework for implementing the data processing method of an embodiment of the present invention may be deployed in a K8s environment in the form of a container for operation.

[0099] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the electronic device 601 , and accordingly, the data processing device is generally disposed in the electronic device 601 .

[0100] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of an electronic device suitable for implementing an embodiment of the present invention. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0101] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0102] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0103] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0104] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be arranged in a processor. For example, it may be described as follows: a processor includes a first processing module, a second processing module, and a third processing module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0107] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining source data from a source server through a first-level thread, and storing the source data in a first-level cache; obtaining the source data from the first-level cache through a second-level thread, and deserializing the source data to obtain target data, and storing the target data in a second-level cache; obtaining the target data from the second-level cache through a third-level thread, and writing the target data to a target server.

[0108] According to the technical solution of the embodiment of the present invention, the source data can be obtained from the source server through the first-level thread and stored in the first-level cache, the source data can be obtained from the first-level cache through the second-level thread, and the source data can be deserialized to obtain the target data, and the target data can be stored in the second-level cache, the target data can be obtained from the second-level cache through the third-level thread, and the target data can be written to the target server. That is, in the embodiment of the present invention, the data in the source server can be synchronized to the target server by calling each thread, so it is only necessary to configure the operating environment of the thread framework composed of each thread, which reduces the requirements for technical personnel, does not rely on external components to provide services, and reduces operation and maintenance costs.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: Applicable to synchronization servers, including: A source client is created through a primary thread, the source client is used to obtain source data subscribed by the primary thread from a topic queue created by a source server, and the source data is stored in a primary cache, and a data consumption record of the source client is created through the primary thread, and there is one primary thread; Acquire the source data from the first-level cache in parallel through multiple second-level threads, deserialize the source data to obtain target data, and store the target data in the second-level cache; The target data is obtained from the secondary cache in parallel through multiple third-level threads, and the target data is written to the target server; wherein the writing of the target data to the target server comprises: creating a target client through the third-level thread, and sending the data consumption record to the target client created by the third-level thread through the second-level thread; using the target client to write the target data in batches to the target server according to the data consumption record; The data consumption record is sent to the source server through a fourth-level thread.

2. The data processing method according to claim 1, characterized in that: The storing the source data into the first-level cache comprises: performing a hash modulus operation on the source data according to the number of the first-level cache to obtain a first modulus value, determining a first-level target cache from the first-level cache according to the first modulus value, and storing the source data into the first-level target cache; Storing the target data into the secondary cache includes: performing a hash modulus operation on the target data according to the number of the secondary caches to obtain a second modulus value, determining a secondary target cache from the secondary cache according to the second modulus value, and storing the target data into the secondary target cache.

3. The data processing method according to claim 1, characterized in that: The obtaining the source data from the first-level cache in parallel by using a plurality of second-level threads includes: determining the first-level cache corresponding to the second-level thread according to the first-level corresponding relationship, and obtaining the source data from the first-level cache corresponding to the second-level thread by using the second-level thread; The method of obtaining the target data from the secondary cache in parallel through multiple level 3 threads includes: determining the secondary cache corresponding to the level 3 thread according to a secondary corresponding relationship, and obtaining the target data from the secondary cache corresponding to the level 3 thread through the level 3 thread.

4. The data processing method according to claim 1, characterized in that: Writing the target data into the target server through the three-level thread includes: Determining a storage format supported by the target server; The target data is written into the target server according to the storage format through the third-level thread.

5. The data processing method according to claim 1, characterized in that: The source server includes a Kafka server, and the target server includes any one of a distributed file system HDFS server, an HBase server, a Clickhouse server, a Hive server, and an Elasticsearch server.

6. A data processing device, characterized in that: Applicable to synchronization servers, including: A first processing module is used to create a source client through a primary thread, use the source client to obtain source data subscribed by the primary thread from a topic queue created by a source server, store the source data into a primary cache, and create a data consumption record of the source client through the primary thread, wherein there is one primary thread; A second processing module is used to obtain the source data from the first-level cache in parallel through multiple second-level threads, deserialize the source data to obtain target data, and store the target data in the second-level cache; A third processing module is configured to obtain the target data from the secondary cache in parallel through multiple third-level threads, and write the target data to a target server; wherein, writing the target data to the target server comprises: creating a target client through the third-level thread, and sending the data consumption record to the target client created by the third-level thread through the second-level thread; and using the target client to write the target data in batches to the target server according to the data consumption record; The fourth processing module is used to send the data consumption record to the source server through a fourth-level thread.

7. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method as claimed in any one of claims 1 to 5 when executing the program.

8. A data processing system, comprising a source server, a target server, and an electronic device for executing the data processing method as claimed in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method according to any one of claims 1 to 5 is implemented.

Images