Data processing method and device, computing equipment and storage medium

Abstract

The invention provides a data processing method, which is used for processing target data fragments, the target data fragments are distributed to a target server by a corresponding database or a data analysis system, the target server comprises a plurality of nodes, and the nodes are distributed to the target server. Each of the plurality of nodes comprises at least one processing core and a corresponding memory to run a corresponding thread, the method comprises: creating at least one thread pool, each of the at least one thread pool comprising at least one thread, each thread pool in the at least one thread pool runs on a corresponding node in the plurality of nodes; according to the load condition of each node in the multiple nodes, a target node is selected from the multiple nodes, and a corresponding thread pool in the at least one thread pool runs on the target node; and processing the target data fragment by using a corresponding thread pool running on the target node, such as query operation.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data processing method, apparatus, computing device, computer-readable storage medium, and computer program product. Background Art

[0002] In a server including multiple processing cores (e.g., central processing units), the multiple processing cores can share the same memory (physical memory). In this case, the time required for each processing core to access the memory is the same. However, since each processing core is connected to the north bridge chip through the front side bus (FSB), and the north bridge chip is then connected to the memory (the memory controller is integrated in the north bridge chip), as the number of processing cores increases, the contention between the processing cores for the bus and the north bridge chip becomes more and more intense, resulting in an increasingly obvious performance bottleneck in response time.

[0003] In the related art, each processing core can be provided with a directly connected memory (also called local memory), and each processing core is connected through a Quick Path Interconnect (QPI) bus. Each processing core can access the memory that is not directly connected to itself (also called remote memory) through the QPI bus. However, when the server runs different threads, the nodes where the processing cores are located may be allocated uneven memory, resulting in frequent memory swapping (conversion between physical memory and virtual memory). In addition, the delay caused by the memory access of each processing core across nodes may cause system jitter. Summary of the invention

[0004] In view of this, the present disclosure provides a data processing method, apparatus, computing device, computer-readable storage medium, and computer program product to alleviate, mitigate, or even eliminate the above-mentioned problems.

[0005] According to one aspect of the present disclosure, there is provided a data processing method for processing a target data slice allocated to a target server, wherein the target server comprises a plurality of nodes, each of the plurality of nodes comprises at least one processing core and corresponding memory to run a corresponding thread, the method comprising: creating at least one thread pool, each thread pool in the at least one thread pool comprises at least one thread, and each thread pool in the at least one thread pool runs on a corresponding node in the plurality of nodes; selecting a target node from the plurality of nodes according to a load condition of each node in the plurality of nodes, the target node having a corresponding thread pool in the at least one thread pool running on it; and processing the target data slice using the corresponding thread pool running on the target node.

[0006] According to some embodiments of the present disclosure, the load condition of each node among the multiple nodes is determined based on the amount of data of other data shards different from the target data shard currently processed by each node, and wherein selecting a target node from the multiple nodes based on the load condition of each node among the multiple nodes includes: selecting a node with the smallest amount of data from the multiple nodes as the target node based on the amount of data of the other data shards currently processed by each node.

[0007] According to some embodiments of the present disclosure, the data volume of the other data shards currently processed by each node is determined based on at least one of the following: the number of the other data shards currently processed by each node; and the sum of the sizes of the other data shards currently processed by each node.

[0008] According to some embodiments of the present disclosure, the load condition of each of the multiple nodes is determined based on the number of idle processing cores and the size of free memory included in each node, and wherein selecting a target node from the multiple nodes based on the load condition of each of the multiple nodes includes: determining the product of the number of idle processing cores and the size of free memory included in each of the multiple nodes, and selecting a node with the largest product from the multiple nodes as the target node; or determining the weighted sum of the number of idle processing cores and the size of free memory included in each of the multiple nodes, and selecting a node with the largest weighted sum from the multiple nodes as the target node.

[0009] According to some embodiments of the present disclosure, the method also includes: in response to the target data shard being deleted from the target server or moved to another server different from the target server, updating the load condition of the target node, and performing load balancing processing on the multiple nodes based on the updated load condition of the target node and the load conditions of other nodes among the multiple nodes different from the target node.

[0010] According to some embodiments of the present disclosure, the balancing processing performed on the multiple nodes includes: moving at least one data shard currently processed by a node with a heavier load among the multiple nodes to a node with a lighter load among the multiple nodes.

[0011] According to some embodiments of the present disclosure, the at least one thread pool corresponds one-to-one to the multiple nodes.

[0012] According to some embodiments of the present disclosure, the processing of the target data shard includes at least one of the following: writing the target data shard into the memory of the target node; querying the target data shard from the memory of the target node; and merging at least a portion of the sub-data in the target data shard.

[0013] According to another aspect of the present disclosure, a business system is provided, which includes at least one server, the at least one server includes multiple nodes, each of the multiple nodes includes at least one processing core and corresponding memory to run corresponding threads, wherein the at least one server uses any method provided according to the aforementioned aspects of the present disclosure to process target data shards, and the target data shards are allocated to the at least one server by the business system.

[0014] According to another aspect of the present disclosure, a data processing device is provided for processing a target data slice allocated to a target server, wherein the target server includes multiple nodes, each of the multiple nodes includes at least one processing core and corresponding memory to run corresponding threads, and the device includes: a thread pool creation module, configured to create at least one thread pool, each thread pool in the at least one thread pool includes at least one thread, and each thread pool in the at least one thread pool runs on a corresponding node in the multiple nodes; a node selection module, configured to select a target node from the multiple nodes according to the load conditions of each node in the multiple nodes, and a corresponding thread pool in the at least one thread pool runs on the target node; and a data processing module, configured to process the target data slice using the corresponding thread pool running on the target node.

[0015] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute any one of the methods provided according to the aforementioned aspects of the present disclosure when the computer-executable instructions are executed by the processor.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, any method provided according to the aforementioned aspects of the present disclosure is executed.

[0017] According to yet another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, wherein when the computer executable instructions are executed by a processor, any one of the methods provided according to the aforementioned aspects of the present disclosure is performed.

[0018] According to the data processing method provided by the present disclosure, at least one thread pool can be created, each thread pool in the at least one thread pool includes at least one thread, and each thread pool in the at least one thread pool runs on a corresponding node in the multiple nodes, and then according to the load of each node in the multiple nodes, a target node (on which a corresponding thread pool in the at least one thread pool runs) is selected from the multiple nodes, so that the memory of each node in the multiple nodes is more evenly distributed, reducing the possibility of memory exchange. Finally, the corresponding thread pool running on the target node is used to process the target data fragments allocated to the target server. This process does not require cross-node memory access, thereby improving system stability and resource utilization.

[0019] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the following description of exemplary embodiments in conjunction with the accompanying drawings, more details, features and advantages of the technical solutions of the present disclosure are disclosed, in which:

[0021] Figure 1 Schematically illustrates example scenarios in which the technical solutions provided according to some embodiments of the present disclosure can be applied;

[0022] Figure 2A An example principle diagram schematically shows a technical solution in which multiple processing cores share the same memory in the related art;

[0023] Figure 2B An example schematic diagram of a memory allocation scheme for multiple processing cores with corresponding local memories in the related art is schematically shown;

[0024] Figure 2C An example principle diagram schematically shows a data processing process corresponding to a data structure used in the related art;

[0025] Figure 3 An example flow chart of a data processing method according to some embodiments of the present disclosure is schematically shown;

[0026] Figure 4A-4B Schematically shows Figure 3 The difference between the data processing method in the present invention and the data processing method in the related art in terms of memory access;

[0027] Figure 5 An example block diagram of a data processing device according to some embodiments of the present disclosure is schematically shown;

[0028] Figure 6 An example system is illustrated that includes an example computing device that is representative of one or more systems and / or devices that may implement the various techniques described herein. DETAILED DESCRIPTION

[0029] Several embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the technical solutions of the present disclosure. The technical solutions of the present disclosure can be embodied in many different forms and purposes, and should not be limited to the embodiments described herein. These embodiments are provided to make the technical solutions of the present disclosure clear and complete, but the described embodiments do not limit the protection scope of the present disclosure.

[0030] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined as such herein.

[0031] Before describing the embodiments of the present disclosure in detail, some related concepts are first explained.

[0032] 1. Cloud computing: refers to the delivery and use model of IT infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use model of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0033] With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model from a conceptual perspective.

[0034] 2. Big data: refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive, high-growth, and diversified information asset that requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process large amounts of data within a tolerable time frame. Technologies suitable for big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0035] Figure 1 An example scenario 100 is schematically shown in which the technical solutions provided according to some embodiments of the present disclosure can be applied. Figure 1 As shown, the scenario 100 may include a server 110 (e.g., a physical machine, a cloud virtual machine), a network 120, and a terminal device 130. The server 110 and the terminal device 130 may communicate via the network 120. The server 110 may include multiple nodes, such as Figure 1 As shown, server 110 includes node 111, node 112 and node 113, each of which includes at least one processing core and corresponding memory to run a corresponding thread. Figure 1 As shown, node 111 may include processing core 1111 and memory 1112, node 112 may include processing core 1121 and memory 1122, and node 113 may include processing core 1131 and memory 1132. It should be noted that server 110 may include fewer (e.g., two) or more (e.g., four) nodes, and each node in server 110 may include a greater number of processing cores (e.g., in addition to processing core 1111, node 111 may also include other more processing cores), and the present disclosure does not limit this.

[0036] First, at least one thread pool is created, each of the at least one thread pool includes at least one thread, and each of the at least one thread pool runs on a corresponding node in the plurality of nodes. Exemplarily, three thread pools corresponding to nodes 111, 112, and 113 (hereinafter referred to as the first thread pool, the second thread pool, and the third thread pool, respectively) can be created, wherein the first thread pool includes at least one thread and runs on node 111, the second thread pool includes at least one thread and runs on node 112, and the third thread pool includes at least one thread and runs on node 113. Taking node 111 as an example, the thread pool running thereon is used to process the corresponding data shards (for example, write, query, merge, etc., different processing can be completed by different threads in the thread pool), specifically, the threads in the thread pool are loaded into memory 1112, and then wait to enter the processing core 1111 (depending on the current state of the processing core 1111, queuing may be required), and after entering the processing core 1111, the corresponding threads run on the processing core 1111 to complete the processing of the corresponding data shards.

[0037] It should be noted that although Figure 1 In the example, three thread pools (i.e., the first thread pool, the second thread pool, and the third thread pool) corresponding to the nodes 111, 112, and 113 in the server 110 are created, but it should be understood by those skilled in the art that fewer or more thread pools can be created, for example, only a thread pool corresponding to the node 111 can be created (i.e., only the first thread pool is created). In addition, the threads of the same or different numbers can be included in each thread pool created, for example, the first thread pool can include two threads for performing read and write operations (processing), and the second thread pool can include the same thread as the first thread pool, i.e., also including two threads for performing read and write operations (processing), alternatively, the second thread pool can include only one thread (e.g., for performing a thread for a read operation). In addition, the threads of the same or different functions can be included in each thread pool created, for example, the first thread pool can include two threads for performing read and merge operations, and the third thread pool can include the same thread as the first thread pool, i.e., also including two threads for performing read and merge operations, alternatively, the second thread pool can include only one thread (e.g., for performing a thread for a write operation). In addition, in the present application, the number of thread pools created may be less than or equal to the number of nodes in the server. When the number of thread pools created is less than the number of nodes in the server, the created thread pools may correspond one-to-one to some but not all of the nodes in the server; when the number of thread pools created is equal to the number of nodes in the server, the created thread pools may correspond one-to-one to all of the nodes in the server.

[0038] Next, according to the load of each node in the multiple nodes, a target node is selected from the multiple nodes, and a corresponding thread pool in the at least one thread pool is running on the target node. Figure 1 As shown, the load of nodes 111, 112, and 113 can be determined according to the amount of data of other data slices that are currently processed by nodes 111, 112, and 113, which are different from the target data slice to be processed and allocated to the target server at this time. Alternatively, the load of nodes 111, 112, and 113 can be determined according to the number of idle processing cores and the size of idle memory included in nodes 111, 112, and 113. The target node selected from the plurality of nodes can be the node with the lightest load among the plurality of nodes.

[0039] Finally, the target data shards assigned to the target server (e.g., server 110) are processed using the corresponding thread pool running on the target node. Exemplarily, the target node may be node 111, and the target data shards assigned to server 110 are processed (e.g., write, query, and merge operations) using the first thread pool running on node 111.

[0040] For example, a distributed business system (e.g., data storage / analysis system ElasticSearch, MongoDB, Kafka, etc.) may be deployed on the server 110, and the user may access the system deployed on the server 110 through the terminal device 130 via the network 120. The user's different access requirements / instructions may cause the system to perform different types of processing on the data shards on the server 110, i.e., the various processing performed on the target data shards allocated to the server 110 as described above. Then, the user may obtain the corresponding processing results (e.g., query results) through the terminal device 130.

[0041] As a result, the memory of each node in the server 110 can be more evenly distributed, reducing the possibility of memory swapping. In addition, since the target data shards allocated to the target server are processed by the corresponding thread pool running on the selected target node, this process does not require cross-node memory access, which improves system stability and resource utilization, and further helps the server 110 provide users with high-performance distributed data storage / analysis services.

[0042] It should be noted that the term "user" used in this article refers to any party that can interact with the cloud storage service system for data, including but not limited to people, program software, network platforms, and even machines.

[0043] In the present disclosure, server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0044] In the present disclosure, the terminal device 130 may be various types of devices, such as mobile phones, tablet computers, laptop computers, wearable devices such as smart watches, vehicle-mounted devices, etc. A client may be deployed on the terminal device 130, which may provide an interface for access operations such as data upload and download, and optionally provide other services, and may take any of the following forms: a locally installed application, a web program accessed via a browser, etc. The terminal device 130 and the server 110 may be directly or indirectly connected via wired or wireless communication, which is not limited in the present disclosure.

[0045] In addition, in the present disclosure, the network 120 can be a wired network connected via cables, optical fibers, etc., or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, etc.

[0046] Below through Figure 2A-2C To explain the relevant technical background of the present invention and the problems existing in the prior art in more detail.

[0047] Figure 2A The following schematically shows an example principle diagram of a technical solution in which multiple processing cores share the same memory in the related art. Figure 2A As shown, the server 210 includes three processing cores (CPU_1, CPU_2, and CPU_3), and CPU_1, CPU_2, and CPU_3 share the same memory 211 (in this case, each processing core takes the same amount of time to access the memory 211). CPU_1, CPU_2, and CPU_3 are connected to the north bridge (chip) through the front-end bus, and then connected to the memory 211 by the north bridge chip. Figure 2A As can be seen from the figure, as the number of processing cores increases, the contention between each processing core for the bus and north bridge chip will become more intense, resulting in an increasingly obvious performance bottleneck in response time. In the computer field, this architecture where multiple processing cores share the same physical memory and each processing core takes the same amount of time to access the memory is also called Unified Memory Access (UMA).

[0048] Figure 2BSchematically shows an example principle diagram of a memory allocation scheme for multiple processing cores with corresponding local memories in the related art. Figure 2B As shown, server 220 includes two nodes (Node_1 and Node_2), where Node_1 includes a processing core (CPU_4) and a memory 221 directly connected thereto, and Node_2 includes a processing core (CPU_5) and a memory 222 directly connected thereto. CPU_4 and CPU_5 are connected via a QPI bus. For example, in addition to accessing local memory 221, CPU_4 can access memory 222 that is not directly connected to itself. When server 220 runs different threads, the nodes where the processing cores are located may be unevenly allocated memory, such as Figure 2B As shown, assuming that each node (Node_1 and Node_2) in server 220 is allocated 32GB of memory and adopts a strategy of allocating memory locally first (this strategy is commonly used in Linux systems), then when the shared buffer occupies 50GB of memory, it will allocate 32GB and 18GB of memory in Node_1 and Node_2 respectively. Assuming that the memory upper limit of server 220 is 64GB, in this case, although the 50GB of memory has not reached the memory upper limit, when some data must be allocated on the memory corresponding to Node_1, it will cause the memory in Node_1 to be swapped out to make room for new memory requests. As described above in the background technology section, the frequent swapping in and out of the shared buffer memory will cause the performance of server 220 to degrade (for example, causing greater service delays). In the computer field, this architecture of setting different physical memories for multiple processing cores is also called Non-Uniform Memory Access (NUMA). It should be noted that the present disclosure does not limit the architecture adopted by the target server, although it is combined here. Figure 2B An example of NUMA architecture is given, and those skilled in the art should understand that the data processing method provided by the present disclosure can be applied to various servers that have cross-node access in the processing of data shards.

[0049] Figure 2C The following schematic diagram shows an example of a data processing process corresponding to a data structure used in the related art. For distributed data storage / analysis systems (e.g., ElasticSearch, MongoDB, Kafka, etc.), in order to achieve high scalability, data is generally segmented to obtain data shards. Figure 2CAs shown in the figure, there are several data shards on both Server_1 and Server_2. Specifically, there are data shards Shard_1, Shard_2 and Shard_3 on Server_1, and there are data shards Shard_4, Shard_5 and Shard_6 on Server_2. Taking the core data structure LSM tree (LogStructured Merge Tree) relied on by many key-value or log-type data storage / analysis systems (for example, ElasticSearch, BigTable, HBase, Cassandra, LevelDB, SQLite, Scylla, RocksDB, etc.) as an example, the LSM processing process mainly includes the following: Write, that is, write the initial data segment, for example, write Figure 2C The data segments Seg_1, Seg_2, Seg_3, Seg_4, Seg_5 and Seg_20 shown (they can be regarded as different sub-data of the same data shard); Merge: Merge files regularly, that is, merge a series of small segments to generate a large segment. For example, Figure 2C As shown, data segments Seg_1, Seg_2 and Seg_3 are merged to generate data segment Seg_6, and data segments Seg_4 and Seg_5 are merged to generate data segment Seg_7, and then data segments Seg_6 and Seg_7 are merged to generate data segment Seg_10; Search: query the currently valid data segments, such as Figure 2C As shown, Seg_10 and Seg_20 can be queried. Figure 2C As can be seen from the example, there are many processing stages for the overall processing of a data slice, such as Write / Merge / Search. In fact, in a NUMA architecture server, the processing threads of different stages of the same data slice are likely to run on the processing cores of different nodes. For further analysis of this phenomenon and the corresponding issues, please refer to the following article about Figure 3 as well as Figure 4A-4B Description.

[0050] Figure 3 The following schematically shows an example flow chart of a data processing method 300 according to some embodiments of the present disclosure. Figure 1The data processing method 300 is implemented by the server 110 shown, which is of course not restrictive. The data processing method 300 is used to process the target data slices assigned to the target server, the target server comprising a plurality of nodes, each of the plurality of nodes comprising at least one processing core and corresponding memory to run a corresponding thread.

[0051] Specifically, at step 310, at least one thread pool may be created, each of the at least one thread pool including at least one thread, and each of the at least one thread pool runs on a corresponding node among the plurality of nodes. Figure 2B For example, two thread pools corresponding to nodes Node_1 and Node_2 (hereinafter referred to as the fourth thread pool and the fifth thread pool, respectively) can be created, wherein the fourth thread pool includes at least one thread and runs on node Node_1, and the fifth thread pool includes at least one thread and runs on node Node_2. Taking node Node_1 as an example, the fourth thread pool running thereon is used to process the corresponding data shards (for example, writing, querying, merging, etc., different processing can be completed by different threads in the fourth thread pool). Specifically, the threads in the fourth thread pool are loaded into the memory 221, and then wait to enter the processing core CPU_4 (depending on the current state of the processing core CPU_4, queuing may be required). After entering the processing core CPU_4, the corresponding thread runs on the processing core CPU_4 to complete the processing of the corresponding data shards.

[0052] It should be noted that although in reference Figure 2B In the above-described example, two thread pools (i.e., the fourth thread pool and the fifth thread pool) corresponding to the nodes Node_1 and Node_2 in the server 220 (i.e., the target server) are created, but those skilled in the art will appreciate that fewer or more thread pools can be created, for example, only a thread pool corresponding to the node Node_1 can be created (i.e., only the fourth thread pool is created). In addition, the threads of the same or different numbers can be included in each thread pool created, for example, the fourth thread pool can include two threads for performing read and write operations (processing), and the fifth thread pool can include the same thread as the fourth thread pool, i.e., also including two threads for performing read and write operations (processing), alternatively, the fourth thread pool can include only one thread (e.g., for performing a thread for a read operation). In addition, the threads of the same or different functions can be included in each thread pool created, for example, the fourth thread pool can include two threads for performing read and merge operations, and the fifth thread pool can include the same thread as the fourth thread pool, i.e., also including two threads for performing read and merge operations, alternatively, the fifth thread pool can include only one thread (e.g., for performing a thread for a write operation).

[0053] In step 320, a target node may be selected from the plurality of nodes according to the load of each node in the plurality of nodes, and a corresponding thread pool in the at least one thread pool is running on the target node. Figure 2B For example, Figure 2B As shown, the load of nodes Node_1 and Node_2 can be determined based on the amount of data of other data slices that are currently processed by nodes Node_1 and Node_2 and are different from the target data slice to be processed and allocated to server 220 at this time. Alternatively, the load of nodes Node_1 and Node_2 can be determined based on the number of idle processing cores and the size of idle memory included in nodes Node_1 and Node_2. Exemplarily, when the idle memory included in node Node_1 is greater than the idle memory included in node Node_2, node Node_1 can be used as the node with the lightest load (i.e., the target node) among nodes Node_1 and Node_2.

[0054] Finally, in step 330, the target data shards allocated to the target server may be processed using the corresponding thread pool running on the target node. Exemplarily, the node Node_1 is used as the target node, and the fourth thread pool running on the node Node_1 is used to process the target data shards allocated to the server 220 (e.g., write, query, and merge operations).

[0055] pass Figure 3 The data processing method 300 shown can select the target node according to the load of each node, which helps to distribute the memory of each node more evenly, thereby reducing the possibility of memory swapping and improving resource utilization. In addition, by using the corresponding thread pool running on the selected target node, the target data shards allocated to the target server can be processed without cross-node memory access, which improves system stability and helps the target server provide users with high-performance distributed data storage / analysis services.

[0056] Figure 3 The difference between the data processing method in the present invention and the data processing method in the related art in terms of memory access is Figure 4A-4B It should be noted that, in order to facilitate comparison with related technologies, Figure 4A-4B The configuration and Figure 2B Same as in.

[0057] like Figure 4AAs shown above, in the related art, for example, for the processing of data shard X, the writes (Write) at three different times (T1, T2 and T3) are completed on the processing cores CPU_4, CPU_5 and CPU_4 respectively. Since the processing cores CPU_4 and CPU_5 belong to different nodes (Node_1 and Node_2), the cache data generated in the write process will use the memory of different nodes. Specifically, Figure 4A As shown on the upper side, the write operation (write processing) performed on the processing core CPU_4 at time T1 and time T3 will use the memory on the node Node_1, while the write operation performed on the processing core CPU_5 at time T2 will use the memory on the node Node_2. The subsequent merge operation is performed on the processing core CPU_5 at time T4, and the query operation is performed on the processing core CPU_4 at time T5. It can be seen that in the subsequent merge and query stages, no matter which node the data shard is processed on, cross-access to the memory may occur. Therefore, in the related art, for the processing of data shards, since the processing threads at different times may run on the processing cores of different nodes, it is often necessary to cross-access the memory of each node, thereby generating a large amount of delay and causing system jitter.

[0058] For comparison, use Figure 3 In the data processing method, two thread pools (i.e., the fourth thread pool and the fifth thread pool) can be created, wherein the fourth thread pool includes different threads for completing write, merge, and query operations and runs on the node Node_1. Exemplarily, with the node Node_1 as the target node, for the processing of the same data shard X (i.e., the target data shard), the writing at three different times (T6, T7, and T8) is completed on the processing core CPU_4 of the node Node_1. Since the processing core CPU_4 belongs to the node Node_1, the cache data generated in the writing process will use the memory of the same node. Specifically, Figure 4A As shown on the lower side, the write operations performed on the processing core CPU_4 at times T6, T7, and T8 will use the memory on the node Node_1. The subsequent merge operation is performed on the processing core CPU_4 at time T9, and the query operation is performed on the processing core CPU_4 at time T10. It can be seen that by utilizing the fourth thread pool running on the node Node_1, various processing of the target data shard at different times are performed on the node Node_1. This process does not require cross-access to the memory of each node (access to the memory on the same node Node_1 is sufficient), thereby avoiding the generation of a large amount of delay and reducing system jitter.

[0059] Figure 4B A further example of how data shards are distributed among different nodes is given. Figure 4BAs shown above, in the related art, server 410 is used to process four data shards (Shard_1, Shard_2, Shard_3, and Shard_4) assigned thereto, wherein the thread pool running on node Node_1 includes three threads (Thread_1, Thread_2, and Thread_3), and the thread pool running on node Node_2 also includes three threads (Thread_4, Thread_5, and Thread_6). Exemplarily, Thread_1 and Thread_4 are threads used to complete write operations, Thread_2 and Thread_5 are threads used to complete merge operations, and Thread_3 and Thread_6 are threads used to complete query operations. For the processing of Shard_1, Shard_2, Shard_3, and Shard_4, the processing threads at different times may run on the processing cores of different nodes, resulting in frequent cross-access to the memory of each node and causing delays. As mentioned above, Figure 4A This has been elaborated on in detail and will not be repeated here.

[0060] As a comparison, Figure 4B As shown below, use Figure 3 In the data processing method, the two created thread pools (i.e., the fourth thread pool and the fifth thread pool) run on nodes Node_1 and Node_2, respectively. Exemplarily, the processing of data shards Shard_1 and Shard_2 is completed on the processing core CPU_4 of node Node_1, and the cache data generated in the corresponding processing will use the memory of node Node_1. The processing of data shards Shard_3 and Shard_4 is completed on the processing core CPU_5 of node Node_2, and the cache data generated in the corresponding processing will use the memory of node Node_2. It can be seen that by utilizing the fourth thread pool running on node Node_1 and the fifth thread pool running on node Node_2, various processing of the same data shard at different times are performed on the same node, thereby realizing fast access from the processing core to the local memory and avoiding the increased delay caused by remote access (cross access).

[0061] In some embodiments, the load of each of the multiple nodes is determined based on the amount of data of other data shards different from the target data shard currently processed by each node, and wherein selecting a target node from the multiple nodes based on the load of each of the multiple nodes includes: selecting a node with the smallest amount of data from the multiple nodes as the target node based on the amount of data of the other data shards currently processed by each node. Figure 4BTaking the illustrated embodiment as an example, assuming that the target data shard Shard_X (not shown) is assigned to the target server 420 for processing, the data shards currently processed by the node Node_1 include Shard_1 and Shard_2, and the data shards currently processed by the node Node_2 include Shard_3, Shard_4 and Shard_5 (not shown), then the number of the aforementioned data shards currently processed by the node Node_1 and the node Node_2 can be used as the data amount of other data shards different from the target data shard Shard_X currently processed by the corresponding nodes, and then the node Node_1 with the smallest data amount (2 in this example) is used as the target node, so that the corresponding thread pool running on the node Node_1 with a lighter load (fewer data shards to process) is used to process the target data shard Shard_X, thereby improving resource utilization and helping to reduce latency.

[0062] Alternatively, in some embodiments, the data volume of the other data slices currently processed by each node may be determined based on the sum of the sizes of the other data slices currently processed by each node. Figure 4B Taking the illustrated embodiment as an example, assuming that the target data shard Shard_X is assigned to the target server 420 for processing, the data shards currently processed by the node Node_1 include Shard_1 and Shard_2, and the data shards currently processed by the node Node_2 include Shard_3 and Shard_4, wherein the sum of the sizes of Shard_1 and Shard_2 is 1.5MB, and the sum of the sizes of Shard_3 and Shard_4 is 0.5MB. Then, the sum of the sizes of the aforementioned data shards currently processed by the node Node_1 and the node Node_2 can be used as the data volume of other data shards different from the target data shard Shard_X currently processed by the corresponding nodes, and then the node Node_2 with the smallest data volume (0.5MB in this example) is used as the target node, so that the target data shard Shard_X is processed by the corresponding thread pool running on the node Node_2 with a lighter load (the sum of the sizes of the processed data shards is smaller), thereby improving resource utilization and helping to reduce latency.

[0063] Alternatively, in some embodiments, the data volume of the other data slices currently processed by each node can comprehensively consider the number of the other data slices currently processed by each node, and the sum of the sizes of the other data slices currently processed by each node. In other words, compared with the aforementioned two schemes for determining the amount of data, in this embodiment, the two can be comprehensively considered to determine the load of each node more comprehensively. Exemplarily, the number of the other data slices currently processed by each node and the sum of the sizes of the other data slices currently processed by each node can be weighted and summed, and then the node with a smaller weighted sum can be used as a node with a lighter load (i.e., the target node). Those skilled in the art should understand that the number of the other data slices currently processed by each node and the sum of the sizes of the other data slices currently processed by each node do not belong to the same dimension, and they can be normalized according to the situation in the process of weighted summation. In addition, the weights of the two can be flexibly set as needed in the process of weighted summation, and the present disclosure is not limited here.

[0064] In some embodiments, the load condition of each of the multiple nodes is determined based on the number of idle processing cores and the size of idle memory included in each node, and wherein selecting a target node from the multiple nodes based on the load condition of each of the multiple nodes includes: determining the product of the number of idle processing cores and the size of idle memory included in each of the multiple nodes, and selecting a node with the largest product from the multiple nodes as the target node. Figure 4B As an example, assume that the target data shard Shard_X is assigned to the target server 420 for processing. At this time, the idle processing core included in the node Node_1 is CPU_4 (see Figure 2B ), the free memory size is 16GB, and the free processing core included in node Node_2 is CPU_5 (see Figure 2B ), the free memory size is 6GB, then the product of the number of free processing cores and the size of the free memory included in each node can be determined. Specifically, in this example, the product corresponding to node Node_1 is 16, and the product corresponding to node Node_2 is 6. Therefore, node Node_1 with the largest product can be selected from node Node_1 and node Node_2 as the target node.

[0065] Alternatively or additionally, in some embodiments, the load of each node may be measured by using the weighted sum of the number of idle processing cores and the size of idle memory included in each of the multiple nodes, and then the node with the largest weighted sum is selected from the multiple nodes as the target node. Figure 4BTake the illustrated embodiment as an example, and the node Node_1 and the node Node_2 have the same configuration as described in the previous paragraph. In this case, assuming that the weights of the number of idle processing cores and the size of free memory included in each node are both 0.5, the weighted sum of the number of idle processing cores and the size of free memory corresponding to the node Node_1 is 8.5, and the weighted sum of the number of idle processing cores and the size of free memory corresponding to the node Node_2 is 3.5, so the node Node_1 with the largest weighted sum can be selected from the nodes Node_1 and Node_2 as the target node. As described above, those skilled in the art should understand that the number of idle processing cores and the size of free memory corresponding to each node do not belong to the same dimension, and they can be normalized according to the situation in the process of weighted summation. In addition, in the process of weighted summation, the weights of the two can be flexibly set as needed, and the present disclosure is not limited here. Whether considering the product of the number of idle processing cores and the size of idle memory included in each node, or the weighted sum of the number of idle processing cores and the size of idle memory included in each node, it is helpful to select nodes with more idle resources to process the target data shards, which not only improves resource utilization but also helps to reduce the processing delay of the target data shards.

[0066] In some embodiments, the data processing method further includes: in response to the target data shard being deleted from the target server or being moved to other servers different from the target server, updating the load condition of the target node, and performing load balancing processing on the multiple nodes according to the updated load condition of the target node and the load conditions of other nodes among the multiple nodes different from the target node. When there are many data shards to be processed by the target server, obtaining the latest and accurate load conditions of each node included in the target server helps to improve resource utilization and reduce delays. Therefore, when the target data shard is deleted from the target server or moved to other servers different from the target server, since the target data shard no longer exists on the target server in this case, the load condition of the target node that originally processed the target data shard can be updated, and then the multiple nodes can be load balanced, thereby further improving resource utilization.

[0067] Exemplarily, in some embodiments, the balancing process performed on the multiple nodes includes: moving at least one data shard currently processed by a node with a heavier load among the multiple nodes to a node with a lighter load among the multiple nodes. Figure 4BAs an example, assume that the target data shard Shard_X is assigned to the node Node_1 in the target server 420 for processing. At this time, the idle processing core included in the node Node_1 is CPU_4 (see Figure 2B ), the free memory size is 12GB, and the free processing core included in node Node_2 is CPU_5 (see Figure 2B ), the free memory size is 3GB, and the load of the corresponding node is measured by determining the product of the number of free processing cores and the free memory size. In this example, it can be determined that the load of node Node_2 is heavier, while the load of node Node_1 is lighter. In this case, at least one data shard (for example, Shard_3) currently processed by node Node_2 can be moved to node Node_1, so that the load of each node is more balanced. Alternatively, other methods can be used to balance the multiple nodes. For example, memory swapping can be performed on the heavily loaded node Node_2, that is, some temporarily unused or less frequently used memory pages of the process running on node Node_2 are saved to the disk, and then physical memory pages are allocated to more urgent or more frequently used data shards for use, and these memory pages are read back from the disk to the physical memory when the process needs them.

[0068] It should be noted that, although in this example, the load of the corresponding node is measured by determining the product of the number of idle processing cores and the free memory size, that is, the node with a smaller product of the number of idle processing cores and the free memory size is considered to have a heavier load, while the node with a larger product of the number of idle processing cores and the free memory size is considered to have a lighter load. However, those skilled in the art should understand that other methods can be used to measure the "lightness" and "heaviness" of the node load. As a first example, the load of the corresponding node can be measured based on only one factor of the number of idle processing cores and the free memory size. For example, the node with a smaller number of idle processing cores is considered to be a node with a heavier load, while the node with a larger number of idle processing cores is considered to be a node with a lighter load. As a second example, the amount of data described above can be used to measure the load of the corresponding node, and the node with a smaller amount of data is considered to be a node with a lighter load, while the node with a larger amount of data is considered to be a node with a heavier load (the method for determining the amount of data has been described in detail above and will not be repeated here).

[0069] In some embodiments, the at least one thread pool corresponds to the plurality of nodes one by one. Figure 4B Taking the embodiment shown in FIG. 4 as an example, the nodes Node_1 and Node_2 in the target server 420 correspond to two thread pools respectively. Specifically, in Figure 4BIn the example shown, the thread pool corresponding to the node Node_1 includes three threads (Thread_1, Thread_2 and Thread_3), and the thread pool corresponding to the node Node_2 also includes three threads (Thread_4, Thread_5 and Thread_6). By setting a corresponding thread pool for each node in the target server 420, resource utilization can be optimized, that is, in this case, the processing of each data shard is performed on the corresponding same node (for example, the writing, merging and querying operations of Shard_1 can be completed using the three threads Thread_1, Thread_2 and Thread_3 running on the node Node_1), thereby avoiding cross-node memory access to the greatest extent.

[0070] In some embodiments, the processing of the target data shard includes at least one of the following: writing the target data shard to the memory of the target node; querying the target data shard from the memory of the target node; and merging at least a portion of the sub-data in the target data shard. For further understanding of these processes, please refer to the above Figure 2B-2C ,as well as Figure 4A-4B It should be noted that the present disclosure does not limit the functions of the threads in the created thread pool, that is, in addition to the above processing, other processing may be performed alternatively or additionally. Figure 2C Some examples of threads are given (including write, merge, and query threads), and those skilled in the art should understand that, depending on the specific scenario and the functional setting of the data storage / analysis system, a thread pool containing various threads for completing various operations (processing) can be flexibly created.

[0071] In some embodiments, a business system is provided, the business system includes at least one server, the at least one server includes multiple nodes, each of the multiple nodes includes at least one processing core and corresponding memory to run corresponding threads, wherein the at least one server uses any method provided in the foregoing embodiments of the present disclosure to process the target data shard, and the target data shard is allocated to the at least one server by the business system. Examples of such business systems may include, but are not limited to, various data storage / analysis systems that use the data processing method provided in the present disclosure, such as ElasticSearch, MongoDB, Kafka, etc.

[0072] Figure 5The following schematically shows an example block diagram of a data processing device 500 according to some embodiments of the present disclosure. The data processing device 500 is used to process target data slices allocated to a target server, wherein the target server includes multiple nodes, each of which includes at least one processing core and corresponding memory to run a corresponding thread. For example, the data processing device 500 can be deployed on Figure 1 As shown on the server 110. Figure 5 As shown, the data processing device 500 includes a thread pool creation module 510 , a node selection module 520 and a data processing module 530 .

[0073] Specifically, the thread pool creation module 510 can be configured to create at least one thread pool, each thread pool in the at least one thread pool includes at least one thread, and each thread pool in the at least one thread pool runs on a corresponding node among the multiple nodes; the node selection module 520 can be configured to select a target node from the multiple nodes according to the load conditions of each node in the multiple nodes, and the target node runs a corresponding thread pool in the at least one thread pool; and the data processing module 530 can be configured to use the corresponding thread pool running on the target node to process the target data slice.

[0074] It should be understood that the data processing device 500 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules in the device can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0075] In addition, the data processing device 500 can be used to implement the data processing method 300 described above, and the relevant details have been described in detail above, and for the sake of brevity, they are not repeated here. In addition, these devices can have the same features and advantages as described in the corresponding method.

[0076] Figure 6 An example system is illustrated that includes an example computing device 600 that represents one or more systems and / or devices that may implement the various techniques described herein. The computing device 600 may be, for example, a server used by a node in a blockchain, a device associated with a server, a system on a chip, and / or any other suitable computing device or computing system. Figure 5 The depicted data processing apparatus 500 may take the form of a computing device 600. Alternatively, the data processing apparatus 500 may be implemented as a computer program in the form of an application 616.

[0077] like Figure 6The illustrated example computing device 600 includes a processing system 611, one or more computer-readable media 612, and one or more I / O interfaces 613 that are communicatively coupled to each other. Although not shown, the computing device 600 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0078] Processing system 611 represents the functionality of performing one or more operations using hardware. Thus, processing system 611 is illustrated as including hardware elements 614 that may be configured as processors, functional blocks, and the like. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 614 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0079] Computer readable media 612 is illustrated as including memory / storage 615. Memory / storage 615 represents memory / storage capacity associated with one or more computer readable media. Memory / storage 615 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 615 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer readable media 612 may be configured in various other ways as further described below.

[0080] One or more I / O interfaces 613 represent functionality that allows a user to input commands and information to the computing device 600 using various input devices and optionally also allows information to be presented to the user and / or other components or devices using various output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can detect motion that does not involve touch as gestures using visible or invisible wavelengths (such as infrared frequencies)), and the like. Examples of output devices include a display device (e.g., a projector), a speaker, a printer, a network card, a tactile response device, and the like. Thus, the computing device 600 can be configured in various ways as further described below to support user interaction.

[0081] Computing device 600 also includes application 616. Application 616 may be, for example, a software instance of data processing apparatus 500, and in combination with other elements in computing device 600 implement the techniques described herein.

[0082] Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module", "function" and "component" as used herein generally represent software, firmware, hardware or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0083] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include a variety of media accessible by computing device 600. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0084] As opposed to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, computer-readable storage media refers to non-signal bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium or article of manufacture suitable for storing desired information and accessible by a computer.

[0085] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 600, such as via a network. Signal media typically can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information transfer media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed so as to encode information into the signal. By way of example and not limitation, communication media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared, and other wireless media.

[0086] As previously described, hardware elements 614 and computer-readable media 612 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices that perform program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0087] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 614. The computing device 600 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 614 of a processing system, the module may be implemented as a module executable by the computing device 600 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles (e.g., one or more computing devices 600 and / or processing systems 611) to implement the techniques, modules, and examples described herein.

[0088] In various embodiments, computing device 600 can be implemented in various configurations. For example, computing device 600 can be implemented as a computer-type device including a personal computer, a desktop computer, a multi-screen computer, a laptop computer, a netbook, etc. Computing device 600 can also be implemented as a mobile device-type device including mobile devices such as mobile phones, portable music players, portable game devices, tablet computers, multi-screen computers, etc. Computing device 600 can also be implemented as a television-type device, which includes a device with or connected to a generally larger screen in a casual viewing environment. These devices include televisions, set-top boxes, game consoles, etc.

[0089] The techniques described herein may be supported by these various configurations of computing device 600 and are not limited to the specific examples of the techniques described herein. Functionality may also be implemented in whole or in part on the "cloud" 620 by using a distributed system, such as through platform 622 as described below.

[0090] Cloud 620 includes and / or represents a platform 622 for resources 624. Platform 622 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 620. Resources 624 may include applications and / or data that may be used when executing computer processing on a server remote from computing device 600. Resources 624 may also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network.

[0091] The platform 622 may abstract resources and functionality to connect the computing device 600 with other computing devices. The platform 622 may also be used to abstract hierarchies of resources to provide hierarchies of corresponding levels of demand encountered for resources 624 implemented via the platform 622. Thus, in an interconnected device embodiment, implementation of the functionality described herein may be distributed throughout the system 600. For example, functionality may be implemented in part on the computing device 600 and through the platform 622 that abstracts functionality of the cloud 620.

[0092] It should be understood that, for the sake of clarity, embodiments of the present disclosure are described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to a suitable unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0093] It will be understood that although the first, second, third, etc. terms may be used to describe various devices, elements, parts or portions in this article, these devices, elements, parts or portions should not be limited by these terms. These terms are only used to distinguish one device, element, part or portion from another device, element, part or portion.

[0094] Although the present disclosure has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present disclosure is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and inclusion in different claims does not imply that the combination of features is not feasible and / or advantageous. The order of features in the claims does not imply any specific order in which the features must work. Furthermore, in the claims, the word "comprising" does not exclude other elements, and the term "one" or "an" does not exclude a plurality. The figure marks in the claims are provided only as clear examples and should not be interpreted as limiting the scope of the claims in any way.

[0095] It should be understood that, for the sake of clarity, embodiments of the present disclosure are described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to a suitable unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0096] The present disclosure provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed, the above-mentioned data processing method is implemented.

[0097] The present disclosure provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device executes the data processing method provided in the above-mentioned various optional implementations.

[0098] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0099] It is understandable that in the specific implementation of this application, data sharding and related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A data processing method for processing a target data slice allocated to a target server, wherein the target server comprises a plurality of nodes, each of the plurality of nodes comprises at least one processing core and corresponding memory to run a corresponding thread, the method comprising: Creating at least one thread pool, each thread pool in the at least one thread pool includes at least one thread, and each thread pool in the at least one thread pool runs on a corresponding node in the plurality of nodes; According to the load condition of each node in the multiple nodes, a target node is selected from the multiple nodes, and a corresponding thread pool in the at least one thread pool is running on the target node; as well as The target data slice is processed using a corresponding thread pool running on the target node.

2. The method according to claim 1, wherein the load condition of each of the plurality of nodes is determined according to the amount of data of other data shards different from the target data shard currently processed by each node, and wherein selecting the target node from the plurality of nodes according to the load condition of each of the plurality of nodes comprises: According to the data amount of the other data shards currently processed by each node, a node with the smallest data amount is selected from the multiple nodes as the target node.

3. The method according to claim 2, wherein the data amount of the other data shards currently processed by each node is determined according to at least one of the following: the number of other data shards currently processed by each node; and The sum of the sizes of the other data shards currently processed by each node.

4. The method according to claim 1, wherein the load condition of each of the plurality of nodes is determined according to the number of idle processing cores and the size of idle memory included in each of the nodes, and wherein selecting a target node from the plurality of nodes according to the load condition of each of the plurality of nodes comprises: Determine the product of the number of idle processing cores and the size of the idle memory included in each of the multiple nodes, and select a node with the largest product from the multiple nodes as the target node; or A weighted sum of the number of idle processing cores and the size of idle memory included in each of the multiple nodes is determined, and a node with the largest weighted sum is selected from the multiple nodes as the target node.

5. The method according to claim 1, further comprising: In response to the target data shard being deleted from the target server or being moved to another server different from the target server, updating the load status of the target node, and Load balancing is performed on the multiple nodes according to the updated load condition of the target node and the load conditions of other nodes among the multiple nodes that are different from the target node.

6. The method according to claim 5, wherein the balancing process performed on the plurality of nodes comprises: At least one data shard currently processed by a node with a heavier load among the multiple nodes is moved to a node with a lighter load among the multiple nodes.

7. The method according to claim 1, wherein the at least one thread pool corresponds one-to-one to the plurality of nodes.

8. The method according to claim 1, wherein the processing performed on the target data slice comprises at least one of the following: Writing the target data slice into the memory of the target node; Querying the target data shard from the memory of the target node; as well as At least a portion of the sub-data in the target data slice is merged.

9. A business system, comprising at least one server, wherein the at least one server comprises a plurality of nodes, each of the plurality of nodes comprises at least one processing core and a corresponding memory to run a corresponding thread, wherein: The at least one server processes the target data slice using the method described in any one of claims 1-8, and the target data slice is allocated to the at least one server by the business system.

10. A data processing device, for processing a target data slice allocated to a target server, wherein the target server comprises a plurality of nodes, each of the plurality of nodes comprises at least one processing core and a corresponding memory to run a corresponding thread, the device comprising: A thread pool creation module is configured to create at least one thread pool, each of the at least one thread pool includes at least one thread, and each of the at least one thread pool runs on a corresponding node among the plurality of nodes; A node selection module is configured to select a target node from the multiple nodes according to the load of each node in the multiple nodes, and a corresponding thread pool in the at least one thread pool is running on the target node; as well as The data processing module is configured to process the target data slice using the corresponding thread pool running on the target node.

11. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 8 when the computer executable instructions are executed by the processor. 12 . A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to claim 1 .

13. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, perform the method according to any one of claims 1 to 8.

Images