Data processing method, computer equipment, storage medium and computer program product

Abstract

The invention provides a data processing method, computer equipment, a storage medium and a computer program product. The method comprises the following steps: determining a first fragment value according to a preset initial value of a data node; the first data node writes a first fragment value into the service registry, if the first data node has written the first fragment value into the service registry, the first data node loads first data according to the first fragment value, the first data node is any one of the data nodes, and if the first data node does not write the first fragment value into the service registry, the first data node loads the second data according to the first fragment value. If the number of the fragment values stored in the service registry is smaller than the preset total number of the data nodes, the second data node writes the second fragment value into the service registry, and the second data node is the data node which does not write the first fragment value into the service registry in the data nodes, so that the delay of data loading and processing can be reduced, and the data loading efficiency is improved. And the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, a computer device, a storage medium, and a computer program product. Background Art

[0002] Existing distributed task scheduling platforms, such as those based on XXL-JOB or Elastic-Job, adopt a centralized management architecture. In these scheduling platforms, the scheduling center dynamically assigns shard values ​​to each node, and the node loads data based on the assigned shard values. However, the shard values ​​assigned to each node by the scheduling center are random, which means that each node may receive a different shard value each time. Due to the randomness of the shard value, the application service node needs to load data based on the new shard value each time, increasing the delay in data loading and processing. Summary of the invention

[0003] The embodiments of the present application disclose a data processing method, a computer device, a storage medium and a computer program product to reduce the delay of data loading and processing and improve the data processing efficiency.

[0004] In a first aspect, the present application provides a data processing method, the method comprising: determining a first shard value according to a preset initial value of a data node, the first data node writing the first shard value to a service registry, if the first data node has written the first shard value to the service registry, then the first data node loads the first data according to the first shard value, the first data node is any one of the data nodes, if the first data node has not written the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, then the second data node writes the second shard value to the service registry, the second data node is a data node among the data nodes that has not written the first shard value to the service registry.

[0005] In the embodiment of the present application, the data nodes all determine the first shard value according to the preset initial value, and write the first shard value to the service registry. The first data node has written the first shard value to the service registry, and the first data node loads the first data according to the first shard value. That is, the data node has written the first shard value to the service registry, and the data node can load the corresponding data according to the first shard value, thereby allowing the data node to use the cache mechanism to reduce the delay of data loading, thereby improving the efficiency and speed of data access.

[0006] Specifically, if the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the second data node writes the second shard value to the service registry, that is, for the data node that does not write the first shard value to the service registry, the data node can continue to determine the next shard value, such as the second shard value, the first shard value is different from the second shard value, ensuring that the shard values ​​written by each data node are different, thereby, the data nodes that have written shard values ​​to the service registry can load the corresponding data according to the shard values ​​written by each data node, that is, the shard value obtained by each data node is unique and fixed. Based on the unique and fixed shard value obtained by each data node, when a data node failure occurs, other data nodes can still load data based on the shard value they successfully competed for, reducing the impact of data node failure on other data nodes. Based on the execution of an embodiment by each data node, a data processing method is provided, each data node manages its own scheduling, and does not adopt a centralized management architecture, so the defects brought about by centralized management of sampling can be avoided.

[0007] In a second aspect, the present application provides a computer device, the computer device comprising a processor and a memory, and the processor implements any one of the above data processing methods when executing a computer program stored in the memory.

[0008] In a third aspect, the present application provides a computer-readable storage medium, which stores at least one instruction, and when the at least one instruction is executed by a processor, implements any of the above data processing methods.

[0009] In a fourth aspect, the present application provides a computer program product, which stores at least one instruction, and when the at least one instruction is executed by a processor, implements any of the above data processing methods.

[0010] The technical effects brought about by the second to third aspects can be found in the descriptions related to the methods of each design in the above method part, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a structural diagram of a distributed task scheduling platform provided in an embodiment of the present application.

[0012] Figure 2 It is a structural diagram of the data sharding scheduling system provided in an embodiment of the present application.

[0013] Figure 3 It is a flow chart of a data processing method provided in an embodiment of the present application.

[0014] FIG. 4A to FIG. 4B This is a schematic diagram of an interface for modifying the total number of preset data nodes provided in an embodiment of the present application.

[0015] Figure 5 This is a flow chart of another data processing method provided in an embodiment of the present application.

[0016] Figure 6 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0018] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0019] Data Sharding is to split a data set into multiple independent data blocks and assign each data block to a corresponding data node for storage and processing. During data sharding, the data node can determine which data blocks should be stored based on the value of the shard key. The shard key value can also be called the shard value or data shard value. A data block can also be called a shard. The shard key can be a field of a data item in a data set. The shard key can include but is not limited to: user ID, primary key, date.

[0020] See also Figure 1 , exemplarily introduces the structure of a distributed task scheduling platform 10 provided in an embodiment of the present application.

[0021] The distributed task scheduling platform 10 includes a scheduling center 11, a database 12 and at least two application service nodes 13. The scheduling center 11 is connected to the database 12 and the application service node 13 in communication.

[0022] Specifically, the administrator can configure the information of the task through the visual interface of the scheduling center 11, and configure the information persistently in the database 12. The information of the configured task may include but is not limited to: task name, description, execution time, execution logic, and sharding parameters. After the application service node 13 is started, it registers information with the scheduling center 11. The registered information may include the address of the application service node 13, etc. The scheduling center 11 maintains a service node registry and records the information of all registered service nodes. When the scheduling time of the configured task is reached, the scheduling center 11 calculates and generates a sharding value based on the information of the task, such as sharding parameters. The sharding parameters may include but are not limited to: the total number of shards, sharding strategy, etc. The scheduling center 11 dynamically allocates the calculated sharding value to the registered application service node through a preset allocation strategy (such as a random allocation strategy, etc.). After the registered application service node receives the scheduling request and the allocated sharding value from the scheduling center 11, it loads the corresponding data according to the sharding value. After the application service node 13 loads the data, it executes the task and returns the execution result and log information to the scheduling center 11 after the task is executed. The dispatch center 11 saves the execution results and log information to the database 12 for subsequent query and analysis.

[0023] The inventors found that the above-mentioned distributed task scheduling platform has at least the following defects when implementing the embodiments of the present application:

[0024] (1) When executing task scheduling, the dispatch center dynamically calculates the shard value and randomly assigns the shard value to each application service node. Based on the random assignment of shard values, the shard value obtained by each application service node may be different from the previous one. Because the obtained shard value is different from the previous one, the data previously stored by the application service node based on the old shard value becomes invalid, and the application service node needs to reload the corresponding data from the database based on the newly obtained shard value. However, loading data from the database, especially when processing large amounts of data or when the database performance is limited, will significantly increase the delay of data loading.

[0025] (2) In the event of a failure of a single application service node, the tasks being processed on the application service node may face the risk of interruption or delay. In order to maintain the normal operation of the system, the scheduling center generally needs to reallocate the shard values ​​to reallocate the shard values ​​originally allocated to the failed application service node to other application service nodes that are operating normally. However, the scheduling center reallocates the shard values, resulting in the shard values ​​obtained by the fault-free application service nodes being different from the previous ones. Because the obtained shard values ​​are different from the previous ones, the data previously stored by the fault-free application service nodes based on the old shard values ​​become invalid, and the fault-free application service nodes also need to reload the corresponding data from the database based on the newly obtained shard values.

[0026] (3) Using a centralized management architecture to deploy the scheduling platform requires the configuration of additional server resources, usually including application servers and database servers. The application server is used to deploy and run the central scheduling management application of the scheduling platform.

[0027] (4) The scheduling center is prone to single-node failures because most centralized scheduling platforms have not yet implemented effective cluster deployment. In addition, if the network between the scheduling center and the task execution node fails, it will also cause task execution to be interrupted or fail.

[0028] (5) Applications accessing the dispatch center are subject to certain restrictions in terms of development language, because such applications accessing the dispatch center usually require integration of the Agent dependency package provided by the dispatch center, a process that is invasive to the application code.

[0029] In view of this, an embodiment of the present application provides a data processing method, which ensures that each written shard value in the service registry corresponds to only one data node by setting the written shard value to be unique in the service registry. The data nodes all determine the first shard value according to the preset initial value and write the first shard value to the service registry. Based on the uniqueness of the written shard value in the service registry, when multiple data nodes write the first shard value to the service registry, there is a data node (such as the first data node) that has written the first shard value to the service registry among the multiple data nodes, and the other remaining data nodes (such as the second data node) that have not written the first shard value to the service registry. In the case where the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the other remaining data nodes that have not written the first shard value to the service registry can write a shard value different from the first shard value (such as the second shard value) to the service registry.

[0030] Based on the setting that the written shard value is unique in the service registry, it is ensured that each existing shard value in the service registry strictly corresponds to and only corresponds to one data node, thereby firmly establishing the binding relationship between the data node and the shard value it has written to the service registry. Therefore, if a data node has written a specific shard value to the service registry, the data node can continuously and stably hold the shard value it has written to the service registry.

[0031] Since the data node can continuously and stably hold the shard value written to the service registry, the data node can load the corresponding data according to the shard value, ensuring that the same data is always assigned to the same data node for processing. Since the same data is always assigned to the same data node, the data node can store the data, and then directly read the data from the storage in subsequent accesses without reloading the data, which greatly reduces the delay of data loading and improves the efficiency and speed of data access, thereby avoiding the above-mentioned defect (1).

[0032] When other data nodes fail, based on the fact that the data node can continuously and stably hold the shard value it has written to the service registry, the data node can still load the corresponding data according to the shard value it has written to the service registry, thereby reducing the impact of the failed data node, thereby avoiding the above-mentioned defect (2).

[0033] By executing a data processing method provided in the embodiment through each data node, decentralized management is achieved, which reduces the dependence on the central scheduling management platform, reduces hardware, network and maintenance costs, and avoids the above-mentioned defect (3). Based on decentralized management, the situation of single node failure in the scheduling center is avoided, and the above-mentioned defect (4) is avoided. Based on decentralized management, data nodes do not need to integrate dependency packages Agent, so there is no limitation on application development language, avoiding the above-mentioned defect (5).

[0034] Furthermore, if a data node fails and goes offline, the shard value that the data node has written to the service registry will be cleared by other data nodes, that is, the shard value originally held by the offline data node does not exist in the service registry. When the offline data node recovers from the failure and comes back online, such as restarting after a shutdown, the data node can execute the data processing method provided in the embodiment of the present application to write the shard value to the service registry. Based on the fact that the shard value originally held by the data node does not exist in the service registry, and the written shard value is unique in the service registry, the data node can successfully write the shard value it originally held to the service registry, that is, the data node has written the shard value it originally held to the service registry, and thus, the data node can still obtain the shard value it originally held after the failure is recovered. Based on the failure recovery, the data node can still obtain the shard value it originally held, ensuring that the same data is always assigned to the same data node for processing, avoiding the loss or confusion of shard values ​​caused by data node failures, thereby maintaining the consistency and integrity of the data.

[0035] Further, in the case of expanding data nodes, the newly added data nodes execute the data processing method provided in the embodiment of the present application, and the shard value that has been written is set to be unique in the service registry, so that the newly added data node cannot write the shard value of the existing service registry, and a new shard value can be written to the service registry. Based on the fact that the newly added data node cannot write the shard value of the existing service registry, other data nodes can continue to load and process data. The newly added data node can also quickly obtain data, ensuring the scalability and flexibility of the system.

[0036] The following is an illustrative introduction to the application scenarios of the data processing method provided in the embodiments of the present application.

[0037] Cloud monitoring alarm rule scheduling and processing scenarios

[0038] In a cloud service environment, tenants can purchase and use cloud resources, such as MySQL clusters, Redis clusters, and ECS hosts. For each cloud resource, tenants can set cloud monitoring alarm rules, which can include cloud monitoring alarm rules for the entire cluster and / or a single node. Cloud monitoring alarm rules are a set of indicator monitoring thresholds set for cloud resource indicator monitoring. Cloud monitoring alarm rules define thresholds for various states of cloud resources. When the indicators detected in real time exceed these thresholds, alarm notifications are triggered.

[0039] As the number of tenants and the types of cloud resources increase, the data set storing cloud monitoring alarm rules becomes huge. In order to monitor the status of cloud resources in real time according to the cloud monitoring alarm rules and send alarms in time, these cloud monitoring alarm rules need to be processed efficiently. If only one data node is used to process such a large data set, the query and matching process may take too long and the alarm may not be issued in time. Therefore, it is necessary to distribute the cloud monitoring alarm rules to multiple data nodes, that is, to shard the data set storing the cloud monitoring alarm rules, distribute each cloud monitoring alarm rule to each data node, and each data node performs matching processing according to the distributed cloud monitoring alarm rules. However, even if each cloud monitoring alarm rule is distributed to each data node for processing, it is still time-consuming for each data node to load the corresponding cloud monitoring alarm rule each time it matches.

[0040] In view of this, each data node executes the data processing method provided in the embodiment of the present application to write the shard value to the service registry, and the data node obtains and stores the data it is responsible for, such as the cloud monitoring alarm rule, according to the shard value written to the service registry. Therefore, when matching real-time indicators, the data node can directly use the locally stored cloud monitoring alarm rules without frequently loading the cloud monitoring alarm rules from the database.

[0041] The frequency of changes (such as addition, modification or deletion) of the established cloud monitoring alarm rules is low, that is, the update frequency of cloud monitoring alarm rules is low, and the frequency of data nodes updating the cloud monitoring alarm rules they store is also low. When the cloud monitoring alarm rules change, based on the uniqueness of the written shard value in the service registry, the data node can continuously and stably hold the shard value it has written to the service registry. Therefore, the data node can update the corresponding cloud monitoring alarm rule according to the shard value it has written to the service registry, ensuring that the same cloud monitoring alarm rule is always assigned to the same data node for processing, thereby realizing the dynamic update of the cloud monitoring alarm rule.

[0042] High-speed toll collection scenario

[0043] For example, in the transportation network from A to B, there are multiple routes for vehicles to choose from, and the toll for each route varies depending on factors such as the length of the road section, construction standards, geographical location, and age. A car can choose any route from A to B, and needs to quickly calculate the fee when getting off the highway at B.

[0044] In view of this, each data node executes the data processing method provided in the embodiment of the present application to write the shard value to the service registry. The data node obtains and stores the data it is responsible for, such as the road section charging standard, according to the shard value it has written to the service registry. Therefore, when on a certain underground highway, the data node can directly use the locally stored road section charging standard without frequently loading the road section charging standard from the database. For example, a data node caches the road section charging standard of all sections from A to B. The data node only needs to match the corresponding road section charging standard according to the actual driving route of the vehicle to quickly calculate the fee.

[0045] Since the established road section charging standards change less frequently (such as addition, modification or deletion), the road section charging standards are updated less frequently, and the data nodes update the road section charging standards they store less frequently. When the road section charging standards change, based on the uniqueness of the written shard value in the service registry, the data node can continuously and stably hold the shard value it has written to the service registry. Therefore, the data node can update the corresponding road section charging standards according to the shard value it has written to the service registry, ensuring that the same road section charging standards are always assigned to the same data node for processing, thereby realizing the dynamic update of the road section charging standards.

[0046] In an embodiment of the present application, by setting the written shard value to be unique in the service registry, it is ensured that each written shard value in the service registry corresponds to only one data node, which can well support the scenario where the data node stores data relatively fixedly, and can also well support the scenario where the data node stores data relatively fixedly and the stored data does not change, or the update frequency is very low.

[0047] It can be understood that the data processing method, computer device, storage medium and computer program product provided in the embodiments of the present application can be applicable to scenarios where data sharding is required, and are not limited to the above-mentioned scenarios. For example, the data processing method provided in the embodiments of the present application can quickly restore its data after a data node crashes, thereby maintaining data consistency and system stability. This is particularly important for application scenarios that require high availability and fault tolerance (such as finance, e-commerce, etc.). Therefore, application scenarios that require high availability and fault tolerance can also be applied. The present application does not specifically limit the application scenarios.

[0048] In order to better understand a data processing method, computer device, storage medium and computer program product provided in an embodiment of the present application, the data processing system provided in an embodiment of the present application is first described below.

[0049] See also Figure 2 , the data processing system 100 includes a concurrency control server 101 and at least two data nodes 102. The concurrency control server 101 and the data nodes 102 are connected via a communication network.

[0050] Among them, the communication network may include but is not limited to: the Internet, local area network (LAN), wireless local area network (WLAN), wide area network (WAN), personal area network (PAN), third generation mobile communication technology (3rd Generation, 3G), fourth generation mobile communication technology (4th Generation, 4G), long term evolution technology (Long Term Evolution, LTE), and wireless fidelity technology (Wi-Fi).

[0051] The concurrency control server 101 is used to provide a service registry. The service registry is used to store the registration information of the data node. The registration information may include, but is not limited to: the shard value, the unique identifier of the data node, the time when the data node is successfully registered, and the time when the successfully registered data node last updated the heartbeat. The service registry can be updated in real time. For example, when the data node 102 goes offline or fails, the concurrency control server 101 or other data nodes 102 can remove it from the service registry in a timely manner.

[0052] The concurrency control server 101 is used to provide concurrency control services. Concurrency control services refer to services that perform access control and management on shared resources in a multi-user, multi-process or multi-threaded environment in order to ensure data consistency and integrity. The concurrency control server 101 ensures that the written shard value is unique in the service registry through the concurrency control service, and further ensures that each written shard value in the service registry corresponds to only one data node. That is, when multiple data nodes 102 write the same shard value to the service registry, only one data node 102 writes successfully.

[0053] In the embodiment of the present application, the concurrency control server 101 implements the concurrency control service in the following ways, including but not limited to:

[0054] Method 1: unique index. A unique index is an index used in a database to ensure that the values ​​of a column or columns are unique. In the service registry, the unique index feature of the database is used to ensure that each shard value is unique in the service registry, thereby avoiding data duplication and inconsistency.

[0055] Method 2: Mutex. Mutex is a synchronization primitive used in multithreaded programming to ensure mutual exclusion when multiple threads access shared resources. In a multithreaded environment, a mutex is used to synchronize access to shared resources. Mutex ensures that only one thread can access a specific shared resource at any time, effectively preventing data contention and inconsistency.

[0056] Mode 3, distributed lock. Distributed lock is a mechanism for coordinating access to shared resources between multiple processes or threads in a distributed system. In a distributed system, in order to coordinate the access of multiple processes or threads to shared resources, the concurrency control server 101 implements a distributed lock mechanism. Through the operation of locking and releasing the lock, it ensures exclusive access to shared resources in a distributed environment and maintains the consistency and integrity of the data.

[0057] It can be understood that, while ensuring that each written shard value in the service registry corresponds to only one data node, the specific implementation method of the concurrency control server 101 to implement the concurrency control service can be selected according to actual conditions, and this application does not make specific limitations on this.

[0058] The concurrency control server 101 may include but is not limited to: a relational database, a remote dictionary server (RemoteDictionary Server, Redis) and Zookeeper. The relational database includes but is not limited to: MySQL (MyStructured Query Language).

[0059] It is understandable that the electronic device used as the concurrent control server 101 can be selected according to actual conditions, and this application does not make any specific limitation on this.

[0060] The data node 102 is used to execute the data processing method provided in the embodiment of the present application. The data node 102 writes registration information to the service registration table. The written registration information may include, but is not limited to: the shard value, the unique identifier of the data node, the time when the data node is successfully registered, and the time when the heartbeat of the successfully registered data node is last updated.

[0061] The data node 102 obtains and stores the corresponding data according to the shard value written to the service registry. The data node 102 can also update the stored data, including but not limited to: modifying, deleting, disabling, adding, etc.

[0062] In the embodiment of the present application, the data node 102 has written a shard value to the service registry, and the data node can continuously and relatively stably hold the shard value written to the service registry. The shard value held by the data node 102 is relatively fixed. Relatively fixed means that except for application upgrades or other situations where the shard value needs to be reallocated, the data node 102 has written a specific shard value to the service registry, and the data node can continuously and stably hold the shard value written to the service registry.

[0063] The data node 102 can execute the corresponding scheduling task based on the stored data, and the scheduling task may include but is not limited to: a periodic task or a one-time task that is executed at a fixed time. The data loaded and cached by the data node 102 can serve the execution of other scheduling tasks. For example, in the above-mentioned cloud monitoring alarm rule scheduling processing scenario, the monitoring alarm will have a periodic timing task. After the data node 102 loads and caches the corresponding cloud monitoring alarm rule based on the shard value, it periodically matches the real-time indicator based on the cached cloud monitoring alarm rule, and triggers the alarm notification when the real-time indicator meets the threshold.

[0064] In the embodiment of the present application, each data node 102 has a data processing implementation that is consistent with all other data nodes 102, that is, a data processing device 200. Each data node 102 includes a data processing device 200, and the data processing method provided by the embodiment of the present application can be executed by the data processing device 200, so that each data node 102 can independently manage its own data shard scheduling, thereby realizing a decentralized management architecture.

[0065] In some embodiments, the data processing device 200 can serve as a module of a target application so that when the target application is running, the data processing method provided in the embodiments of the present application can be executed.

[0066] Among them, the data processing device 200 can implement the corresponding logic through the development language of its own application, thereby achieving decentralized management scheduling.

[0067] The data node 102 may include, but is not limited to, the following computer devices: servers, workstations, laptops, desktop computers, storage devices (such as dedicated storage device arrays or network attached storage (NAS) devices), nodes in a cloud computing environment (such as virtual machines (VMs) or containers), embedded systems (such as Internet of Things (IoT) devices), and high-performance computing (HPC) clusters.

[0068] It is understood that the data node 102 can be configured according to actual conditions, and this application does not make specific limitations on this. For example, when constructing and managing the data node 102, multiple factors can be considered, including but not limited to: device performance (such as CPU, memory, disk speed), network bandwidth, security, reliability and scalability.

[0069] The concurrency control server 101 can also be used to store data, such as data to be sharded. The data node 102 can load corresponding data from the concurrency control server 101 according to the shard value written to the service registration table.

[0070] In some other embodiments, the data processing system 100 may further include a node for storing data to be sharded, such as a database. The data node 102 may load corresponding data from the database according to the shard value written to the service registry.

[0071] Understandably, Figure 2The concurrency control server 101, the communication network and the data node 102 are merely illustrative. Depending on the implementation requirements, there may be any number of concurrency control servers 101, communication networks and data nodes 102. Exemplarily, the current data processing system 100 includes three data nodes A, B and C. Based on the increase in processing pressure, the data node D is connected to the cluster of data nodes A, B and C through the communication network. The data node D can also execute the data processing method provided in the embodiment of the present application. The data processing system 100 includes data nodes A, B, C and D.

[0072] Exemplarily, the concurrency control server 101 or the data node 102 may be an independent physical server, or a server cluster composed of multiple servers.

[0073] The following uses the concurrency control server 101 as the relational database MySQL to exemplarily introduce how to obtain the shard value based on MySQL.

[0074] Exemplarily, a service registration and data sharding table, i.e., a service registry, is created in MySQL. The service registry may be named shard_table. The service registry is used to manage the registration information of the successfully registered data nodes 102, including the sharding values ​​written by the successfully registered data nodes 102. The service registry ensures that each written sharding value is unique in the service registry, i.e., each sharding value corresponds to only one data node 102.

[0075] Exemplarily, the fields of the service registry may include, but are not limited to:

[0076] Primary key field (such as id): used to record the primary key, as the unique identifier of the table, self-increasing, and may not have any business attributes.

[0077] Data node field (eg, server_ip): used to store the unique identifier of the successfully registered data node 102. The unique identifier of the data node 102 may be a communication address, such as an IP address, to facilitate tracking and management of the shard values ​​written by the data node 102.

[0078] Shard value field (e.g., shard_num): used to store the shard value written by the data node 102. This field has a unique index to ensure that each shard value appears only once in the service registry.

[0079] Update time field (eg update_time): used to record the time when the successfully registered data node 102 last updated its heartbeat, and used to monitor the survival status of the data node 102 .

[0080] For ease of understanding, the time stored in the update time field (ie, the time when the data node 102 last updated the heartbeat) is named the heartbeat update time.

[0081] The survival status of the data node 102 can be determined by a heartbeat mechanism, for example, the heartbeat information is updated every 20 seconds.

[0082] Total data amount field (eg data_num): used to record the total amount of data (which may be the amount of data or the number of data items) processed by the successfully registered data node 102, for performance analysis and statistics.

[0083] Registration time field (e.g., create_time): used to record the successful registration time of the successfully registered data node 102. The successful registration of the data node 102 means that the data node 102 successfully writes the registration information (including the shard value) to the service registration table. The registration time field records the time when the data node 10 successfully registers and writes the shard value.

[0084] For example, the field data type of the primary key field (id) is INT NOT NULL. The field data type of the data node field (server_ip) is VARCHAR(20) NOT NULL. The field data type of the shard value field (shard_num) is INT NOT NULL and is set as a unique index (UNIQUE INDEX). The field data type of the update time field (update_time) is DATETIME NOT NULL. The field data type of the total number of data fields (data_num) is BIGINT NULL. The field data type of the registration time field (create_time) is DATETIME NOT NULL.

[0085] Each data node 102 executes the data processing method provided in the embodiment of the present application, and each data node 102 writes the registration information of the data node 102 to the service registry of MySQL, such as the information of the data node 102 corresponding to the above fields. In a concurrent environment, multiple data nodes 10 may try to register the same shard value at the same time. Since the shard_num field of shard_table is set as a unique index, MySQL will use this mechanism to ensure the uniqueness of the shard value.

[0086] Specifically, when the data node 102 attempts to write a shard value, MySQL checks whether the shard value already exists in the shard_num field.

[0087] If the shard value does not exist, MySQL allows a data node 102 to write the shard value. When the data node 102 successfully writes the shard value in the shard value field (shard_num), the data node 102 can record the registration information corresponding to the data node 102 in the primary key field, data node field, update time field, total data field and registration time field. That is, MySQL allows the data node 102 to write the shard value, and creates a new record in shard_table, including: the primary key, the unique identifier of the data node 102, the shard value written by the data node 102, the time when the data node 102 last updated the heartbeat (i.e., the heartbeat update time), the total amount of data processed by the data node 102, and the successful registration time of the data node 102, completing the registration process.

[0088] If the shard value already exists, the unique index of MySQL will block the write operation of the data node 102 and return an error response to other data nodes 102 that attempt to write the same shard value, informing them that the shard value has been occupied by other nodes 102 and cannot be registered.

[0089] In this way, the MySQL service registry not only records the registration information of all successfully registered data nodes 102, including their shard values, survival status, etc., but also ensures the uniqueness of the shard values ​​through a unique index mechanism, thereby effectively avoiding data conflicts and duplications in a high-concurrency environment.

[0090] It can be understood that the above-mentioned MySQL-based method mainly utilizes the mutual exclusion lock feature of the MySQL unique index in concurrent situations. For example, when competing for the same shard value in concurrent situations, it can be guaranteed that only one data node succeeds, and the same shard value will not be obtained by multiple competing data nodes at the same time.

[0091] It can be understood that when obtaining the shard value, it is not limited to being implemented based on MySQL. The unique index of other relational database tables can also be used to achieve competition and allocation of shard values.

[0092] Taking the concurrency control server 101 as Redis or Zookeeper, the acquisition of shard values ​​by using a Redis-based distributed lock or a Zookeeper-based distributed lock is introduced as an example.

[0093] Redis or Zookeeper maintains a service registry. For more information about the service registry, refer to the above.

[0094] The process of obtaining shard values ​​based on distributed locks can include the following:

[0095] (1) Obtaining lock authority: Data node 102 obtains lock authority based on Redis or Zookeeper.

[0096] All data nodes 102 go to Redis or Zookeeper to obtain the same lock, which ensures that only one data node 102 can execute the logic of obtaining the shard value at the same time.

[0097] (2) Waiting for lock: The data node 102 that has not obtained the lock authority will enter the listening lock state, waiting for the data node 102 that currently obtains the lock authority to release the lock authority.

[0098] In some embodiments, a waiting times threshold may be set. If the waiting times of the data node 102 that has not obtained the lock authority exceed the threshold and still has not obtained the lock authority, the process is exited and no longer competes for the shard value.

[0099] (3) Verify the shard value: The data node 102 that successfully obtains the lock permission needs to verify in the service registry whether the current shard value has been occupied by others.

[0100] Exemplarily, the data node 102 that successfully obtains the lock permission determines the first shard value according to a preset initial value (such as 0), and verifies in the service registry whether the first shard value is occupied.

[0101] (4) Allocating a shard value: If the current shard value is not occupied, the data node 102 will mark it as occupied as its own shard value and continue the subsequent data processing process, such as releasing lock permissions, loading data according to the obtained shard value, etc. If the current shard value is occupied, the data node 102 will try the next shard value until an unoccupied shard value is found or the upper limit of the total number of shards is reached.

[0102] Exemplarily, the data node 102 that successfully obtains the lock uses the preset initial value 0 as the first shard value. The data node 102 that successfully obtains the lock verifies in the service registry whether the first shard value 0 is occupied. If it is not occupied, the first shard value 0 is obtained, otherwise the second shard value is determined. For example, by sequentially adding 1 to the first shard value 0, the second shard value 1 is obtained. Continue to verify whether the second shard value 1 is occupied. If it is not occupied, the second shard value 1 is obtained, otherwise the third shard value is determined. For example, by sequentially adding 1 to the second shard value 1, the third shard value 2 is obtained, and so on, until an unoccupied shard value is found or the upper limit of the total number of shards is reached. Exemplarily, the upper limit of the total number of shards can be the preset total number of data nodes (total).

[0103] (5) Handling failure: If the data node 102 that successfully obtains the lock authority still cannot find an unoccupied shard value after reaching the upper limit of the total number of shards, it will release the lock authority and retry the entire process (such as starting from obtaining the lock authority from (1) above), or choose to exit the process.

[0104] As in the above example, the current shard value is greater than or equal to the total number of server nodes, reaching the upper limit of the total number of shards. Data node 102 does not find an unoccupied shard value, releases the lock authority and retries the entire process (such as starting from obtaining the lock authority from (1) above), or chooses to exit the process.

[0105] (6) Release lock: Data node 10 that successfully obtains the lock authority releases the lock authority.

[0106] Regardless of whether the shard value allocation is successful, the data node 102 that successfully obtains the lock authority will release the lock after completing the corresponding operation so that other data nodes can continue to compete. For example, the lock authority is released when an unoccupied shard value is found or the upper limit of the total number of shards is reached.

[0107] In the above manner, Redis or Zookeeper can effectively ensure that in a concurrent environment, only one data node 102 can successfully compete for the same shard value, thereby avoiding data conflicts and inconsistencies. At the same time, through the heartbeat mechanism and service registry, Redis or Zookeeper can also handle offline or faulty data nodes 102 in a timely manner to maintain the accuracy and consistency of the service registry.

[0108] It can be understood that the method of obtaining the above-mentioned sharding value is not limited to MySQL, relational database, Redis or Zookeeper, and this application does not make any specific limitation on this.

[0109] The specific embodiments described below may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0110] See also Figure 3 , exemplarily introduces the data processing method provided in the embodiment of the present application. The data processing method is applied to a data node and can be executed by a data node or a data processing device.

[0111] Step S301, determining a first shard value according to a preset initial value of a data node.

[0112] In the embodiment of the present application, all data nodes executing the data processing method provided in the embodiment of the present application store the same preset initial value. Before all data nodes write the shard value to the service registry, the first shard value is determined according to the same preset initial value, thereby ensuring that the first shard value determined by all data nodes is the same.

[0113] The preset initial value may be 0 or other values, such as 1, and this application does not make any specific limitation on this.

[0114] Exemplarily, all data nodes determine the first shard value according to the preset initial value of the data node, including: all data nodes use the preset initial value as the first shard value, and then in subsequent steps all data nodes write the preset initial value to the service registry.

[0115] Step S302: A first data node writes a first shard value to a service registration table, where the first data node is any one of the data nodes.

[0116] In the embodiment of the present application, all data nodes write the first shard value to the service registry, that is, any one of all data nodes performs the first write operation to write the first shard value to the service registry. Based on the first shard value determined in step S301 being the same, the first shard value written to the service registry by all data nodes in step S302 is the same.

[0117] For ease of understanding, the operation of writing the first shard value to the service registry is named the first write operation.

[0118] In the embodiment of the present application, the data node that executes the data processing method provided in the embodiment of the present application includes at least one.

[0119] When there is only one data node executing the data processing method provided in the embodiment of the present application, the only data node is the first data node. When there are two or more data nodes executing the data processing method provided in the embodiment of the present application, any one of the two or more data nodes is the first data node.

[0120] Step S303: If the first data node has written the first shard value into the service registration table, the first data node loads the first data according to the first shard value.

[0121] In an embodiment of the present application, any one of all the data nodes performs a write operation to write the same shard value (such as a first shard value or a second shard value) to the service registry, and only one data node successfully performs the write operation. After performing the write operation, only one data node has written the shard value to the service registry.

[0122] Exemplarily, any data node among all data nodes executes the first write operation to write the first shard value to the service registry, and only one data node successfully executes the first write operation, that is, only one data node has written the first shard value to the service registry, and the data node that has written the first shard value to the service registry can be any data node among all data nodes.

[0123] In an embodiment of the present application, if the first data node has written the first shard value to the service registration table, the first data node can continuously and stably hold the first shard value, and the first data node loads the first data according to the first shard value to ensure that the first data is always assigned to the first data node.

[0124] Step S304: If the first data node has not written the first shard value to the service registration table, and the number of shard values ​​stored in the service registration table is less than the total number of preset data nodes, the second data node writes the second shard value to the service registration table. The second data node is the data node among the data nodes that has not written the first shard value to the service registration table.

[0125] In an embodiment of the present application, any one of all the data nodes may fail to successfully execute the first write operation to write the first shard value to the service registry. When all the data nodes include two or more, based on the fact that only one data node successfully executes the first write operation to write the first shard value to the service registry, other data nodes except the data node that successfully executes the first write operation fail to successfully execute the first write operation, that is, other data nodes fail to write the first shard value to the service registry.

[0126] For ease of understanding, all data nodes except the data node that successfully performs the first write operation are named as second data nodes. That is, after the first data node performs step S302, if the first data node does not write the first shard value to the service registration table, the first data node is named as the second data node.

[0127] In an embodiment of the present application, if any data node fails to successfully perform a write operation to write a shard value to a service registry, the data node can determine the next shard value to be written to the service registry. In other embodiments, if any data node fails to successfully perform a write operation to write a shard value to a service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the data node can determine the next shard value to be written to the service registry. The shard value that the data node fails to write successfully is different from the next shard value determined to be written to the service registry. Determining the next shard value to be written to the service registry will be described in detail below.

[0128] Specifically, after the first data node executes step S302, if the first data node (i.e., the second data node) has not written the first shard value to the service registration table, and the number of shard values ​​stored in the service registration table is less than the total number of preset data nodes, the second data node determines the next shard value to be written to the service registration table, such as the second shard value, and writes the second shard value to the service registration table. The first shard value is different from the second shard value.

[0129] Exemplarily, the first fragmentation value is 0 and the second fragmentation value is 1.

[0130] In the embodiment of the present application, all data nodes executing the data processing method provided in the embodiment of the present application store the same preset total number of data nodes. The preset total number of data nodes is a value set by the user. The setting of the preset total number of data nodes will be described in detail below.

[0131] In an embodiment of the present application, the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, that is, when the first data node has not written the first shard value to the service registry, the total number of shard values ​​written in the service registry is less than the total number of preset data nodes.

[0132] In some embodiments, step S304 may specifically include:

[0133] Step 1: If the first data node does not write the first shard value to the service registration table, determine the second shard value according to the second data node.

[0134] Step 2: If the second shard value is less than the total number of preset data nodes, the second data node writes the second shard value to the service registration table.

[0135] In an embodiment of the present application, if the first data node does not write the first shard value to the service registry, the first data node that does not write the first shard value to the service registry is the second data node, and the second data node can determine the next shard value to be written to the service registry based on the unwritten first shard value, that is, the second shard value. The second shard value determined by the second data node is less than the total number of preset data nodes, which indicates that the number of shard values ​​stored in the service registry at this time is less than the total number of preset data nodes.

[0136] Specifically, after the first data node executes step S302, if the first data node (i.e., the second data node) does not write the first shard value to the service registry, the second data node determines the next shard value to be written to the service registry, such as the second shard value. If the second data node determines that the second shard value is less than the total number of preset data nodes, the second data node writes the second shard value to the service registry. If the second data node determines that the second shard value is greater than or equal to the total number of preset data nodes, the second data node suspends writing shard values ​​to the service registry and records the suspension duration.

[0137] In some embodiments, the first data node writes the first shard value to the service registry. If the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the second data node writes the second shard value to the service registry until the number of shard values ​​stored in the service registry is greater than or equal to the total number of preset data nodes. The second data node is a data node among the data nodes that does not write the first shard value to the service registry.

[0138] Specifically, if data node A has not written the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, then data node B writes the second shard value to the service registry, and data node B is the data node that has not written the first shard value. If data node B has written the second shard value to the service registry, data node B loads the second data according to the second shard value. If data node B has not written the second shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, then data node C writes the third shard value to the service registry, and data node C is the data node that has not written the second shard value, and this cycle is repeated until the number of shard values ​​stored in the service registry is greater than or equal to the total number of preset data nodes.

[0139] In the above step S304, the second data node writes the second shard value to the service registration table under the following circumstances:

[0140] Scenario 1: The second data node has written the second shard value to the service registry, and based on the second data node having written the second shard value to the service registry, the number of shard values ​​stored in the service registry is equal to the total number of preset data nodes.

[0141] In scenario 1, for the data node that has written the second shard value to the service registry, the data node loads the second data according to the second shard value.

[0142] In scenario 1, if there is a data node that has not written the second shard value to the service registry, based on the fact that the number of shard values ​​stored in the service registry is equal to the total number of preset data nodes, the data node can suspend writing shard values ​​to the service registry and record the suspension duration.

[0143] Scenario 2: The second data node has written the second shard value to the service registry, and based on the second data node having written the second shard value to the service registry, the number of shard values ​​stored in the service registry is less than the total number of preset data nodes.

[0144] In scenario 2, for the data node that has written the second shard value to the service registry, the data node loads the second data according to the second shard value.

[0145] In scenario 2, for the data node that has not written the second shard value to the service registry, based on the fact that the number of shard values ​​stored in the service registry is less than the preset total number of data nodes, the data node determines the next shard value to be written to the service registry, such as the third shard value, and writes the third shard value to the service registry.

[0146] In an embodiment of the present application, if the second data node has not written the second shard value to the service registry, and the number of first shard values ​​stored in the service registry is less than the total number of preset data nodes, the steps of determining the third shard value according to the third data node and writing the third shard value to the service registry are executed until the number of shard values ​​stored in the service registry is equal to the total number of preset data nodes, and the third data node is the data node among the data nodes that has not written the shard value to the service registry.

[0147] It can be understood that the third data node is a data node among the data nodes that has not written a shard value to the service registry, and may include the third data node, the fourth data node, etc. Correspondingly, the third shard value may include the third shard value and the fourth shard value.

[0148] It can be understood that if the second data node does not write the second shard value to the service registry, the service registry stores the first shard value, that is, the number of first shard values ​​stored in the service registry is 1.

[0149] Exemplarily, the total number of preset data nodes is 4, and all data nodes executing the data processing method provided in the embodiment of the present application include nodes 1 to 6 as an example.

[0150] All data nodes determine the first shard value (i.e., the first shard value) according to the preset initial value, and write the first shard value to the service registry. Among nodes 1 to 6, only node 1 has written the first shard value to the service registry. Node 1 loads the first data according to the first shard value.

[0151] Based on the fact that node 1 has written the first shard value to the service registry, the number of shard values ​​stored in the service registry is 1. Based on the fact that the number of shard values ​​stored in the service registry is 1 and is less than the total number of preset data nodes 4, the remaining nodes 2 to 6 that have not written the first shard value to the service registry determine the second shard value (i.e., the second shard value) and write the second shard value to the service registry.

[0152] Among nodes 2 to 6, only node 2 has written the second shard value to the service registry. Node 2 loads the second data according to the second shard value. Based on the fact that node 2 has written the second shard value to the service registry, the number of shard values ​​stored in the service registry is 2. Based on the fact that the number of shard values ​​stored in the service registry is 2 and is less than the total number of preset data nodes 4, the remaining nodes 3 to 6 that have not written the second shard value to the service registry determine the third shard value and write the third shard value to the service registry.

[0153] Among nodes 3 to 6, only node 3 has written the third shard value to the service registry. Node 3 loads the third data according to the third shard value. Based on the fact that node 3 has written the third shard value to the service registry, the number of shard values ​​stored in the service registry is 3. Based on the fact that the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, 4, the remaining nodes 4 to 6 that have not written the third shard value to the service registry determine the fourth shard value and write the fourth shard value to the service registry.

[0154] Among nodes 4 to 6, only node 4 has written the fourth shard value to the service registry. Node 4 loads the fourth data according to the fourth shard value. Based on the fact that node 4 has written the fourth shard value to the service registry, the number of shard values ​​stored in the service registry is 4. Based on the fact that the number of shard values ​​stored in the service registry is 4, which is equal to the total number of preset data nodes 4, the remaining nodes 5 to 6 that have not written the fourth shard value to the service registry suspend writing shard values ​​to the service registry and record the suspension duration.

[0155] If the second data node does not write the second shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the third data node executes the steps of determining the third shard value and writing the third shard value to the service registry. The third data node may include any one of the nodes 3 to 6 in the above example, and the third shard value may include the third shard value, that is, nodes 3 to 6 all execute the steps of determining the third shard value and writing the third shard value to the service registry. When nodes 4 to 6 in the above example do not write the third shard value to the service registry, the third data node may also include any one of the nodes 4 to 6 in the above example, and the third shard value may also include the fourth shard value, that is, nodes 4 to 6 all execute the steps of determining the fourth shard value and writing the fourth shard value to the service registry, and so on, until the number of shard values ​​stored in the service registry is equal to the total number of preset data nodes.

[0156] In some embodiments, if the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is greater than or equal to the total number of preset data nodes, the second data node pauses writing the first shard value to the service registry and records the pause duration. When the pause duration reaches the preset duration, the second data node writes the first shard value to the service registry.

[0157] In the case where the pause duration reaches the preset duration, the second data node writes the first shard value to the service registration table, which may include the following situations:

[0158] Scenario 1: If the second data node has written the first shard value to the service registration table, the second data node loads the first data according to the first shard value.

[0159] Scenario 2: If the second data node does not write the first shard value to the service registry, the second data node executes the step of determining the second shard value until the determined shard value is greater than or equal to the total number of preset data nodes. If the determined shard value is less than the total number of preset data nodes, the determined shard value is written to the service registry.

[0160] In some embodiments, the second data node suspends writing the first shard value to the service registry and records the duration of the suspension, which may include: the second data node suspends writing the first shard value to the service registry and records the duration of the suspension and the number of suspensions. In the case where the suspension duration reaches a preset duration, the second data node writes the first shard value to the service registry, which may include: if the number of suspensions is greater than or equal to the preset number of times, the second data node stops writing the first shard value to the service registry. If the number of suspensions is less than the preset number of times, the data of the fourth data node in the service registry is deleted, and / or the second data node writes the first shard value to the service registry, and the fourth data node is the data node to be deleted.

[0161] Exemplarily, the fourth data node may be an offline or faulty data node.

[0162] Exemplarily, if the first data node has not written the first shard value to the service registry, and the number of shard values ​​stored in the service registry is greater than or equal to the total number of preset data nodes, the second data node suspends writing the shard value to the service registry, and records the duration and number of pauses. When the duration of the pause reaches the preset duration, the second data node writes the first shard value to the service registry. If the second data node has written the first shard value to the service registry, the second data node loads the first data according to the first shard value. If the second data node has not written the first shard value to the service registry, the second data node determines the second shard value, and based on the fact that the second shard value is less than the total number of preset data nodes, the second data node writes the second shard value to the service registry. If the second data node has written the second shard value to the service registry, the second data node loads data according to the second shard value. If the second data node has not written the second shard value to the service registry, the second data node determines the third shard value. Based on the fact that the third shard value is less than the total number of preset data nodes, the second data node writes the third shard value to the service registry. If the second data node has written the third shard value to the service registry, the second data node loads the third data according to the third shard value. If the second data node has not written the third shard value to the service registry, the second data node determines the fourth shard value until the determined shard value is equal to the total number of preset data nodes. When the determined shard value is equal to the total number of preset data nodes, the second data node suspends writing the shard value to the service registry and records the duration of the suspension.

[0163] The following example describes determining the next shard value to write to the service registry.

[0164] When a data node does not write a shard value to a service registry, the data node may determine the next shard value to be written to the service registry based on the shard value that has not been written. For example, if the first data node does not write a first shard value to the service registry, the first data node may determine the next shard value to be written to the service registry based on the first shard value, such as the second shard value. If the second data node does not write a second shard value to the service registry, the second data node may determine the next shard value to be written to the service registry based on the second shard value, such as the third shard value.

[0165] In an embodiment of the present application, the data node that has not written a shard value to the service registry is determined to have the same shard value as the next shard value written to the service registry. The data node can perform the same processing on the shard value that has not been written, such as performing the same operation, thereby ensuring that the next shard value written to the service registry is the same.

[0166] For example, if a fixed difference is added to the unwritten shard values, the fixed difference may be 1 or 2, etc., and this application does not impose any specific limitation on this.

[0167] Specifically, the first shard value determined based on step S301 is the same, and then the data node that has not written the first shard value to the service registration table subsequently performs the same processing on the first shard value, then the determined second shard value is the same, and so on, the same processing is performed on the second shard value, then the determined third shard value is the same.

[0168] Exemplarily, the first shard value is 0, and the data node that has not written the first shard value 0 to the service registry adds 1 to the first shard value 0 to obtain the second shard value 1, so that the second shard value 1 determined by the data node that has not written the first shard value 0 to the service registry is the same. The data node that has not written the second shard value 1 to the service registry adds 1 to the second shard value 1 to obtain the third shard value 2, so that the third shard value 2 determined by the data node that has not written the second shard value 1 to the service registry is the same, and so on.

[0169] The following is an example of setting the total number of preset data nodes.

[0170] In an embodiment of the present application, an administrator can set a preset total number of data nodes total through a system configuration or management interface, and the preset total number of data nodes total is stored in a concurrent control server. For example, a database maintains a preset total number of data nodes total, a user modifies the preset total number of data nodes total, and the modified preset total number of data nodes total is stored in the database, and all data nodes access the database to realize centralized reading of the preset total number of data nodes total. All data nodes store the same preset total number of data nodes.

[0171] See also Figure 4A , Figure 4A The management interface 401 of the display system displays the total number of preset data nodes 402, the edit control 403, and the delete control 404. The total number of preset data nodes 402 indicates that the value of the total number of preset data nodes is 2, that is, there are two data nodes in total corresponding to the data shard scheduling task SERVER_SHARD. In response to the operation of the edit control 403, the following is displayed: Figure 4B The configuration modification interface 405 is shown. The configuration modification interface 405 displays a preset data node total number input box 406, a confirmation control 407, and a cancel control 408. Figure 4B As shown, 3 is input in the preset data node total number input box 406, indicating that the preset data node total number is adjusted to 3. In response to the input to the confirmation control 407, the value of the preset data node total number of SERVER_SHARD is adjusted from 2 to 3, that is, 3 data nodes actually need to be scheduled.

[0172] In some other embodiments, all data nodes jointly set the same preset total number of data nodes total. When the value of the preset total number of data nodes total needs to be adjusted, all data nodes jointly adjust the preset total number of data nodes total.

[0173] It can be understood that the method of configuring the preset total number of data nodes total can be set according to actual conditions, and the embodiments of the present application do not make specific limitations on this.

[0174] It should be noted that the total number of preset data nodes is not fixed. With the development of business and the expansion of the system, it may be necessary to increase or decrease data nodes to adapt to different needs and scenarios. In practical applications, the selection of the total number of preset data nodes can be based on comprehensive considerations of specific business needs, data volume, query performance requirements, and budget factors. Usually, the optimal total number of preset data nodes can be determined through performance testing and simulation experiments to ensure that the system can meet business needs and provide the best performance and reliability. This embodiment of the application does not specifically limit this.

[0175] The following example describes how a data node loads data based on shard values.

[0176] In an embodiment of the present application, the data node that has written the shard value to the service registry can determine the identification value of the corresponding data based on the preset total number of data nodes and the shard value it has written, and then load the corresponding data according to the identification value.

[0177] Specifically, the first data node loads the first data according to the first shard value, including: the first data node determines the identification value of the first data according to the first shard value and a preset total number of data nodes, and then loads the first data according to the identification value of the first data.

[0178] Correspondingly, the second data node loads the second data according to the second shard value, including: the second data node determines the identification value of the second data according to the second shard value and the preset total number of data nodes, and then loads the second data according to the identification value of the second data, and the second number may be different from the data loaded according to the first shard value.

[0179] In an embodiment of the present application, the data node that has written the shard value to the service registry can load the corresponding data according to the unified data query rules and the shard value it has written, ensuring that a single or multiple data node failure occurs without affecting other normal data nodes, and after the failed data node is restored, it can also load the corresponding data based on the data query rules and the shard value it has written.

[0180] The data query rule may indicate the relationship between the data to be loaded, the shard value, and the total number of preset data nodes, and then determine which data node corresponding to which shard value the data to be loaded should be loaded by based on the relationship. The data query rule may depend on a specific sharding strategy and technical implementation, and this application does not specifically limit the data query rule.

[0181] Exemplarily, the data query rule may indicate that the remainder of the unique identification value of the data to be loaded divided by the total number of preset data nodes is the shard value. Exemplarily, each piece of data to be loaded has a unique primary key ID, and the data query rule may be: primary key ID% total number of preset data nodes = shard value. Then the identification value of the data loaded from the data node that has written the shard value to the service registry is: K times the total number of preset data nodes plus the written shard value, where K is an integer and can be 0, 1, 2, etc.

[0182] For example, the shard values ​​written by data nodes A, B, and C to the service registry are 0, 1, and 2 respectively, the primary key IDs of the data to be cached to the data nodes are 0-5 respectively, and the total number of preset data nodes is 3.

[0183] For data with primary key ID 0, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID0% total number of preset data nodes3=0 (shard value), it can be determined that data with primary key ID 0 should be stored in data node A which has written shard value 0 to the service registry.

[0184] For data with primary key ID 1, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID1% total number of preset data nodes3=1 (shard value), it can be determined that the data with primary key ID 1 should be stored in data node B which has written shard value 1 to the service registry.

[0185] For data with primary key ID 2, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID2% total number of preset data nodes3 = 2 (shard value), it can be determined that the data with primary key ID 2 should be stored in data node C which has written shard value 2 to the service registry.

[0186] For data with primary key ID 3, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID3% total number of preset data nodes 3 = 0 (shard value), it can be determined that the data with primary key ID 3 should be stored in data node A which has written shard value 0 to the service registry.

[0187] For data with primary key ID 4, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID4% total number of preset data nodes 3 = 1 (shard value), it can be determined that data with primary key ID 4 should be stored in data node B, which has shard value 1 written to the service registry. The same applies to other data. Data with primary key ID 5 is stored in data node C, which has shard value 2 written to the service registry.

[0188] In some embodiments, for any data node that loads corresponding data according to the shard value, when the preset total number of data nodes is not updated, the data node updates the data stored in the data node according to the processing instructions of the data, where the processing instructions include deletion, disabling, modification and addition.

[0189] Specifically, after the first data node loads the first data according to the first shard value, the data processing method further includes: in the case where the total number of preset data nodes is not updated, determining the processing instruction of the first data according to the data state of the first data, and then updating the first data stored in the first data node according to the processing instruction of the first data. The data state includes deletion, disabling, modification and addition. When the data state of the data is deletion, the processing instruction is to delete the data. When the data state of the data is disabling, the processing instruction is to disable the data. When the data state of the data is modification, the processing instruction is to modify the data. When the data state of the data is increase, the processing instruction is to increase the data. Correspondingly, after the second data node loads the second data according to the second shard value, the data processing method further includes: in the case where the total number of preset data nodes is not updated, determining the processing instruction of the second data according to the data state of the second data, and then updating the second data stored in the second data node according to the processing instruction of the second data, wherein the data state includes deletion, disabling, modification and increase.

[0190] Exemplarily, taking the case where the first data node updates the first data stored in the first data node, and the first data includes data D1 to D5, if the data status of data D1 is deleted, the first data node deletes the stored data D1. If the data status of data D2 is disabled, the first data node sets data D2 to be disabled. If the data status of data D3 is modified, the first data node saves the modified data D3. If the data status of data D4 is added, the first data node saves the added data D4.

[0191] In some embodiments, for any data node that loads corresponding data according to a shard value, when the total number of preset data nodes increases, the data node that loads corresponding data according to the shard value obtains the total number of preset data nodes after the increase, and loads the corresponding data according to the first shard value and the total number of preset data nodes after the increase. When the total number of preset data nodes increases, it also includes a newly added data node, and the newly added data node executes the data processing method provided in the embodiment of the present application to write the shard value to the service registry.

[0192] Specifically, after the first data node loads the first data according to the first shard value, the data processing method further includes: when the total number of preset data nodes increases, the total number of preset data nodes after the increase is obtained, and the first data node loads the corresponding second data according to the first shard value and the total number of preset data nodes after the increase. Correspondingly, after the second data node loads the second data according to the second shard value, the data processing method further includes: when the total number of preset data nodes increases, the total number of preset data nodes after the increase is obtained, and the second data node loads the corresponding data according to the second shard value and the total number of preset data nodes after the increase.

[0193] As shown in the above example, the total number of preset data nodes changes from 3 to 4, a new data node D is added, and the data node has written a shard value of 3 to the service registry.

[0194] For data with primary key ID 0, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID0% total number of preset data nodes4=0 (shard value), it can be determined that data with primary key ID 0 should be stored in data node A which has written shard value 0 to the service registry.

[0195] For data with primary key ID 1, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID1% total number of preset data nodes4=1 (shard value), it can be determined that the data with primary key ID 1 should be stored in data node B which has written shard value 1 to the service registry.

[0196] For data with primary key ID 2, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID2% total number of preset data nodes4=2 (shard value), it can be determined that the data with primary key ID 2 should be stored in data node C which has written shard value 2 to the service registry.

[0197] For data with primary key ID 3, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID3% total number of preset data nodes4=3 (shard value), it can be determined that the data with primary key ID 3 should be stored in data node D which has written shard value 3 to the service registry.

[0198] For data with primary key ID 4, according to the data query rule: primary key ID% total number of preset data nodes = shard value, that is, primary key ID4% total number of preset data nodes 4 = 0 (shard value), it can be determined that the data with primary key ID 4 should be stored in data node A, which has written shard value 0 to the service registry. The same applies to the others. Data with primary key ID 5 is stored in data node B, which has written shard value 1 to the service registry.

[0199] The data with primary key ID 3 was originally loaded into data node A. After the preset total number of data nodes was changed, the data with primary key ID 3 was loaded into data node D. The data with primary key ID 4 was originally loaded into data node B. After the preset total number of data nodes was changed, the data with primary key ID 4 was loaded into data node A. The data with primary key ID 5 was originally loaded into data node C. After the preset total number of data nodes was changed, the data with primary key ID 5 was loaded into data node B.

[0200] In some embodiments, for any data node that loads corresponding data according to the shard value, when the total number of preset data nodes decreases, the data node stops updating the data stored in the data node and deletes the data stored in the data node.

[0201] Specifically, after the first data node loads the first data according to the first shard value, the method further includes: when the total number of preset data nodes decreases, the first data node stops updating the first data stored in the first data node and deletes the first data stored in the first data node. After the second data node loads the second data according to the second shard value, the method further includes: when the total number of preset data nodes decreases, the second data node stops updating the second data stored in the second data node and deletes the second data stored in the second data node.

[0202] The following is an exemplary introduction to another data processing method provided in an embodiment of the present application.

[0203] See also Figure 5 , exemplarily introduces the data processing method provided in the embodiment of the present application. The data processing method is applied to a data node and can be executed by a data node or a data processing device.

[0204] Step S501: Acquire a first instruction for instructing data processing.

[0205] In the embodiment of the present application, the first instruction is used to instruct the execution of the data processing method provided in the embodiment of the present application. The data node executes the data processing method provided in the embodiment of the present application, enters the data processing flow, and performs data sharding scheduling.

[0206] In some embodiments, the data node is installed with a target application, and the data processing device is used as a module of the target application. When the target application is started or the data processing device is started, the data node can obtain the first instruction.

[0207] In some embodiments, the data node may receive a user operation and obtain a first instruction in response to the user operation. For example, there are five data nodes in total. Based on the preset total number of data nodes being 3, only three data nodes are scheduled, and the remaining two data nodes are idle. When the node needs to be expanded later, the administrator may manually trigger the two idle data nodes, and the two idle data nodes obtain the first instruction in response to the user operation.

[0208] Step S502, obtaining the total number of preset data nodes.

[0209] In an embodiment of the present application, when a data node executes the data processing method provided in the embodiment of the present application, it can obtain the total number of configured preset data nodes by accessing the concurrency control server, and store the obtained total number of preset data nodes, such as storing it locally in the data node.

[0210] Exemplarily, the user sets the total number of preset data nodes to 3, and starts five data nodes. The five data nodes all execute the data processing method provided in the embodiment of the present application. Three of the five data nodes successfully execute the operation of writing the shard value to the service registry, and then can obtain and store the corresponding data according to the written shard value, and subsequently can execute the corresponding scheduling task according to the obtained data. The remaining two data nodes of the five data nodes did not successfully execute the operation of writing the shard value to the service registry. The two data nodes can exit the process, or restart to re-execute the data processing method provided in the embodiment of the present application.

[0211] Step S503: Delete the registration information of the fourth data node in the service registration table.

[0212] In an embodiment of the present application, the data node accesses the service registry maintained by the concurrency control server. The service registry maintains the registration information of the successfully registered data nodes, including the heartbeat update time of the successfully registered data nodes. Any data node accessing the service registry can clear all registration information of the fourth data node in the service registry according to the time it accesses the service registry and the preset duration.

[0213] The fourth data node is the data node to be deleted. For example, when a data node fails to update the heartbeat update time for a preset period of time, that is, the data node fails to update the heartbeat for a preset period of time, it is considered that the data node has lost the heartbeat, and the data node is the fourth data node. The heartbeat is the behavior of a data node updating the heartbeat update time of the update time field in the service registration table at a fixed periodic frequency (i.e., the heartbeat update frequency).

[0214] Exemplarily, the data node that has successfully registered updates the heartbeat update time stored in the update time field every 20 seconds to maintain the heartbeat, and the preset duration is 1 minute. The successfully registered data node A updates the time (heartbeat update time) in the update time field corresponding to data node A in the service registration table at a fixed frequency every 20 seconds. At time T1, data node B accesses the service registration table, and data node B obtains the heartbeat update time a stored in the update time field corresponding to data node A, and detects that the time difference between the time when data node B accesses the service registration table (i.e., T1) and the heartbeat update time a exceeds the preset duration of 1 minute, that is, data node A has lost at least three update heartbeats, and data node B determines that data node A has lost its heartbeat. Data node B clears all registration information of data node A in the service registration table, including the data recorded by data node A in the primary key field, data node field, shard value field, update time field, total data field, and registration time field. Please refer to the following Table 1, Table 1 is a service registration table, if data node A is the fourth data node, data node B deletes all data in the row where data node A is located in Table 1. Based on deleting the shard value 0 that data node A has written to the service registry, the shard value 0 occupied by data node A is released, and other data nodes can compete for the shard value 0.

[0215] Among them, the preset duration and heartbeat update frequency can be set according to actual conditions, and the embodiments of the present application do not make specific limitations on this.

[0216] In some embodiments, when any data node accesses the service registration table at any time and detects the presence of a fourth data node in the service registration table, such as a data node that has lost its heartbeat, the data node may clear the registration information of the fourth data node. Step S503 is not limited to being performed after step S502.

[0217] Step S504: determine the first fragmentation value according to a preset initial value.

[0218] In the embodiment of the present application, all data nodes executing the data processing method provided in the embodiment of the present application store the same preset initial value. All data nodes access the service registry to register, and all data nodes determine the first shard value according to the preset initial value to write the first shard value to the service registry when registering.

[0219] In some embodiments, step S504 may be performed simultaneously with step S503, step S504 may be performed before step S503, or step S504 may not be performed. If step S504 is not performed, based on the fact that all data nodes store the same preset initial value, all data nodes directly use the preset initial value as the first shard value and write the first shard value to the service registry.

[0220] Step S505: Use the first fragmentation value as the current fragmentation value and compete with the current fragmentation value.

[0221] In an embodiment of the present application, based on the fact that all data nodes store the same preset initial value, the first shard value determined in step S504 or the first shard value written in step S505 is the same, so that all data nodes in step S505 write the same first shard value to the service registration table to compete for the same first shard value.

[0222] In step S505, all data nodes use the same first shard value as the current shard value to compete for the same current shard value, wherein the current shard value is the shard value to be competed for. During the first competition, there is no shard value in the service registration table at this time, and the current shard value is the first shard value determined according to the preset initial value. After the competition fails, there is already a shard value in the service registration table at this time, and the current shard value can be the next shard value determined according to the shard value that failed the competition.

[0223] Exemplarily, taking MySQL as an example, all data nodes register their own nodes in the service registry of MySQL and compete for the current shard value to obtain the current shard value. In order to ensure that the shard value that all data nodes compete for starts from the same first shard value and that the shard values ​​that they compete for satisfy a certain rule, such as satisfying the continuity of natural numbers, each data node starts from num=first shard value when competing to obtain the shard value, that is, the value of the shard value field shard_num in the service registry starts from the first shard value. Although the AUTO_INCREMENT attribute of MySQL can ensure the continuity of the shard value field shard_num, it ensures that the written shard value is unique in the service registry. Therefore, by setting the shard value field shard_num to have a unique index, it can be ensured that when multiple data nodes simultaneously perform a write operation to write (insert) the same shard value to the service registry, only one data node successfully performs the write operation.

[0224] In an embodiment of the present application, all data nodes write the same shard value (such as the first shard value or the shard value re-determined after the competition fails) to the service registry. Since the shard values ​​written by all data nodes to the service registry are the same and are unique in the service registry based on the written shard value, all data nodes need to compete for the shard value. All data nodes compete for the same shard value, that is, all data nodes write the same shard value to the service registry. Based on the uniqueness of the written shard value in the service registry, if an existing data node occupies the shard value, all other data nodes competing for the shard value will fail. When the competition is successful, that is, the data node has written the shard value to the service registry, the data node executes step S506. When the competition fails, that is, the data node has not written the shard value to the service registry, the data node executes step S514.

[0225] In some embodiments, step S503 and step S505 may be performed simultaneously, and step S505 may also be performed before step S503.

[0226] Step S506, obtain the shard value of the successful competition and update the heartbeat of the data node.

[0227] In the embodiment of the present application, when the data node has written the first shard value to the service registration table, the data node successfully competes for the first shard value, and the first shard value uniquely belongs to the data node. The data node that successfully competes updates the registration information in the service registration table, which may include but is not limited to: id, IP address, the successfully competed shard value, the total amount of data processed, the time of successful registration, and the heartbeat update time.

[0228] As in the above example, when data node A successfully competes for the first shard value 0, data node A will write its corresponding id, IP address, the shard value successfully competed for, the total amount of data processed, the time of successful registration (i.e., the time of obtaining the shard value), and the heartbeat update time into the service registry. At this time, the service registry is as shown in Table 1.

[0229] Table 1

[0230]

[0231] As shown in Table 1, data node A records the first shard value 0 that data node A has written in the shard value field shard_num, records the IP address 10.144.20.95 of data node A in the data node field, records the total amount of data 1214 processed by data node A in the total data field data_num, and records the time when data node A successfully writes the first shard value 0 to the service registration table in the registration time field create_time, which corresponds to the time when data node A successfully registers 2024-4-24 11:27:00. Based on the fact that data node A will update its heartbeat at the same time when it successfully registers, the heartbeat update time recorded in the update time field update_time can be consistent with the time when data node A successfully registers 2024-4-24 11:27:00.

[0232] In an embodiment of the present application, when a data node successfully competes for the current shard value, the data node needs to start a heartbeat update mechanism. The heartbeat update mechanism includes a heartbeat update frequency. After the data node starts the heartbeat update mechanism, the data node updates its heartbeat to the service registration table according to the heartbeat update frequency. The subsequent data node continuously updates the time recorded by update_time according to the heartbeat update frequency, and the time recorded by the updated update_time is different from the time recorded by the registration time field create_time.

[0233] Taking the heartbeat update frequency of every 20 seconds as an example, when a data node successfully competes for the current shard value, the data node updates the value of the update time field update_time in the service registry every 20 seconds to prove that its node is alive and not cleared.

[0234] Step S507, loading and caching corresponding data according to the successfully competed shard value.

[0235] In an embodiment of the present application, for any successfully registered data node, the data node loads and caches the corresponding data according to the shard value for which it has successfully competed. For example, data node A has written shard value 0 to the service registry, that is, data node A has successfully competed for shard value 0, and data node A loads the corresponding data a1 according to shard value 0 and stores the loaded data a1 to data node A. Data node B has written shard value 1 to the service registry, that is, data node B has successfully competed for shard value 1, and data node B loads the corresponding data b1 according to shard value 1 and stores the loaded data b1 to data node B. Data node A and data node B can load data based on the same data query rules. Based on the different shard values ​​0 and shard values ​​1, the data a1 and data b1 loaded by data nodes A and B are different.

[0236] In an embodiment of the present application, for any successfully registered data node, the data node can load the corresponding data according to the shard value and data query rules of the successful competition. The specific content can be referred to above and will not be repeated here.

[0237] It can be understood that the data nodes that successfully compete all load data based on the same data query rules. Since the shard values ​​written by the data nodes that successfully compete to the service registry are different, the data loaded by the data nodes that successfully compete are different according to the written shard values.

[0238] In the embodiment of the present application, the data node competes to obtain the shard value. After establishing the heartbeat mechanism between the data node and the service registry (i.e., the data node updates the heartbeat to the service registry according to the heartbeat update frequency), the data node performs initial data loading and first loads the corresponding data according to the shard value that successfully competes. The data first loaded by the data node are all valid data, that is, except for the deleted and disabled data, the first loaded data to be processed are all valid data, and the valid data are stored in the data node.

[0239] Step S508, start the scheduled update cache task.

[0240] In an embodiment of the present application, after the data node first loads the data to be processed, the data node starts a timed update cache task to update the data stored in the data node according to a fixed data update frequency. The data node updates the data stored in the data node according to the data update frequency as a timed incremental update.

[0241] Specifically, after the data node executes step S507, the data to be processed corresponding to the shard value that successfully competes is stored in the data node. The data node determines the filtering time according to the data update frequency. Each scheduled update query will carry the filtering time, and all the data to be processed whose data status has changed within the filtering time are queried according to the data update frequency. In the scheduled incremental update, all the data to be processed whose data status has changed includes the newly added data, modified data, and disabled or deleted data within this filtering time range.

[0242] The data update frequency may be once per minute, once per three minutes, or once per four minutes, which may be set according to actual conditions, and the present application embodiment does not specifically limit this. If the data update frequency is once per minute, the filtering time may be determined to be 1 minute. If the data update frequency is once per three minutes, the filtering time may be determined to be 3 minutes.

[0243] Step S509: determine whether the total number of preset data nodes changes.

[0244] In the embodiment of the present application, the administrator can adjust the value of the preset total number of data nodes according to actual needs. For example, when the amount of data and business needs increase, the number of data nodes can be increased. When the business needs and data processing volume are not large, the number of data nodes can be reduced. FIG. 4A to FIG. 4B , the administrator can adjust the total number of preset data nodes according to actual conditions.

[0245] In an embodiment of the present application, after the data node starts the scheduled update cache task, a scheduled incremental update is performed. Before performing the scheduled incremental update, the data node again obtains the total number of preset data nodes configured by the system, and compares the total number of preset data nodes obtained with the total number of preset data nodes obtained in step S502, that is, the total number of preset data nodes obtained again is compared with the total number of preset data nodes stored locally in the data node. When the total number of preset data nodes obtained again is equal to the total number of preset data nodes stored locally in the data node, step S510 is executed. When the total number of preset data nodes obtained again is greater than the total number of preset data nodes stored locally in the data node, step S511 is executed. When the total number of preset data nodes obtained again is less than the total number of preset data nodes stored locally in the data node, step S513 is executed.

[0246] Step S510, loading data that meets the filtering time, and updating the cached data according to the data status.

[0247] In an embodiment of the present application, after the data node starts the scheduled update cache task, it performs a scheduled incremental update, loads the data that meets the filtering time, and updates the cached data according to the data status. When the data status of the data that meets the filtering time is disabled or deleted, the data is deleted from the data node. When the data status of the data that meets the filtering time is modified, the modified data is stored in the data node. When the data status of the data that meets the filtering time is increased, the modified data is stored in the data node.

[0248] In some embodiments, the data node executes step S507 to store the corresponding data to be processed according to the shards and data query rules that have successfully competed, and determines the data to be processed whose data status has changed during the filtering time according to the data update frequency. After loading the data to be processed whose data status has changed during the filtering time, the node cache is updated according to the data status. The data status may include deletion, modification, addition, and disabling.

[0249] Taking the data update frequency of 1 minute as an example, after data node A loads data a1 belonging to data node A from the database according to the shard value that successfully competes, and stores data a1 in data node A, data node A executes the scheduled cache update task, and each update query of data node A carries a filter time. Each time the scheduled cache update task is executed, data node A will query the database to find all records whose status has changed in data a1 within a specific time range (i.e., the last minute, from T1 (one minute before the current time T2) to T2 (the current time)). These changes include but are not limited to: newly added records, modifications to existing records (such as changes in data content or changes in data status, such as from enabled to disabled), and deleted records (i.e., changes in data status, such as marking as deleted). The query result is assumed to be a2, which includes a subset of all a1 data sets that have changed in the last minute. Data node A synchronizes the queried changed data a2 to its local storage or cache to reflect the latest data status. If there are new or modified records, they will be added to or updated in the cache of data node A. If there are records marked as deleted or disabled, data node A may need to update its cache to reflect the latest status of these records, which may involve removing these records from the cache or marking them as deleted / disabled.

[0250] Step S511, determining whether the data node is a newly added data node.

[0251] When the total number of preset data nodes obtained in step S509 becomes larger than the total number of preset data nodes obtained in step S502, the data node determines whether it is a newly added data node. That is, when the total number of preset data nodes becomes larger, if the data node has not obtained a shard value, the data node is a newly added data node, and if the data node has obtained a shard value, the data node is not a newly added data node.

[0252] In an embodiment of the present application, a data node determines whether it is a newly added data node. If not, step S512 is executed. If so, the data node starts executing from step S501 and competes for the shard value again. Based on the fact that each data node that successfully competes has a fixed and unique shard value belonging to it, the newly added data node starts executing from step S501 and competes for the shard value again. If it cannot compete for the shard value occupied by other data nodes, the newly added data node can obtain a new shard value. Other data nodes can load data according to their original shard values ​​that successfully competed, so as to achieve the situation where the service processing pressure increases, the data node expansion is processed without stopping the service and the data is quickly distributed. In some embodiments, the newly added data node starts executing from step S501. If the shard value occupied by a data node is released, the newly added data node can obtain the released shard value.

[0253] Step S512, obtaining the increased total number of preset data nodes, and updating the cached data.

[0254] In step S509, when it is determined that the total number of preset data nodes changes and the total number of preset data nodes increases, the sharding value held by the data node that has successfully competed remains unchanged, the data node that has successfully competed obtains the increased total number of preset data nodes, and stores the increased total number of preset data nodes.

[0255] In an embodiment of the present application, when the data query rule is irrelevant to the total number of preset data nodes, the data node may refer to step S510 to update the node cache.

[0256] In an embodiment of the present application, when the data query rule indicates that when loading data according to the shard value of the successful competition, it is necessary to refer to the total number of preset data nodes, then the data query rule is related to the total number of preset data nodes. For example, the data query rule indicates the relationship between the shard value of the successful competition, the total number of preset data nodes, and the shard value, then the data query rule is related to the total number of preset data nodes.

[0257] In an embodiment of the present application, when the data query rules are related to the preset total number of data nodes, the data node loads the corresponding data based on the successful shard value, data query rules, and the total number of preset data nodes after the increase, and stores the loaded data to the data node, and then executes the scheduled update cache task.

[0258] In an embodiment of the present application, the data node caches data, and the data node can make full use of the cache mechanism to store and reuse previously loaded data. The data node can update the node cache according to the shard value uniquely belonging to it and the data query rule. Thus, it is well supported to support the use scenario where the data needs to be sharded and the shard value of each data node needs to be relatively fixed. The data node can perform corresponding scheduling tasks based on the cached data.

[0259] Step S513, stop updating the heartbeat, stop the scheduled cache update task, clear the cached data, and re-compete for the shard value.

[0260] In an embodiment of the present application, when the total number of preset data nodes becomes smaller, the data nodes that have successfully competed stop updating the heartbeat, that is, stop updating the heartbeat update time recorded in the update time field, so that other data nodes or concurrent control servers can delete the data of the data node that has lost the heartbeat in the service registration table, thereby releasing the shard value occupied by the data node.

[0261] In the embodiment of the present application, when the total number of preset data nodes becomes smaller, the data nodes that have successfully competed all stop the scheduled update cache task, including stopping the update of the cached data according to the data status.

[0262] In the embodiment of the present application, when the total number of preset data nodes becomes smaller, the data nodes that have successfully competed can clear the cached data, including clearing the data cached in the above steps S507, S510, and S512.

[0263] In an embodiment of the present application, when the total number of preset data nodes becomes smaller, the data nodes that have successfully competed compete for the shard value again, that is, execution starts from step S502, and it is determined whether the new total number of preset data nodes is greater than or equal to 1. If so, execution starts again from step S503. If not, the data node exits the competition.

[0264] In some embodiments, when the total number of preset data nodes becomes smaller, the data nodes that have successfully competed obtain the reduced total number of preset data nodes. When the reduced total number of preset data nodes is greater than or equal to 1, the data nodes can re-execute the data processing method provided in the embodiment of the present application or exit the competition.

[0265] In some embodiments, when the total number of preset data nodes becomes smaller, the data node that has successfully competed can first stop updating the heartbeat and stop the scheduled update cache task, and then compete for the shard value again. When it is determined that the shard value that has been exited from the competition or successfully re-competed is different from the shard value originally held, the data node clears its cached data. When the shard value that the data node successfully re-competed is consistent with the shard value obtained previously, that is, the data assigned to the data node is consistent, the data node may not clear its cached data.

[0266] Step S514, obtaining a next shard value according to the current shard value that failed in the competition, and comparing the next shard value with the total number of preset data nodes.

[0267] In an embodiment of the present application, for any data node that fails to compete for the current shard value, the data node obtains the next shard value based on the current shard value that failed to compete, that is, obtains the next shard value written to the service registry. There is a numerical correspondence between the current shard value that failed to compete and the next shard value (next shard value) written to the service registry, so that the current shard value that failed to compete and the next shard value satisfy natural number continuity. The numerical correspondence may include: if the difference between the current shard value that failed to compete and the next shard value is a fixed difference, then the next shard value can be obtained by increasing the current shard value that failed to compete by the fixed difference.

[0268] In an embodiment of the present application, after determining the next shard value, it is necessary to compare the next shard value with the total number of preset data nodes. For example, compare the next shard value with the total number of preset data nodes obtained in step S502. Alternatively, when the data node executes step S509, the latest total number of preset data nodes can be obtained, and the latest total number of data nodes can include: the total number of preset data nodes that has not changed, the total number of preset data nodes after increase, or the total number of preset data nodes after decrease, and compare the next shard value with the latest total number of preset data nodes. Determine whether the data node that has not successfully competed can continue to compete for the shard value based on the comparison result between the next shard value and the total number of preset data nodes. When the next shard value is less than the total number of preset data nodes (or the minimum total number of preset data nodes), the data node that has not successfully competed can continue to compete for the shard value, and execute step S515. When the next shard value is greater than or equal to the total number of preset data nodes (or the minimum total number of preset data nodes), the data node that has not successfully competed needs to further determine whether it can continue to compete for the shard value, and execute step S516.

[0269] Step S515: taking the next fragmentation value as the current fragmentation value and competing with the current fragmentation value.

[0270] In an embodiment of the present application, the data nodes that have not successfully competed use the next shard value determined in step S514 as the current shard value and compete for the current shard value, that is, they all write the next shard value determined in step S514 to the service registration table.

[0271] Exemplarily, the next shard value 1 is used as the current shard value and competes for the current shard value 1. The data node that failed the competition executes step S514, obtains the next shard value 2 according to the current shard value 1 that failed the competition, and compares the next shard value 2 with the preset total number of data nodes.

[0272] Take the preset initial value as 0, the fixed difference value as 1, the preset total number of data nodes as 3, and all data nodes including data nodes A to data node D as an example. Data nodes A to D determine the shard value 0 according to the preset initial value, and data nodes A to D all write the shard value 0 to the service registry. When data node A successfully executes the write operation of writing the shard value 0 to the service registry, data node A has written the shard value 0 to the service registry. Based on the setting that the written shard value is unique in the service registry, data nodes B to D have not successfully executed the write operation of writing the shard value 0 to the service registry, that is, data nodes B to D have not written the shard value 0 to the service registry. Based on the setting that the written shard value is unique in the service registry, the shard value 0 uniquely belongs to data node A, data node A exits the competition, and data node A executes step S506.

[0273] Based on the fact that data nodes B to D have not written the shard value 0 to the service registry, data nodes B, C, and D determine the next shard value based on the current shard value that failed the competition (i.e., shard value 0), such as increasing the current shard value 0 that failed the competition by 1, to obtain the next shard value 1.

[0274] When data nodes B to D determine that the next shard value 1 is less than the total number of preset data nodes, the next shard value 1 is used as the current shard value to be competed, and competes for shard value 1. Data nodes B to D all write shard value 1 to the service registry. When data node B successfully executes the write operation of writing shard value 1 to the service registry, data node B has written shard value 1 to the service registry. Based on the setting that the written shard value is unique in the service registry, data nodes C to D have not successfully executed the write operation of writing shard value 1 to the service registry, that is, data nodes C to D have not written shard value 1 to the service registry. Based on the setting that the written shard value is unique in the service registry, shard value 1 uniquely belongs to data node B, data node B exits the competition, and data node B executes step S506.

[0275] Because data nodes C to D have not written shard value 1 to the service registry, data nodes C and D determine the next shard value based on the current shard value that failed the competition (i.e., shard value 1), such as increasing the current shard value 1 that failed the competition by 1, to obtain the next shard value 2.

[0276] When data nodes C to D determine that the next shard value 2 is less than the preset total number of data nodes, the next shard value 2 is used as the current shard value to be competed, and competes for shard value 2. Data nodes C to D all write shard value 2 to the service registry. When data node C successfully executes the write operation of writing shard value 2 to the service registry, data node C has written shard value 2 to the service registry. Based on the setting that the written shard value is unique in the service registry, data node D fails to successfully execute the write operation of writing shard value 2 to the service registry, that is, data node D does not write shard value 2 to the service registry.

[0277] Based on the setting that the written shard value is unique in the service registration table, shard value 2 uniquely belongs to data node C, data node C exits the competition, and data node C executes step S506.

[0278] In the embodiment of the present application, when or after the data node successfully writes the corresponding shard value, the data node writes its corresponding id, IP address, the successfully competed shard value, the total amount of data processed, the time of obtaining the shard value, and the time of the most recent heartbeat update to the service registration table. If data node A obtains shard value 0 and data node B obtains shard value 1, the service registration table is as shown in Table 2.

[0279] Table 2

[0280]

[0281] Step S516, suspending the contention for the fragment value until the suspension duration reaches a preset duration.

[0282] In an embodiment of the present application, when the next shard value is greater than or equal to the total number of preset data nodes, the competition for the shard value can be suspended until the suspension time reaches the preset duration. By suspending the competition for the shard value until the suspension time reaches the preset duration, the data node that occupies the shard value but no longer exists in the service registration table has no heartbeat update (not smooth or this sentence is not finished), which facilitates the clearing of the data node to release the shard value occupied by the data node.

[0283] Step S517, determining whether the number of pauses for pausing the contention fragment value is less than a preset number.

[0284] In the embodiment of the present application, when the next shard value is equal to or greater than the preset total number of data nodes, the competition shard value is suspended and the number of suspensions is recorded. When the number of suspensions for suspending the competition shard value is greater than or equal to the preset number of times, the data node executes step S518 to exit the competition. When the number of suspensions for suspending the competition shard value is less than the preset number of times, the data node re-executes step S503.

[0285] Taking the preset duration of 1 minute and the preset number of times of 4 as an example, as in the above example, based on the fact that data node D has not written the shard value 2 to the service registry, data node D determines the next shard value based on the current shard value that failed the competition (i.e., shard value 2), that is, the current shard value 2 that failed the competition is increased by 1, and the next shard value 3 is obtained. Data node D determines that the next shard value 3 is equal to the preset total number of data nodes 3, then data node D suspends the competition for the shard value until the suspension duration reaches the preset duration, that is, data node D will put the write operation of writing the shard value to the service registry to sleep for 1 minute. This one minute will prevent the data node that occupies the shard value but no longer exists in the service registry from having a heartbeat update, which facilitates the subsequent deletion of the data node, thereby releasing the shard value occupied by the data node.

[0286] At time T1, data node D suspends competing for the shard value and records the number of suspensions as 1. At time T1+1 minute, data node D determines that the number of suspensions 1 is less than the preset number of times 3, and then re-executes step S503. The data node re-executes step S503, determines shard value 0 according to the preset initial value and competes for shard value 0, and the competition fails because shard value 0 has been written by data node A. Data node D determines the next shard value 1 according to the current shard value 0 that failed to compete, and competes for shard value 1, and the competition fails because shard value 1 has been written by data node B. Data node D determines the next shard value 2 according to the current shard value 1 that failed to compete, and competes for shard value 2, and the competition fails because shard value 2 has been written by data node C. Data node D determines the next shard value 3 according to the current shard value 2 that failed to compete, and the next shard value 3 is equal to the preset total number of data nodes, and then data node D executes step S516. At time T2, data node D suspends competing for the shard value and records the number of suspensions as 2. At the moment of T2+1 minute, data node D determines that the number of pauses 2 is less than the preset number of times 3, and then re-executes step S503. The relevant content of data node D re-executing step S503 can be referred to above. At T3, data node D suspends the competition for the shard value and records the number of pauses as 3. At the moment of T2+1 minute, data node D determines that the number of pauses 3 is equal to the preset number of times 3, and then re-executes step S518 to exit the competition.

[0287] In another exemplary embodiment, if at the moment of T2+1 minute, data node D determines that the number of pauses 2 is less than the preset number of times 3, then step S503 is re-executed, and data node D detects that the time difference between the heartbeat update time of data node A and the moment of T2+1 minute exceeds the preset duration, that is, data node A has not updated the heartbeat for more than the preset duration, then data node D deletes the registration information of data node A, and then there is no record of shard value 0 in the service registration table, and then shard value 0 is released. Data node D determines shard value 0 according to the preset initial value and competes for shard value 0. Based on the fact that shard value 0 is not written in the service registration table, data node D successfully executes the write operation of writing shard value 0 to the service registration table, and data node D has written shard value 0 to the service registration table, and data node D executes step S506.

[0288] Among them, the preset number of times can be 3, 4 or 5, and the preset number of times can be set according to actual conditions, and the embodiment of the present application does not make any specific limitation on this.

[0289] Exemplarily, the preset number is set to 4, that is, the repeatable contention fragment is set to 3 times. Compare whether the number of pauses is less than 4, if so, execute step S503, if not, execute step S518.

[0290] Step S518, exit the data processing flow.

[0291] When a data node exits the data processing flow, the data node no longer executes Figure 5 until it gets the first instruction.

[0292] In an embodiment of the present application, when a data node recovers after a failure, it can re-execute the above-mentioned data sharding scheduling process and regain the sharding value originally obtained through competition based on the above-mentioned data sharding scheduling process.

[0293] Exemplarily, data nodes A, B, and C compete for shard values ​​0, 1, and 2 respectively. If data node B fails or is down, then data node B has not updated its heartbeat for a preset period of time. Data nodes A and B can delete all registration information of data node B in the service registry, and the shard value 1 held by data node B will be released. After data node B recovers, data node B competes again, and it needs to compete continuously from the preset initial value. Data node A already holds shard value 0, so data node B will no longer get the shard value 0 already held by data node A. Data node B continues to compete for the next shard value 1. Since shard value 1 has been released, data node B will still compete for shard value 1.

[0294] In an embodiment of the present application, the shard value obtained based on the competition of each data node uniquely belongs to the data node, the shard value of each data node is unique (no duplication), the shard value satisfies the natural number self-increment, and each data node loads data according to the data query rule, which can ensure that the data to be processed assigned by the data node is not repeated. At the same time, the shard value of each data node is relatively fixed. When the data changes, the changed data to be processed can be updated based on the same data query rule to ensure that the same data is always assigned to the same data node. Because the shard value of each data point is fixed, the data query rules for loading data on each data node are the same. When a hidden extended data node or a data node fails, it does not interfere with other data nodes.

[0295] See also Figure 6 , exemplarily introduces the structure of the computer device 300 provided in an embodiment of the present application.

[0296] like Figure 6 As shown, the computer device 300 includes: a processor 301, a memory 302, a communication interface 303 and a bus 304. The processor 301, the memory 302 and the communication interface 303 communicate through the bus 304. The computer device 300 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computer device 300.

[0297] The bus 304 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus 104 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computer device 300 (for example, the memory 302, the processor 301, and the communication interface 303).

[0298] The processor 301 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0299] The memory 302 may include a volatile memory, such as a random access memory (RAM). The processor 301 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD).

[0300] An executable program code is stored in the memory 302, and the processor 301 executes the executable program code to realize the function of the front data processing device 200, thereby realizing the data processing method provided in the embodiment of the present application. That is, the memory 302 stores instructions for executing the data processing method provided in the embodiment of the present application. Alternatively, an executable code is stored in the memory 302, and the processor 301 executes the executable code to realize the function of the front data processing device 200, thereby realizing the data processing method provided in the embodiment of the present application. That is, the memory 302 stores instructions for executing the data processing method provided in the embodiment of the present application. The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card and a transceiver to realize the communication between the computer device 300 and other devices or a communication network.

[0301] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the computer device 300. In other embodiments of the present application, the computer device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0302] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0303] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0304] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0305] An embodiment of the present application also provides a computer program product, which stores at least one instruction. The method implemented when the at least one instruction is executed by a processor can refer to the methods in the above embodiments of the present application.

[0306] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0307] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0308] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0309] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a first shard value according to a preset initial value of the data node; The first data node writes the first shard value to the service registry. If the first data node has written the first shard value to the service registry, the first data node loads the first data according to the first shard value. The first data node is any one of the data nodes. If the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, the second data node writes the second shard value to the service registry, and the second data node is the data node among the data nodes that does not write the first shard value to the service registry.

2. The method according to claim 1, characterized in that If the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is less than the total number of preset data nodes, then the second data node writes the second shard value to the service registry, including: If the first data node does not write the first shard value to the service registration table, determining a second shard value according to the second data node; If the second shard value is less than the preset total number of data nodes, the second data node writes the second shard value to the service registration table.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: If the second data node has not written the second shard value to the service registry, and the number of first shard values ​​stored in the service registry is less than the total number of preset data nodes, the steps of determining the third shard value according to the third data node and writing the third shard value to the service registry are performed until the number of shard values ​​stored in the service registry is equal to the total number of preset data nodes, and the third data node is the data node among the data nodes that has not written a shard value to the service registry.

4. The method according to claim 1, characterized in that The method further comprises: If the first data node does not write the first shard value to the service registry, and the number of shard values ​​stored in the service registry is greater than or equal to the total number of preset data nodes, the second data node suspends writing the first shard value to the service registry and records the suspension duration; When the pause duration reaches a preset duration, the second data node writes the first shard value to the service registration table.

5. The method according to claim 4, characterized in that The second data node suspends writing the first shard value to the service registration table and records the suspension duration, including: The second data node suspends writing the first shard value to the service registration table, and records the suspension duration and the number of suspensions; When the pause duration reaches a preset duration, the second data node writes the first shard value to the service registration table, including: If the number of pauses is greater than or equal to a preset number of times, the second data node stops writing the first shard value to the service registration table; If the number of pauses is less than a preset number, the second data node writes the first shard value to the service registration table.

6. The method according to claim 1, characterized in that The first data node loads the first data according to the first shard value, including: The first data node determines an identification value of the first data according to the first shard value and the total number of preset data nodes; The first data is loaded according to the identification value.

7. The method according to claim 6, characterized in that After the first data node loads the first data according to the first shard value, the method further includes: In a case where the total number of the preset data nodes is not updated, the first data stored in the first data node is updated according to a processing instruction for the first data, wherein the processing instruction includes deletion, disabling, modification and addition.

8. The method according to claim 6, characterized in that After the first data node loads the first data according to the first shard value, the method further includes: When the total number of the preset data nodes increases, obtaining the increased total number of the preset data nodes; The first data node loads the second data according to the first shard value and the increased total number of preset data nodes.

9. The method according to claim 6, characterized in that After the first data node obtains the first data according to the first shard value, the method further includes: When the total number of the preset data nodes decreases, updating of the first data stored in the first data node is stopped and the first data stored in the first data node is deleted.

10. A computer device, characterized in that: The computer device comprises a processor and a memory, and the processor implements the method according to any one of claims 1 to 9 when executing a computer program stored in the memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer program product, characterized in that The computer program product stores at least one instruction, and when the at least one instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images