Data processing method and device, data system, computing node, electronic device, and computer-readable storage medium

Abstract

The present application discloses a data processing method and device, a data system, a computing node, an electronic device, and a computer-readable storage medium. The method includes: receiving a data write request; determining the first partition identification information of the first data storage partition to which the data to be written belongs according to the first data information; determining at least one second partition identification information of the second data storage partition to which the data to be written is written according to the write routing table and the first partition identification information; writing the data to be written to the second data storage partition identified by the second partition identification information according to the second partition identification information. The embodiment of the present application can use the write routing table that reflects the operating status of the processing node to adjust the allocation of shards calculated based on the data involved in the data write request, thereby realizing adaptive adjustment of the shards and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device for data storage, a data system, a computing node, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of data processing technology, the demand for data storage is also increasing. For this reason, distributed storage technology has emerged, which can store large amounts of data through storage servers based on multiple nodes connected physically or through the Internet. Distributed storage has high scalability and flexibility because it distributes data storage to multiple nodes. For scenarios where data is stored in databases, it is necessary to consider the relationship between the data stored in each database and the database. Therefore, distributed database technology based on distributed storage has emerged, which divides a single database into multiple shards and stores them on multiple nodes.

[0003] However, due to the different scales of databases, in other words, the databases used by tenants who rent cloud servers will be different according to the needs of the tenants. Therefore, in the distributed database system, although the database with a large data scale has been divided into multiple small databases by sharding and stored on multiple nodes, the tenant's data usage needs may cause a certain shard to generate a large amount of data write demand within a certain period of time, and such data write demand may also cause a surge in the data write demand of related shards on other nodes in the database. Therefore, in the daily use of distributed databases, a large number of data write operations will be concentrated on a certain node or several nodes in a short period of time, thus generating the so-called "hotspot". The generation of such hotspots will cause a sudden increase in the load of a single node or several nodes of the related shards that bear the shard, resulting in a decrease in the storage performance of the node, thereby affecting the user experience. Therefore, when managing shards, the provider of the distributed database hopes to ensure the uniform distribution of data operations between nodes, try to avoid the generation of hotspots, or eliminate hotspots as soon as possible when they are generated. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and device, a data system, a computing node, an electronic device, and a computer-readable storage medium to solve the defect that hot spots cannot be eliminated in the prior art.

[0005] To achieve the above objectives, the present application provides a data processing method, including:

[0006] receiving a data write request, wherein the data write request includes first data information indicating data to be written;

[0007] Determine first partition identification information of a first data storage partition to which the to-be-written data belongs according to the first data information;

[0008] Determine at least one second partition identification information of the second data storage partition into which the to-be-written data is written according to the write routing table and the first partition identification information, wherein the write routing table is related to the running state of the data storage node corresponding to the data storage partition;

[0009] The to-be-written data is written into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0010] The present application also provides a data processing device, which includes:

[0011] a control node, configured to receive a data write request, and to determine first partition identification information of a first data storage partition to which the to-be-written data belongs according to the first data information, wherein the data write request includes first data information indicating the to-be-written data;

[0012] A plurality of data storage nodes, each of the plurality of data storage nodes is respectively used to store data of a corresponding data storage partition;

[0013] a routing module, configured to determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to a write routing table and the first partition identification information, wherein the write routing table is related to an operating state of a data storage node corresponding to the data storage partition;

[0014] The control node is further configured to write the to-be-written data into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0015] The embodiment of the present application also provides a data system, including: a client node and a data processing device, wherein:

[0016] The client node is used to send a data writing request to the data processing device according to a user operation, the data writing request includes first data information indicating the data to be written, and the data processing device includes:

[0017] a control node, configured to receive a data write request from the client node, and to determine first partition identification information of a first data storage partition to which the to-be-written data belongs according to the first data information;

[0018] A plurality of data storage nodes, each of the plurality of data storage nodes is respectively used to store data of a corresponding data storage partition;

[0019] a routing module, configured to determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to a write routing table and the first partition identification information, wherein the write routing table is related to an operating state of a data storage node corresponding to the data storage partition;

[0020] The control node is further configured to write the to-be-written data into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0021] The present application also provides a data processing device, including:

[0022] a control node, configured to receive a data write request, wherein the data write request includes first data information indicating data to be written, and to determine first partition identification information of a first data storage partition to which the data to be written belongs according to the first data information;

[0023] a routing module, configured to determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to a write routing table and the first partition identification information, wherein the write routing table is related to an operating state of a data storage node corresponding to the data storage partition;

[0024] The control node is further configured to write the to-be-written data into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0025] The present application also provides a computing node, including:

[0026] A state calculation unit, used to determine the operation state of the node;

[0027] a communication unit, configured to send the operating state determined by the state calculation unit,

[0028] Wherein, the communication unit is further used to receive a data processing task determined according to the sent operating status.

[0029] The present application also provides an electronic device, including:

[0030] Memory, used to store programs;

[0031] The processor is used to run the program stored in the memory, and when the program is run, the data processing method provided in the embodiment of the present application is executed.

[0032] An embodiment of the present application also provides a computer-readable storage medium on which a computer program executable by a processor is stored, wherein the program, when executed by the processor, implements the data processing method provided in the embodiment of the present application.

[0033] The data processing method and device, data system, computing node, electronic device, and computer-readable storage medium provided in the embodiments of the present application, after receiving a data write request, first calculate the shard that processes the request, and then perform hot spot check and adjustment on the calculated shard through a write routing table that records the operating status of the processing node, thereby determining the shard that ultimately processes the data write request. Therefore, the data storage scheme of the embodiments of the present application can use a write routing table that reflects the operating status of the processing node to adjust the allocation of shards calculated based on the data involved in the data write request, thereby achieving adaptive adjustment of the shards and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

[0034] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0036] Figure 1a A schematic diagram of a data processing method in the prior art of the art;

[0037] Figure 1b A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of an embodiment of the data processing method provided by the present application;

[0039] Figure 3 A flowchart of another embodiment of the data processing method provided by the present application;

[0040] Figure 4 A schematic diagram of the structure of an embodiment of a data processing device provided by the present application;

[0041] Figure 5 A schematic diagram of the structure of an electronic device embodiment provided in this application. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] Embodiment 1

[0044] The solution provided in the embodiments of the present application can be applied to any data processing system with data processing capabilities. Figure 1b A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application, Figure 1b The scenario shown is only one example of a scenario to which the technical solution of the present application can be applied.

[0045] With the development of distributed database technology, more and more cloud server providers provide distributed database services for tenants. However, due to the different scales of tenants, or the different usage of tenants renting distributed database services, there is a problem of different data read and write operations of the database. In other words, as a distributed database distributed on multiple nodes, especially a large-scale database, more nodes are needed to store its multiple shards. Usually, when the management node of the distributed database receives the data to be stored, it calculates the shard to which it belongs through a hash algorithm, thereby guiding it to the corresponding node for storage. For example, a large distributed database can be sharded according to time, so that the data of one day is aggregated by user ID to be stored on one DN (node), and the data of the next day can be stored on another DN (node), so that when the user queries the data of a certain day, the request can be directly sent to the node without broadcasting to all nodes, thereby ensuring the efficiency of the query. On the other hand, as the use of databases becomes more and more widespread, especially when applied to cloud computing, tenants who rent distributed databases rent multiple nodes to deploy cloud computing. In this case, tenants will create and delete instances according to user needs, and such operations may cause a large load on the data write traffic of the shard nodes when they occur frequently. Therefore, the current technology can only pre-set the tenant category, such as setting large tenants, and / or pre-estimate the tenant traffic to pre-determine the response strategy.

[0046] For example, Figure 1a It is a schematic diagram of a data processing method in the prior art in this field, such as Figure 1aAs shown in the prior art, tenants can be divided into large tenants and small tenants in advance according to the size of the tenants or the amount of data or the size of the database to be provided, for example Figure 1a The large merchant whitelist shown in , and different shard allocation strategies are set accordingly. For example, data with IDs belonging to small tenants are directed to Shard routes for small tenants according to the tenant ID field, and data with IDs belonging to large tenants are directed to Shard routes for large tenants. The data of small tenants can be hashed by tenant ID to determine its shard and sent to the corresponding node for storage. For the database of large tenants, due to the large scale of the database, for example, each node DN may carry the data of 10-1000 tenants. Therefore, the shards can be stored on one DN on a daily basis, so that the data of multiple tenants on the same day are stored on the same DN, thereby achieving the uniformity of shards.

[0047] However, the above scheme obviously requires the management of the distributed database to understand the tenants' situation in advance, pre-set groups and give different management strategies, so that when the management node performs shard calculations on the received data, the data can be determined to the corresponding shard node. Therefore, in the cloud computing scenario, the instances managed by the cloud database have diverse characteristics. Therefore, in the scenario where tenants purchase distributed databases for cloud computing, due to the frequent instance migration changes caused by the tenants' addition and deletion operations on the instances, the above-mentioned prior art method of manually setting the list in advance and giving corresponding management strategies cannot adapt to such a situation. In particular, the above-mentioned pre-manual setting method cannot solve the problem of timely elimination of shard hotspots caused by the frequent operations of tenants, which will cause data writing bottlenecks and affect the tenants' user experience.

[0048] Specifically, for example, as in the present application Figure 1bAs shown in, in the use of the distributed cloud database, the tenant can send data write requests 1-3 to the control node (RC) of the distributed cloud database, wherein the data write request can be a data write for the database provided by the tenant, for example, data write requests 1 and 3 are data write requests for large tenants, and data write request 2 is a data write request for small tenants. In this process, the tenant's data write request will be sent to the corresponding control node RC by the distributed cloud database according to the load balancing SLB strategy, such as the identification strategy of large and small tenants as described above. For example, the control node RC1 can receive data write requests 1 and 3 identified as data write requests for large tenants, and the control node RC2 can receive data write request 2 identified as data write request for small tenants. After receiving the tenant's data write request, the control nodes RC1 and RC2 can perform a hash algorithm calculation to determine the data shard (Shard) for which the received data write request is directed, or the data shard to which the data to be written belongs. For example, the control node RC1 can determine that the data shard targeted by the data write request 1 is shard1, the data shard targeted by the data write request 2 is shard2, and the data shard targeted by the data write request 3 is shard3. Therefore, the control nodes RC1 and RC2 can send the data write request to the corresponding computing nodes in the computing nodes 1-3 for processing to complete the writing of the data. In a specific implementation, for example, the computing node RC1 can first input the corresponding data 1 into a table queue (TableQueue) according to the data write request 1, and then send the data in the table queue to the corresponding table engine (TableEngine) for processing. Specifically, for example, the data shards in each table can correspond to a table engine, and the corresponding data can be first input into the real-time engine (RealTime Engine) when writing to the table engine, and the data in the fact engine can be sorted and indexed through periodic processing to finally write to the corresponding partition data engine (PartitionEngine). Therefore, as described above, in the prior art, the data write request of the tenant is guided to the corresponding control node for processing according to the tenant's situation by pre-setting the SLB strategy, thereby achieving load balancing of data processing.

[0049] However, as mentioned above, since the SLB based on which the control node to which the data write request is assigned in the above scheme is determined is pre-set, it cannot be updated or adjusted in real time according to the actual situation. Therefore, in the current cloud computing big data scenario, the amount of data processed by the computing node will change in real time due to the actual operation of the tenant, so it is easy to generate hot spots. For example, in the above case, data write requests 1 and 3 are data write requests issued by large tenants, and if the amount of data involved in data write request 1 is large, even the change of data involved in data write request 1 will cause the change of data in the relevant shards. For example, the shard targeted by data write request 1 corresponds to computing node 1, but the change of data involved causes the change of data in the shard corresponding to computing node 4. Therefore, in this case, since the data involved in data write request 1 is large, it not only causes computing node 1 to process a large amount of data, but also causes computing node 4 to process a large amount of data, that is, computing nodes 1 and 4 both become hot spots at this time.

[0050] In such a case, if the policy cannot be adjusted in time, the hotspot will eventually become a bottleneck for data writing. Figure 1b As shown in , in the embodiment of the present application, the tenant's data write requests 1-3 can still be assigned to the corresponding control node RC through SLB, and the control nodes RC1-3 can also perform, for example, hash algorithm calculation to determine the data shard for each data write request 1-3. Here, different from the prior art, after the data shard is determined, Figure 1b As shown in the figure, the control nodes RC1-3 can determine the computing node worker to which the data write request 1-3 should be sent based on the data segmentation given by the policy adjustment module in the embodiment of the present application, rather than determining the computing node to which the data write request should be sent based only on the hash algorithm as in the prior art.

[0051] For example, in Figure 1bAs shown in , the distributed database of the embodiment of the present application can send data write requests 1-3 to the control node (RC) of the distributed cloud database, wherein the data write request can be a data write for the database provided by the tenant, for example, data write requests 1 and 3 are data write requests for large tenants, and data write request 2 is a data write request for small tenants. In this process, according to the preset load balancing SLB strategy, data write requests 1-3 can be sent to the corresponding control node RC1 or 2. For example, the control node RC1 can receive data write requests 1 and 3 identified as data write requests for large tenants, and the control node RC2 can receive data write request 2 identified as data write request for small tenants. After receiving the data write request of the tenant, the control nodes RC1 and RC2 can perform a hash algorithm calculation to determine the data shard (Shard) for which the received data write request is directed, or the data shard shard to which the data to be written belongs. For example, the control node RC1 can determine that the data shard for which the data write request 1 is directed is shard1, the data shard for which the data write request 2 is directed is shard2, and the data shard for which the data write request 3 is directed is shard3. At this time, unlike the prior art, the distributed database system in the embodiment of the present application additionally provides a routing module to determine to which computing node among computing nodes 1-3 the control nodes RC1 and RC2 send the data write request for processing. Specifically, the control node RC1 can send the received data write request 1-3 to a cache unit such as a bucket according to the write routing table in the routing module before sending it to the computing node, and then send the data in the bucket to the corresponding computing node according to the computing node specified in the write routing table. In the embodiment of the present application, the write routing table can be updated based on the detection of the running status of the computing node. For example, the running status of the computing node can be evaluated according to a pre-set node status indicator, and each node or the shard corresponding to the node can be scored according to the evaluation result, so that it can be determined whether the current node or shard has a hot spot according to the score, that is, the amount of data written is large. Therefore, in the case of Figure 1b In the embodiment of the present application shown in , the routing module can change the shards targeted by data write requests 1 and 3 according to the real-time data write situation, thereby balancing the load pressure of the computing nodes. For example, when data write request 1 causes an increase in the computing load pressure on computing node 1, and as mentioned above, it also causes an additional increase in the computing load pressure on computing node 4, therefore, the generation of hot spots can be detected at the routing module, thereby adjusting the shards for data write request 1 in the bucket cache unit, so that the data write operation will not be concentrated at computing node 1, thereby achieving load balancing and uniformity between computing nodes.

[0052] Therefore, according to the data storage scheme of the embodiment of the present application, after receiving the data write request, the shard to process the request is first calculated, and then the calculated shard is checked and adjusted for hot spots through a write routing table that records the operating status of the processing node, thereby determining the shard that ultimately processes the data write request. Therefore, the data storage scheme of the embodiment of the present application can use a write routing table that reflects the operating status of the processing node to adjust the allocation of shards calculated based on the data involved in the data write request, thereby achieving adaptive adjustment of the shards and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

[0053] The above embodiments are illustrations of the technical principles and exemplary application frameworks of the embodiments of the present application. The specific technical solutions of the embodiments of the present application are further described in detail below through multiple embodiments.

[0054] Embodiment 2

[0055] Figure 2 This is a flowchart of an embodiment of the data processing method provided by the present application. The execution subject of the method can be various terminals or server devices with data processing capabilities, or it can be a device or chip integrated on these devices. Figure 2 As shown, the data processing method includes the following steps:

[0056] S201, receiving a data write request.

[0057] In an embodiment of the present application, a tenant of a distributed cloud database may send a data write request to a node of the distributed cloud database, such as a control node RC. The data write request may include first data information indicating the data to be written. In other words, the data write request may be a data write request caused by the tenant of the distributed database operating the database. For example, when the cloud database of the present application is a sales record database of an e-commerce website, the data write request may be a data write request for updating and writing the commodity sales record and the user's purchase record in the database of the e-commerce platform as a tenant due to the user's purchase behavior, and therefore the data write request may include the commodity information of the purchased commodity and the user information of the purchased commodity, such as user ID data, or may also include the data source link information of these data, so that the storage node can obtain this information from the data source when storing. In an embodiment of the present application, the data write request may be a batch data write request. For example, in the case of data on an e-commerce platform, the data occurring within a predetermined time interval may be aggregated into a batch data write request to be sent to the control node RC for processing, thereby improving processing efficiency.

[0058] S202: Determine first partition identification information of a first data storage partition to which the data to be written belongs according to the first data information.

[0059] In an embodiment of the present application, when a data write request containing data to be written is received, in an embodiment of the present application, for example, the data shard to which the data to be written belongs, that is, the data storage partition, can be calculated based on the hash key on the control node. For example, a hash function can be calculated for the data to be written based on the data write request to determine the first partition identification information of the data storage partition to which the data to be written should be sent. For example, in the case of the above-mentioned e-commerce platform, it can be determined that the purchase record data to be written should be written to the data storage partition with the partition identification shard0 based on the commodity transaction information occurring within a day. In some embodiments, the corresponding computing node worker can be further determined based on the determined data storage partition.

[0060] S203: Determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to the write routing table and the first partition identification information.

[0061] In an embodiment of the present application, after determining the data storage partition to which the data to be written belongs in step S202, it is possible to further detect whether the data storage partition determined in step S202 is a hotspot according to the write routing table, and the other data storage partition to which the data to be written is written can be determined according to the result of the detection. In particular, the write routing table can be related to the operating state of the data storage node corresponding to the data storage partition, so that the write routing table can reflect the operating state of the data storage node. For example, the operating state of the storage node can be detected regularly, and the operating state of the storage node can be evaluated according to the result of the detection, such as determining whether the storage node is currently in a hotspot state or the degree of being in a hotspot state, so that in an embodiment of the present application, after the data storage partition to which the data to be written belongs has been determined in step S202, the data to be written will not be directly written to the calculated data storage partition, but the storage node corresponding to the calculated data storage partition is checked according to the write routing table that reflects the operating state of the storage node. When the storage node corresponding to the current data storage partition is confirmed to be a hotspot node according to the record in the write routing table, that is, when the load is heavy, the pressure of the current storage node can be reduced by allocating another node for the current data to be stored through the write routing table.

[0062] That is, in step S203, the data storage node to which the current data to be written is to be written is evaluated for status through the write routing table, so that another data storage partition, ie, the second data storage partition, can be allocated for the current data to be written.

[0063] S204: Write the data to be written into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0064] As described above, after the second data storage partition is allocated to the current data to be written in step S203, the data to be written can be guided to the second data storage partition confirmed in step S203 for storage according to the allocation result in the write routing table. In this way, when the data writing to be performed by the tenant causes a higher load on the data node, the data can be guided in time through the write routing table, thereby reducing the pressure on the data storage node and achieving the effect of eliminating hot spots.

[0065] Therefore, according to the data storage solution of the embodiment of the present application, after receiving the data write request, the data storage partition and the corresponding data storage node for processing the request are first calculated, and then the calculated data storage node is checked and adjusted for hot spots through a write routing table that reflects the operating status of the storage node, thereby determining the data storage node to which the data is finally written. Therefore, the data storage solution of the embodiment of the present application can use a write routing table that reflects the operating status of the processing node to adjust the allocation of storage nodes calculated based on the data involved in the data write request, thereby achieving adaptive adjustment of the storage node and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

[0066] Embodiment 3

[0067] Figure 3 This is a flowchart of another embodiment of the data processing method provided by the present application. The execution subject of the method can be various terminals or server devices with data processing capabilities, or it can be a device or chip integrated on these devices. Figure 3 As shown, the data processing method includes the following steps:

[0068] S301: Obtain operating status data of at least one data storage node according to a preset indicator.

[0069] In an embodiment of the present application, in order to accurately determine whether the node of the data storage partition to which the data to be written belongs is a hot spot, the operating status of each data storage node can be obtained in real time or periodically. In particular, indicators that can measure the load of the node can be pre-specified, and the corresponding operating data can be obtained based on these indicators. Of course, in an embodiment of the present application, all relevant status data of the operating node can also be directly obtained, and useful status data can be selected according to the evaluation scheme.

[0070] For example, in the embodiment of the present application, a timed task scheduling collector can be used to periodically obtain data identifying the operating status from the storage node (worker), such as load pressure data. The evaluation indicators used may include: TPS: the number of transactions that the database can process per second; L queue : The length of the table queue (Table Queue) of each table in the node; N rt_ : The number of data items stored in the real-time computing engine memory in the node; GAP raft : The gap of the three-copy sync log apply storage module.

[0071] S302: Obtain specification parameters of at least one database instance.

[0072] In the scenario where the tenant applies the distributed cloud database according to the embodiment of the present application to cloud computing, the tenant will involve various instances of the database when performing cloud computing services. Therefore, according to the embodiment of the present application, various indicators related to the instance can be further used to evaluate the operating status of the node. For example, the following indicators can be used as evaluation indicators related to the instance:

[0073] N cpu : number of CPU cores;

[0074] C mem : Memory size;

[0075] N con : Number of connections;

[0076] N iops : IOPS of the node (the number of read and write operations per second, Input / Output Operations PerSecond);

[0077] N rc : Control the number of nodes;

[0078] N worker : Number of storage nodes;

[0079] n: total number of shards.

[0080] S303: Calculate a status score of each of the at least one data storage node according to the running status.

[0081] After obtaining various operation status data according to the evaluation index in step S301 and obtaining the specification parameters of the database instance in step S302, the status score of the storage node can be calculated according to the operation status data and the specification parameters of the database instance in step S303. For example, the status score of the storage node can be calculated according to the data of the node index (for example, TPS, L queue 、N rt_rowand GAP raft ) to calculate the state score. For example, in actual use, the generation of hot spots occurs gradually. For example, as TPS increases, the number of transactions processed gradually increases and accumulates, causing rt_rows to increase. When rt_rows increases to a certain extent, the node's Lqueue, that is, the table queue, will increase and accumulate data. Finally, when the queue data is full, it will cause GAP. raft In other words, the above four indicators reflect different hotspot situations from four different dimensions. Therefore, in the embodiment of the present application, one of the indicators can be selected in step S303 to calculate the status score, or multiple indicators can be used to comprehensively calculate the status score.

[0082] In particular, when using multiple indicators, such as using all four indicator data to calculate the status score, different weight coefficients can be given to the indicator data according to the impact of the indicator on the hotspot, and the weight sum of these indicator data is calculated as the status score of the node.

[0083] For example, Z shard =αTPS+βN rt_wors +γL queue +δGAP raft

[0084] Among them, α, β, γ and δ are weight coefficients of each indicator data, and as described above, they may have the relationship: α<β<γ<δ.

[0085] In addition, when the database specification parameters are obtained in step S302, the specification score of the database can be further calculated based on the various specification parameters of the database. Since the specification indicators of the cloud database can further reflect the properties of the database to which the data stored on the storage node belongs, they also have an auxiliary and supplementary role in determining the status of the storage node. Therefore, the score of the database on the storage node can be calculated by combining one or all of the above database indicators.

[0086] For example, Z db =F(N cpu ,C mem ,N con ,N iops ,N rc ,N worker ,n).

[0087] When the database score is calculated, the final status score of the node can be calculated by combining the score calculated based on the indicator data of the node's operating status and the database score. For example:

[0088]

[0089] According to the above calculation method, when the database specification is higher, it can be considered that the tolerance of the node becoming a hotspot is higher, so the node score is lower, that is, it is not easy to become a hotspot. On the contrary, if the specification of the database stored on the node is lower, it can be considered that the database has a lower tolerance for the node load, so the node score is higher, that is, it is easy to become a hotspot.

[0090] In addition, when the data storage system is in use, since the state scores of the nodes have been accumulated during use, the state scores of the nodes can be calculated using a mathematical model trained with existing data. For example, after the state data is calculated, the state score Z can be calculated using the following formula.

[0091]

[0092] Among them, λZ score is an adjustment value of the state score calculated by the mathematical model, and therefore, the data processing method according to the embodiment of the present application may further include:

[0093] S304, obtaining a training data set including stored storage data and associated data storage instructions.

[0094] S305: Use the acquired training data set to train the preset data model to obtain an updated data model.

[0095] For example, in the embodiment of the present application, the stored storage data and the related data writing instructions can be periodically obtained as training data, because the stored storage data is stored after being processed based on the data writing instructions, and therefore, it can reflect the actual processing situation of the data processing method according to the embodiment of the present application. And it can also include a training data set of positive samples that eliminate hot spots and a training data set of negative samples that fail to eliminate hot spots or cause or aggravate hot spots.

[0096] For example, in the embodiment of the present application, when using training data to calculate the above formula score When , you can use the density-based dbscan clustering method to distinguish hot spots from non-hot spots, and then use the Pearson correlation coefficient to calculate the correlation between the cluster center of the non-hot spot and each hot spot shard. score .

[0097] For example, when the sample set is D = (x 1 ,x 2 ,......,x n), where n is the number of storage partition shards, Xi is the vector of indicators of the operating status of the storage partition (including TPS, N rt_wors , L queue , GAP raft In the embodiment of the present application, the DBSCAN density may be defined as follows.

[0098] ∈-neighborhood: For xj∈D, its ∈-neighborhood contains the subsample set in the sample set D whose distance from xj is not greater than ∈, that is, N∈(xj)={xi∈D|dist(xi,xj)≤∈}N∈(xj)={xi∈D|distance(xi,xj)≤∈}

[0099] Core object: For any sample xj∈D, if its ∈∈-neighborhood corresponding to N∈(xj) contains at least MinPts samples, that is, if |N∈(xj)|≥MinPts, then xj is a core object.

[0100] Density direct access: If xi is in the ∈-neighborhood of xj and xj is a core object, then xi is said to be density-directly accessible from xj.

[0101] Density reachable: For xi and xj, if there exists a sample sequence p1, p2, ..., pT, satisfying p1 = xi, pT = xj, and pt+1 is directly accessible from pt, then xj is said to be density reachable from xi. In other words, density reachability satisfies transitivity. At this time, the transitive samples p1, p2, ..., pT-1 in the sequence are all core objects, because only core objects can make other samples density reachable. Note that density reachability does not satisfy symmetry either, which can be derived from the asymmetry of density direct accessibility.

[0102] Density connection: For xi and xj, if there exists a core object sample xk such that both xi and xj are density-reachable by xk, then xi and xj are said to be density-connected.

[0103] In an embodiment of the present application, for example, when the input is: a sample set D = (x1, x2, ..., xm) of training data, a neighborhood parameter (∈, MinPts), and a sample distance measurement method, the output can be: cluster division C, that is, whether the data storage node is a hotspot or a non-hotspot.

[0104] 1) Initialize the core object collection Initialize the number of clusters k = 0, initialize the unvisited sample set Γ = D, and divide the clusters

[0105] 2) For j = 1, 2, ... m, find all core objects as follows:

[0106] a) Find the ∈-neighborhood subsample set N∈(xj) of sample xj by using distance measurement;

[0107] b) If the number of samples in the sub-sample set satisfies |N∈(xj)|≥MinPts, add sample xj to the core object sample set: Ω=Ω∪{xj};

[0108] 3) If the core object collection Then, in the core object set Ω, a core object o is randomly selected, the current cluster core object queue Ωcur={o} is initialized, the category sequence number k=k+1, the current cluster sample set Ck={o} is initialized, and the unvisited sample set Γ=Γ-{o} is updated;

[0109] 4) If the current cluster core object queue Then the current cluster CkCk is generated, the cluster partition C={C1,C2,...,Ck} is updated, the core object set Ω=Ω-Ck is updated, and the core object set Ω is determined to be equal to Otherwise, update the core object set Ω=Ω-Ck.

[0110] 5) Take out a core object o′ from the current cluster core object queue Ωcur, find all ∈-neighborhood subsample sets N∈(o′) through the neighborhood distance threshold ∈, let Δ=N∈(o′)∩Γ update the current cluster sample set Ck=Ck∪Δ, update the unvisited sample set Γ=Γ-Δ, update Ωcur=Ωcur∪(Δ∩Ω)-o′, and then return to execute the previous step 4).

[0111] Therefore, in the embodiment of the present application, according to the above steps, the output result can be obtained as: cluster partition C = {C1, C2, ..., Ck} and center point o. As mentioned above, the present application needs to distinguish between hot spots and non-hot spots, so k = 2, that is, cluster partition C = {C1, C2}.

[0112] Randomly calculate the Pearson correlation coefficient between each hotspot shard cluster and the center point o.

[0113]

[0114] Therefore, through the calculation of the mathematical model based on the training data set, Z can be obtained. score , to assist in calculating the state score in step S303.

[0115] In addition, in an embodiment of the present application, the above-mentioned trained data model (for example, the DBSCAN clustering model and the Pearson calculation model) can be stored for use in step S303.

[0116] S306: Determine a data storage node that can be used to store the data to be stored according to the status score of each data storage node.

[0117] After the state score of the storage node is calculated and obtained by step S303, it can be determined whether the storage node is a hotspot according to the storage node. For example, the state score of the storage node can be compared with a predetermined threshold, and the node whose state score is greater than the preset threshold is determined to be in a hotspot state, and then the final storage partition can be allocated for the data to be written, for example, by allocating more write routes to add other storage partitions to the storage partition that is detected to be in a hotspot state through the above steps, that is, a bottleneck occurs. For example, the storage partition with the least number of stock data in all current storage partitions can be used as a storage partition for eliminating hotspots. In particular, in an embodiment of the present application, since the operations of tenants of a distributed database may vary greatly, it is likely that multiple hotspot states will occur within a period of time. Therefore, the number of allocated data storage partitions can also be adjusted according to the number of hotspot eliminations. For example, the storage partitions can be sorted by the number of stock data, and 2^n partitions can be selected from the partition with the least number of stock data as storage partitions for eliminating hotspots. In this way, the history of hotspot elimination can be further considered to allocate more storage partitions to nodes that have appeared in hotspot status multiple times within a period of time to share the pressure, thereby ensuring the realization of the hotspot elimination effect.

[0118] In addition, in the embodiment of the present application, data associated with the data to be stored that causes the hotspot state of the current storage node may be further considered, such as the source data of the current data to be stored, so as to divert data from the source, thereby making it easier to eliminate the generation of hotspots. For example, the data processing method of the present application may further include:

[0119] S307: Obtain the original data flow of the first data storage partition identified by the first partition identification information.

[0120] S308, determining, according to the data information included in the original data flow, the first source data storage partition to which the source data causing the original data flow belongs;

[0121] S309, determining second source partition identification information of a second source data storage partition into which the source data is written according to the write routing table and first source partition identification information of the first source data storage partition;

[0122] S310: Writing source data into a second source data storage partition identified by the second source partition identification information according to the second source partition identification information.

[0123] As described above, in some cases, the load of the storage node, that is, the sharp increase in the amount of written data, may be caused by the sharp increase in the amount of other written data. Therefore, in the embodiment of the present application, the original data flow can be detected based on the current data to be written, and then the data of the original data flow can be directed to other storage partitions through the write routing table, thereby reducing the data write flow of the hot node from the source.

[0124] For example, the total traffic of a single storage partition is equal to the sum of the traffic of all storage partitions corresponding to the written data, that is, Represents shard j Incoming shard i of flow, so

[0125]

[0126]

[0127] in, can represent the total traffic of a single storage partition, n can represent the total number of storage partitions, and can represent the raw traffic of a single storage partition, while It can represent the traffic of a single tenant, m represents the total number of tenants, and F total Indicates the total flow.

[0128] In addition, storage partitions and tenants can have a corresponding relationship: shard i =HASH(ins x )mod n

[0129] therefore,

[0130] In the embodiment of the present application, the purpose of calculating the load pressure of the node (worker) is to detect hot spots, so the data partition on the node (worker) can be used to write the real flow To evaluate, that is, the traffic after the traffic is balanced by the routing table. And in this step, you can refer to the original traffic of the storage partition where the data is written That is, after the control node (RC) is calculated by the hash function, the traffic before the hot spot fluctuation is eliminated according to the routing table. Detect the source of hot traffic and determine which partitions have high traffic and cause the traffic flowing into the storage partition.

[0131] Therefore, according to the data storage solution of the embodiment of the present application, after receiving the data write request, the data storage partition and the corresponding data storage node for processing the request are first calculated, and then the calculated data storage node is checked and adjusted for hot spots through a write routing table that reflects the operating status of the storage node, thereby determining the data storage node to which the data is finally written. Therefore, the data storage solution of the embodiment of the present application can use a write routing table that reflects the operating status of the processing node to adjust the allocation of storage nodes calculated based on the data involved in the data write request, thereby achieving adaptive adjustment of the storage node and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

[0132] Embodiment 4

[0133] Figure 4 The schematic diagram of the structure of the data processing device embodiment provided in the present application can be used to perform the following steps: Figure 2 and Figure 3 The method steps shown. Figure 4 As shown, the data processing device may include: a control node 41 , a plurality of data storage nodes 42 and a routing module 43 .

[0134] The control node 41 may be used to receive a data write request, and to determine, according to the first data information, first partition identification information of a first data storage partition to which the data to be written belongs.

[0135] In an embodiment of the present application, the control node RC of the distributed cloud database may receive a data write request from a tenant of the distributed cloud database. The data write request may include first data information indicating the data to be written. In other words, the data write request may be a data write request caused by the tenant of the distributed database operating the database. For example, when the cloud database of the present application is a sales record database of an e-commerce website, the data write request may be a data write request for updating and writing the commodity sales record and the user's purchase record in the database of the e-commerce platform as a tenant due to the user's purchase behavior, and therefore the data write request may include the commodity information of the purchased commodity and the user information of the purchased commodity, such as user ID data, or may also include the data source link information of these data, so that the storage node can obtain this information from the data source when storing. In an embodiment of the present application, the data write request may be a batch data write request. For example, in the case of data on an e-commerce platform, the data occurring within a predetermined time interval may be aggregated into a batch data write request to be sent to the control node RC for processing, thereby improving processing efficiency.

[0136] Each of the multiple data storage nodes 42 can be used to store data of the data storage partition corresponding thereto. In an embodiment of the present application, when the control node 41 receives a data write request containing data to be written, in an embodiment of the present application, for example, the control node 41 can calculate the data shard (i.e., the data storage partition) to which the data to be written belongs according to the hash key. For example, a hash function can be calculated for the data to be written according to the data write request, so as to determine the first partition identification information of the data storage partition to which the data to be written should be sent. For example, in the case of the above-mentioned e-commerce platform, it can be determined that the purchase record data to be written should be written to the data storage partition with the partition identification shard0 according to the commodity transaction information occurring within a day. In some embodiments, the corresponding storage node 42 can be further determined according to the determined data storage partition.

[0137] The routing module 43 can be used to determine at least one second partition identification information of the second data storage partition to which the data to be written is written according to the write routing table and the first partition identification information. The routing module 43 can detect whether the data storage partition determined by the control node 41 is a hotspot according to the write routing table, and can determine the data storage partition on the other data storage node 42 to which the data to be written is written according to the detection result. In particular, the write routing table can be related to the running state of the data storage node 42 corresponding to the data storage partition, so that the write routing table can reflect the running state of the data storage node 42. For example, the running state of the storage node 42 can be detected regularly, and the running state of the storage node 42 can be evaluated according to the detection result, for example, to determine whether the storage node 42 is currently in a hotspot state or the degree of being in a hotspot state, so that in the embodiment of the present application, after the control node 41 has determined the data storage partition to which the data to be written belongs, the data to be written will not be directly written to the calculated data storage partition, that is, the data storage node 42, but the routing module 43 checks the storage node 42 corresponding to the calculated data storage partition according to the write routing table that reflects the running state of the storage node 42. When the routing module 43 confirms that the storage node 42 corresponding to the current data storage partition is a hot node, that is, the load is heavy, the routing table can be written to allocate another node 42 for the current data to be stored to reduce the pressure on the current storage node 42.

[0138] In an embodiment of the present application, the control node 41 may be further configured to write the to-be-written data into the second data storage partition identified by the second partition identification information according to the second partition identification information.

[0139] In addition, the routing module 43 may further include: a first acquiring unit 431 , a calculating unit 432 , and a determining unit 433 .

[0140] The first acquisition unit 431 may be configured to acquire the operating status of at least one data storage node according to a preset indicator.

[0141] The calculation unit 432 may be configured to calculate a status score of each of the at least one data storage node according to the operating status.

[0142] The determination unit 433 may be configured to determine a data storage node that can be used to store the data to be stored according to the status score of each data storage node.

[0143] In the embodiment of the present application, in order to accurately determine whether the node of the data storage partition to which the data to be written belongs is a hot spot, the first acquisition unit 431 can obtain the operating status of each data storage node in real time or periodically. In particular, indicators that can measure the load of the node can be pre-specified, and the corresponding operating data can be obtained based on these indicators. Of course, in the embodiment of the present application, the first acquisition unit 431 can also directly obtain all relevant status data of the operating node, and then select useful status data according to the evaluation scheme.

[0144] After the first acquisition unit 431 acquires the status data, the calculation unit 432 can calculate the status score of the storage node according to the running status data and the specification parameters of the database instance. For example, the status score of the storage node can be calculated according to the data of the node index (for example, TPS, L queue 、N rt_rows and GAP raft ) to calculate the state score. For example, in actual use, the generation of hot spots occurs gradually. For example, as TPS increases, the number of transactions processed gradually increases and accumulates, causing rt_rows to increase. When rt_rows increases to a certain extent, the node's Lqueue, that is, the table queue, will increase and accumulate data. Finally, when the queue data is full, it will cause GAP. raft In other words, the above four indicators reflect different hotspot situations from four different dimensions. Therefore, in the embodiment of the present application, the calculation unit 432 can select at least one of the indicator data obtained by the first acquisition unit 431 to calculate the state score, or can use multiple indicators to comprehensively calculate the state score.

[0145] In particular, when using multiple indicators, such as using all four indicator data to calculate the status score, the calculation unit 432 can give different weight coefficients to the indicator data according to the impact of the indicator on the hotspot, and thereby calculate the weight sum of these indicator data as the status score of the node.

[0146] In addition, in the embodiment of the present application, the first acquisition unit 431 can be further used to obtain the specification parameters of the database instance, and the calculation unit 432 can be further used to calculate the status score of each of the at least one data storage node according to the operating status data and the specification parameters. Since the specification indicators of the cloud database can further reflect the properties of the database to which the data stored on the storage node belongs, it also has an auxiliary and supplementary role in determining the status of the storage node. Therefore, the calculation unit 432 can combine one or all of the indicators of the above database to calculate the score of the database on the storage node.

[0147] When the data storage system is in use, since the state scores of the nodes have been accumulated during use, the calculation unit 431 can be further used to calculate the state scores of the nodes using a mathematical model trained with existing data. For example, after calculating the state data, the state score Z can be calculated using the following formula.

[0148]

[0149] Among them, λZ score It is an adjustment value of the state score calculated by the mathematical model, and therefore, the routing module 43 may further include: a second acquisition unit 434 and a training unit 435 .

[0150] The second acquisition unit 434 may be configured to acquire a training data set including stored storage data and associated data storage instructions.

[0151] The training unit 435 may be configured to train a preset data model using the acquired training data set to obtain an updated data model.

[0152] Therefore, the calculation unit 432 may be further configured to calculate a state score according to the operating state using the updated data model.

[0153] In addition, the first acquisition unit 431 can be further used to acquire the original data flow of the first data storage partition identified by the first partition identification information, and the determination unit 433 can be further used to determine the first source data storage partition to which the source data causing the original data flow belongs based on the data information contained in the original data flow; determine the second source partition identification information of the second source data storage partition to which the source data is written based on the write routing table and the first source partition identification information of the first source data storage partition. As described above, in some cases, the load of the storage node, that is, the sharp increase in the amount of written data may be caused by a sharp increase in the amount of other written data. Therefore, in an embodiment of the present application, the original data flow can be detected based on the current data to be written, and then the data of the original data flow can be directed to other storage partitions through the write routing table, thereby reducing the data write flow of the hot node from the source.

[0154] In this case, the control node 41 may be further configured to write the source data to the second source data storage partition identified by the second source partition identification information according to the second source partition identification information.

[0155] Therefore, according to the data storage solution of the embodiment of the present application, after receiving the data write request, the data storage partition and the corresponding data storage node for processing the request are first calculated, and then the calculated data storage node is checked and adjusted for hot spots through a write routing table that reflects the operating status of the storage node, thereby determining the data storage node to which the data is finally written. Therefore, the data storage solution of the embodiment of the present application can use a write routing table that reflects the operating status of the processing node to adjust the allocation of storage nodes calculated based on the data involved in the data write request, thereby achieving adaptive adjustment of the storage node and eliminating hot spots, avoiding the impact of node performance degradation on user experience.

[0156] Embodiment 5

[0157] The internal functions and structure of the data processing device are described above. The device can be implemented as an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device embodiment provided by this application. Figure 5 As shown, the electronic device includes a memory 51 and a processor 52 .

[0158] The memory 51 is used to store programs. In addition to the above programs, the memory 51 can also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0159] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0160] The processor 52 is not limited to a central processing unit (CPU), but may also be a processing chip such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an embedded neural network processor (NPU) or an artificial intelligence (AI) chip. The processor 52 is coupled to the memory 51 and executes the program stored in the memory 51. When the program is running, the data processing methods of the above-mentioned embodiments 2 and 3 are executed.

[0161] Further, if Figure 5 As shown, the electronic device may also include: a communication component 53, a power component 54, an audio component 55, a display 56 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 Components shown.

[0162] The communication component 53 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 53 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 53 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0163] The power supply component 54 provides power to various components of the electronic device. The power supply component 54 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0164] The audio component 55 is configured to output and / or input audio signals. For example, the audio component 75 includes a microphone (MIC), and when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 51 or sent via the communication component 53. In some embodiments, the audio component 55 also includes a speaker for outputting audio signals.

[0165] The display 56 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0166] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, comprising: receiving a data write request, wherein the data write request includes first data information indicating data to be written; Determine first partition identification information of a first data storage partition to which the to-be-written data belongs according to the first data information; Obtaining the original data flow of the first data storage partition identified by the first partition identification information; The original data flow is the flow after the control node is calculated by the hash function and before hot spot fluctuations are eliminated according to the routing table; Determine, according to the data information included in the original data flow, the first source data storage partition to which the source data causing the original data flow belongs; Determine, according to the write routing table and the first source partition identification information of the first source data storage partition, the second source partition identification information of the second source data storage partition into which the source data is written; wherein the write routing table is related to the running state of the data storage node corresponding to the data storage partition; The source data is written to a second source data storage partition identified by the second source partition identification information according to the second source partition identification information.

2. The data processing method according to claim 1, wherein: The data processing method further comprises: Obtaining operating status data of at least one data storage node according to preset indicators; calculating a status score for each of the at least one data storage node according to the operating status data; A data storage node that can be used to store the data to be stored is determined according to the status score of each data storage node.

3. The data processing method according to claim 2, wherein: The data processing method further comprises: Get the database instance's specifications, and Calculating the status score of each of the at least one data storage node according to the operating status data comprises: A status score of each of the at least one data storage node is calculated according to the operating status data and the specification parameter.

4. The data processing method according to claim 2, wherein: Calculating the status score of each of the at least one data storage node according to the operating status includes: The status score is calculated according to the operating status using a preset data model.

5. The data processing method according to claim 4, wherein: The data processing method further comprises: Acquiring a training data set including stored storage data and associated data storage instructions; The preset data model is trained using the acquired training data set to obtain an updated data model, and Calculating the status score of each of the at least one data storage node according to the operating status includes: The state score is calculated according to the operating state using the updated data model.

6. The data processing method according to claim 1, wherein: The data processing method further comprises: Determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to the write routing table and the first partition identification information; The to-be-written data is written into the second data storage partition identified by the second partition identification information according to the second partition identification information.

7. A data processing device, comprising: a control node, configured to receive a data write request, and to determine first partition identification information of a first data storage partition to which the data to be written belongs according to first data information, wherein the data write request includes the first data information indicating the data to be written; A plurality of data storage nodes, each of the plurality of data storage nodes is respectively used to store data of a corresponding data storage partition; An original data flow module, used for acquiring the original data flow of the first data storage partition identified by the first partition identification information; The original data flow is the flow after the control node is calculated by the hash function and before hot spot fluctuations are eliminated according to the routing table; A routing module, configured to determine, based on data information included in the original data traffic, a first source data storage partition to which the source data causing the original data traffic belongs; Determine second source partition identification information of a second source data storage partition into which the source data is written based on a write routing table and first source partition identification information of the first source data storage partition; wherein the write routing table is related to an operating state of a data storage node corresponding to the data storage partition; and write the source data to a second source data storage partition identified by the second source partition identification information based on the second source partition identification information.

8. The data processing device according to claim 7, wherein: The routing module further comprises: A first acquisition unit, used to acquire the operating status of at least one data storage node according to a preset indicator; a calculation unit, configured to calculate a status score of each of the at least one data storage node according to the operating status; The determination unit is used to determine a data storage node that can be used to store the data to be stored according to the status score of each data storage node.

9. The data processing device according to claim 8, wherein: The first acquisition unit is further used to: acquire specification parameters of the database instance, and The computing unit is further configured to: A status score of each of the at least one data storage node is calculated according to the operating status data and the specification parameter.

10. The data processing device according to claim 8, wherein: The calculation unit is further used to: calculate the state score according to the operating state using a preset data model.

11. The data processing device according to claim 10, wherein: The routing module further comprises: A second acquisition unit, configured to acquire a training data set including stored storage data and associated data storage instructions; a training unit, configured to train the preset data model using the acquired training data set to obtain an updated data model, and The calculation unit is further used to: calculate the state score according to the operating state using the updated data model.

12. The data processing device according to claim 8, wherein: The first acquisition unit is further used to acquire the original data flow of the first data storage partition identified by the first partition identification information, The determining unit is used for: Determine at least one second partition identification information of a second data storage partition into which the to-be-written data is written according to the write routing table and the first partition identification information; The control node is further configured to write the to-be-written data into the second data storage partition identified by the second partition identification information according to the second partition identification information.

13. A data system comprising: Client nodes and data processing equipment, wherein: The client node is used to send a data writing request to the data processing device according to a user operation, the data writing request includes first data information indicating the data to be written, and the data processing device includes: a control node, configured to receive a data write request from the client node, and to determine first partition identification information of a first data storage partition to which the to-be-written data belongs according to the first data information; A plurality of data storage nodes, each of the plurality of data storage nodes is respectively used to store data of a corresponding data storage partition; The original data flow module is used to obtain the original data flow of the first data storage partition identified by the first partition identification information; the original data flow is the flow after the control node is calculated by the hash function and before the hot spot fluctuation is eliminated according to the routing table; A routing module, used to determine the first source data storage partition to which the source data causing the original data traffic belongs based on the data information contained in the original data traffic; determine the second source partition identification information of the second source data storage partition to which the source data is written based on a write routing table and the first source partition identification information of the first source data storage partition; wherein the write routing table is related to the operating state of the data storage node corresponding to the data storage partition; and write the source data to the second source partition identification information identified by the second source partition identification information based on the second source partition identification information.

14. A data processing device, comprising: a control node, configured to receive a data write request, the data write request comprising first data information indicating data to be written, and to determine first partition identification information of a first data storage partition to which the data to be written belongs according to the first data information; An original data flow module, used for acquiring the original data flow of the first data storage partition identified by the first partition identification information; The original data flow is the flow after the control node is calculated by the hash function and before hot spot fluctuations are eliminated according to the routing table; A routing module, configured to determine, based on data information included in the original data traffic, a first source data storage partition to which the source data causing the original data traffic belongs; Determine second source partition identification information of a second source data storage partition into which the source data is written based on a write routing table and first source partition identification information of the first source data storage partition; wherein the write routing table is related to an operating state of a data storage node corresponding to the data storage partition; and write the source data to a second source data storage partition identified by the second source partition identification information based on the second source partition identification information.

15. A computing node, comprising: A state calculation unit, used to determine the operation state of the node; a communication unit, configured to send the operating state determined by the state calculation unit, Wherein, the communication unit is further used to receive a data processing task determined according to the sent operating status as described in any one of claims 1 to 6.

16. An electronic device comprising: Memory, used to store programs; A processor is used to run the program stored in the memory, and the program executes the data processing method according to any one of claims 1 to 6 when it is run.

17. A computer-readable storage medium having stored thereon a computer program executable by a processor, wherein: When the program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images