Data processing method and apparatus, network device, and storage medium
By splitting commands in a NoSQL database and executing them across multiple shards, the data skew problem is solved, the efficiency of database sharding and the balance of data storage are improved, and fast and accurate read and write operations are ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DUXIAOMAN TECH (BEIJING) CO LTD
- Filing Date
- 2023-02-21
- Publication Date
- 2026-07-10
AI Technical Summary
In NoSQL databases, data skew can lead to data accumulation issues, especially when the amount of data in a particular database shard is too large, which affects the efficiency of database sharding.
By receiving commands from the user and splitting them according to the command content, subcommands corresponding to multiple database shards are generated and executed in these shards respectively. This avoids executing commands in a single shard and ensures that data is distributed across multiple shards. An index table is used to manage spatial address information and data volume information to facilitate command execution.
It effectively avoids data skew issues, improves the efficiency of using multiple database shards, ensures fast and accurate information reading and balanced data writing, and reduces memory usage.
Smart Images
Figure CN116431716B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communications, and more particularly to a data processing method, apparatus, network device, and storage medium. Background Technology
[0002] With the rise of internet technology, non-relational databases (NoSQL) are also constantly developing. Based on the non-relational nature of data in NoSQL, it has very high read and write capabilities, and is characterized by large data volume and high performance, which solves many problems of the internet with large data volumes.
[0003] In related technologies, database architectures are typically built using a cluster model to facilitate the execution of various commands. A typical database architecture consists of a proxy middleware (twemproxy) and multiple NoSQL database shards. In practice, when using database shards through this architecture, network devices can input commands to the proxy middleware, which then executes the commands by operating on one of the database shards based on the addresses corresponding to those commands.
[0004] However, since it often executes commands on a portion of the database shards across multiple database shards, this method of command execution may result in an excessive amount of data on one or more database shards, leading to data skew and data accumulation in individual database shards. Summary of the Invention
[0005] This application provides a data processing method, apparatus, network device, and storage medium to solve the data skew problem that is prone to occur in current NoSQL databases, and the data accumulation problem in individual database shards.
[0006] To address the aforementioned technical problems, this application adopts the following technical solution: Given a database architecture comprising X database shards and a proxy middleware, the system receives commands sent by the user, including read commands or write commands, where X is an integer greater than 1. Based on the command content, the system splits the command into Y sub-commands, each corresponding to one of the Y databases within the X database shards. The Y databases store data related to the Y sub-commands, where Y is less than or equal to X and is an integer. Finally, the system executes the Y sub-commands within the Y database shards.
[0007] The beneficial effects of this application's embodiments are as follows: When the database architecture corresponding to the database includes X database shards and a proxy middleware, a read command or a write command is received. Then, based on the command content, the command is split into Y sub-commands corresponding to Y databases in the X database shards (the Y databases store data information related to the Y sub-commands). Finally, the Y sub-commands are executed in the Y database shards. Thus, by splitting the command, the command does not need to be executed in a single database shard, but rather across multiple database shards. For write commands, data can be distributed across different databases, thus avoiding the data skew problem caused by storing data in a single database shard and improving the efficiency of using multiple database shards.
[0008] Optionally, the database includes an index table, which includes index information. The index information includes: spatial address information of each of the X database shards and data volume information of each spatial address information in each database shard; the data format of the spatial address information corresponds to the data format in the X database shards.
[0009] Thus, compared to related technologies, since the spatial address information mentioned above is not stored as a string in each database shard, even if multiple database shards in X database shards contain the same spatial address information, it will not cause excessive memory consumption due to the increase in spatial address information. At the same time, the index table can accurately represent the spatial address information and the corresponding data volume information, which is convenient for subsequent execution of read and write commands.
[0010] Optionally, before receiving the command sent by the user terminal, the method further includes: traversing the X database shards; generating index information for the X database shards; and generating an index table for the X database shards based on the index information.
[0011] In this way, even if multiple database shards in X database shards contain the same spatial address information, the memory consumption will not be excessive due to the increase in spatial address information. At the same time, the index table can accurately represent the spatial address information and the corresponding data volume information, which facilitates the subsequent execution of read and write commands.
[0012] Optionally, when the above command is a read command, the above splitting of the above command into Y sub-commands includes: splitting the above command into X sub-read commands, the X sub-read commands corresponding to X database shards; after executing the above Y sub-commands in the above Y database shards, the above further includes: obtaining X result data from the above X database shards; and determining the command result of the above command based on the above X result data.
[0013] In this way, by directly splitting the read command into X sub-read commands corresponding to X database shards, the information required by the read command can be retrieved quickly and comprehensively, avoiding omissions and improving the efficiency of information retrieval.
[0014] Optionally, when the above command is a first write command, the first write command is a command to write data in at least two database shards, and the data written by the first write command is a string or an integer. After receiving the command sent by the user terminal, the method further includes: querying the index table for the data volume information of each of the X database shards and the Y database shards containing the first space address information of the first write command; sorting the X database shards according to the data volume information in the X database shards; and obtaining the data volume information of the X database shards. The sorting order; the above-mentioned command is split into Y sub-commands according to the command content of the above-mentioned command, including: splitting the above-mentioned first write command into Y sub-first write commands according to the above-mentioned sorting order, each word first write command in the Y sub-first write commands includes first space address information and at least one data value, and the number of data values in the Y sub-first write commands is negatively correlated with the sorting order of the above-mentioned Y database shards; the above-mentioned execution of the above-mentioned Y sub-commands in the above-mentioned Y database shards includes: executing the Y sub-first write commands according to the data volume information in the above-mentioned Y database shards.
[0015] Thus, before executing the first write command, the data volume information in each database shard is queried through the index table. Then, the first write command is split according to the data volume information, and data is written to multiple database shards. This ensures that the data volume in the X database shards is always in a relatively balanced state, avoiding problems such as data skew.
[0016] Optionally, the above-mentioned execution of Y sub-first write commands based on the data volume information in the Y database shards includes: when the P database shards of the Y database shards include first spatial address information, replacing the original data value corresponding to the first spatial address information in the P database shards with at least one data value of the P write commands, where P is a positive integer less than or equal to Y; and writing the first spatial address information and at least one data value corresponding to YP sub-first write commands into the remaining database shards other than the P database shards.
[0017] Optionally, after executing Y sub-first write commands based on the data volume information in the Y database shards, the method further includes: calculating the first index information of the Y database shards in the index table according to a first preset algorithm; updating the index information of the Y database shards in the index table according to the first index information; wherein the first preset algorithm corresponds to the data structure of the database shards.
[0018] Thus, after new data is written via the first write command, the index information (i.e., spatial address information and the corresponding data volume information) in each database shard may change. Therefore, the above method can be used to modify the index table in a timely manner, so as to execute commands more accurately and quickly in the database shards in the future.
[0019] Optionally, when the above command is a second write command, the second write command includes second spatial address information and an integer data value. The step of splitting the command into Y sub-commands based on its content includes: generating X query commands based on the second write command, each corresponding to one of the X database shards, and using these X query commands to query the second spatial address information within the X database shards; using these X query commands to query the first database shard within the X database shards, where the first database shard includes the second spatial address information of the second write command; and executing the Y sub-commands within the Y database shards includes: if the first database shard is included among the X database shards, executing the second write command within the first database shard.
[0020] Therefore, before executing the second write command, X query commands are generated based on the second write command to query the first database containing the second space address information in X database shards, and then the second write command is executed in the first database. In this way, by generating X query commands based on the second write command, the first database from which the second write command can be executed can be found in a timely manner, ensuring the accuracy and reliability of the execution of the second write command.
[0021] Optionally, if the aforementioned second spatial address information is not included in the aforementioned X database shards, the method further includes: querying the data volume information of the X database shards through the aforementioned index table; sorting the X database shards according to the data volume information of the aforementioned X database shards; obtaining the sorting order of the X database shards; selecting the second database shard with the smallest data volume information among the X database shards; and writing the aforementioned second spatial address information and the aforementioned integer data value into the aforementioned second database shard.
[0022] In this way, if none of the X database shards contain the aforementioned second space address information, data can be written to the database shard with the smallest data volume among the X database shards by obtaining the sorting order, thereby ensuring data balance and avoiding data skew problems.
[0023] Optionally, after writing the second spatial address information and the integer data value into the second database shard, the method further includes: calculating the second index information of the second database shard in the index table according to the second preset algorithm; updating the index information of the second database shard in the index table according to the second index information; wherein the second preset algorithm corresponds to the data structure of the database shard.
[0024] Thus, after new data is written via the second write command, the index information (i.e., spatial address information and the corresponding data volume information) in a certain database shard may change. Therefore, the above method can be used to modify the index table in a timely manner so that commands can be executed more accurately and quickly in the database shards in the future. Attached Figure Description
[0025] The following description, in conjunction with the accompanying drawings, further illustrates this application:
[0026] Figure 1 This is a basic flowchart illustrating a data processing method according to a specific embodiment of this application;
[0027] Figure 2 This is one of the schematic diagrams illustrating a data processing method according to a specific embodiment of this application;
[0028] Figure 3 This is a second schematic diagram of a data processing method according to a specific embodiment of this application;
[0029] Figure 4 This is a schematic diagram of the basic structure of a data processing apparatus according to an embodiment of this application;
[0030] Figure 5 This is a basic structural block diagram of a network device according to an embodiment of this application. Detailed Implementation
[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0033] The following is an explanation of the terms that appear in this application:
[0034] 1. NoSQL databases (Not Only SQL, non-relational SQL, NoSQL)
[0035] NoSQL can be used to solve the problem of large-scale datasets with multiple data types in traditional databases. NoSQL has the following advantages: It is easily scalable; there are many types of NoSQL databases, but all of them share the common characteristic of removing the relational features of relational databases. That is, the data in NoSQL is unrelated to each other, making it very easy to scale. This also brings scalability at the architectural level. It handles large data volumes and performs well; NoSQL has very high read and write performance, especially with large datasets. This is due to its unrelational nature and simple database structure.
[0036] NoSQL includes the following types: key-value stores and column-oriented stores.
[0037] 2. Remote Dictionary Server Cluster (Redis)
[0038] Redis (Remote Dictionary Server) is a NoSQL database. It's a key-value storage system that supports a wide range of value types, including strings, lists, sets, sorted sets (zsets), and hashes.
[0039] 3. Redis Cluster
[0040] In this patent, "Redis cluster" does not refer to the commonly used open-source Redis-cluster mode, but specifically to an architecture composed of multiple shards, where each shard stores a portion of the data, and all shards are stacked together to store the full data of the Redis cluster.
[0041] 4. Proxy Middleware
[0042] The proxy middleware sits between the client and the Redis cluster. It processes the requests sent by the client (such as through sharding) and then forwards them to the actual Redis cluster at the backend.
[0043] The proxy middleware and the Redis cluster can be deployed on the same server or on different servers; this application embodiment does not limit this.
[0044] There are many types of proxy middleware. For example, Twitter open-sourced twemproxy on GitHub, which acts as a proxy middleware for memcache and redis. When a client connects to twemproxy, twemproxy uniformly forwards its read and write requests, distributing them to the redis cluster or memcache.
[0045] The data processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0046] This embodiment provides a data processing method, such as Figure 1 As shown, this data processing method is applied to a network device and includes the following steps 301 to 303:
[0047] Step 301: If the database architecture includes X database shards and proxy middleware, receive commands sent by the client.
[0048] In this embodiment of the application, the above command includes: a read command, or a write command, where X is an integer greater than 1.
[0049] In the embodiments of this application, such as Figure 2 As shown, the database architecture 100 includes a database sharding cluster (X database shards) 101 and a proxy middleware 102.
[0050] It should be noted that;
[0051] First, in the embodiments of this application, the above-mentioned database sharding includes two or more database shards.
[0052] Second, in the database architecture of this application, the X database shards do not directly receive commands (write or read commands) sent by the client. Instead, these commands are received by a proxy middleware and then executed in at least one database shard. It is understandable that when the X database shards and the proxy middleware are not deployed on the same server, the proxy middleware needs to first send the command to at least one database shard before executing it. When the X database shards and the proxy middleware are deployed on the same server, the proxy middleware can execute the command in at least one database shard without needing to send the command to other servers.
[0053] In the embodiments of this application, the data structures supported by the above database architecture can be one or more.
[0054] In one possible embodiment, the data structures supported by the above database architecture include at least one of the following: hash, set, zset, and list, which are not limited in this application embodiment.
[0055] In this embodiment of the application, the above read command is used to read data from X database shards.
[0056] In one possible embodiment, when the data structure is a hash structure, the above read command may include at least one of the following: a command hgetall for retrieving all key / value pairs under a specific address (base key) in X database shards, a command hlen for retrieving the data length at a specific address in X database shards, a command hget for retrieving the value corresponding to a specific address in X database shards, and a command hmget for retrieving the value corresponding to multiple specific addresses in X database shards.
[0057] In this embodiment of the application, the write command is used to write data in at least two database shards.
[0058] In one possible embodiment, when the data structure is a hash structure, the above write command may include at least one of the following: a command hset for writing data in one of the X database shards, a command hmset for writing data in at least two of the X database shards, a command hincrby for incrementing the value at a specific address in a specific database among the X database shards, and a command hdecrby for decrementing the value at a specific address in a specific database among the X database shards.
[0059] Step 302: Based on the command content of the above command, split the above command into Y sub-commands.
[0060] In this embodiment of the application, the above Y sub-commands correspond to Y databases in X database shards. The above Y databases store data information related to the Y sub-commands, where Y is less than or equal to X and Y is an integer.
[0061] In this embodiment of the application, the command is split in different ways depending on the different command contents.
[0062] In one implementation, when the command is a read command, the network device's proxy middleware will split the read command into X sub-read commands, each of which can be used to read data from the database shard corresponding to the address of its command.
[0063] Furthermore, the command content of the above X sub-read commands can be the same, and each sub-read command is used to read data from a database shard.
[0064] In another implementation, when the command is a write command, the network device's proxy middleware first queries the data volume information in X database shards or the correspondence with the write command (correspondence means whether the write command can be executed in the X database shards), obtains the query results, and then splits the write command into at least one sub-write command based on the query results. It is understood that after splitting the write command, the number of sub-write commands is related to the specific command content of the write command, which will be described in detail later and will not be repeated here.
[0065] Step 303: Execute the above Y sub-commands in the above Y database shards.
[0066] In the data processing method provided in this application embodiment, when the database architecture includes X database shards and a proxy middleware, a read command or a write command is received. Then, based on the command content, the command is split into Y sub-commands corresponding to Y databases in the X database shards (the Y databases store data information related to the Y sub-commands). Finally, the Y sub-commands are executed in the Y database shards. Thus, by splitting the command, the command does not need to be executed in a single database shard, but rather across multiple database shards. For write commands, data can be distributed across different databases, thus avoiding the data skew problem caused by storing data in a single database shard and improving the efficiency of using multiple database shards.
[0067] Optionally, in this embodiment of the application, the database includes an index table, the index table includes index information, and the index information includes: spatial address information of each database shard in X database shards and data volume information of each spatial address information in each database shard; the data format of the spatial address information corresponds to the data format in the X database shards.
[0068] For example, the data volume information mentioned above refers to the amount of data at each spatial address in each database shard. For instance, for any database shard, the first spatial address in database shard 1 corresponds to 3 data items, and these 3 items represent the data volume information.
[0069] It is understandable that:
[0070] First, the aforementioned index table is used to indicate the spatial address information of each database shard in the X database shards, as well as the data volume information corresponding to each spatial address.
[0071] Second, the spatial address information in different database shards can be different, and the data volume information corresponding to the spatial address information in different database shards can also be different.
[0072] For example, the index table includes two database shards, database shard 1 and database shard 2. Database shard 1 includes three spatial address information entries: spatial address 1, spatial address 2, and spatial address 3. Database shard 2 includes two spatial address information entries: spatial address 3 and spatial address 4. Specifically, spatial address 1 in database shard 1 corresponds to 3 data entries, spatial address 2 corresponds to 2 data entries, and spatial address 3 corresponds to 2 data entries. Similarly, spatial address 3 in database shard 2 corresponds to 4 data entries, and spatial address 4 corresponds to 2 data entries.
[0073] In one embodiment, when the data structure is a hash structure, the aforementioned spatial address information can be a base key.
[0074] It should be noted that the spatial address information mentioned above is stored using a data format corresponding to the database's data structure. For example, when the database's data structure is a hash structure, the spatial address information for each database shard is stored using a hash structure.
[0075] Thus, compared to related technologies, since the spatial address information mentioned above is not stored as a string in each database shard, even if multiple database shards in X database shards contain the same spatial address information, it will not cause excessive memory consumption due to the increase in spatial address information. At the same time, the index table can accurately represent the spatial address information and the corresponding data volume information, which is convenient for subsequent execution of read and write commands.
[0076] Optionally, in this embodiment of the application, before receiving the command sent by the user terminal in step 301 above, the data processing method provided in this embodiment of the application may include the following steps A1 to A3:
[0077] Step A1: Traverse the above X database shards.
[0078] Step A2: Generate the index information for the above X database shards.
[0079] Step A3: Based on the above index information, generate the index table for the above X database shards.
[0080] It is understood that steps A1 to A3 above can be used to generate a new index table. Therefore, steps A1 to A3 above can be used in scenarios where there is no index table or where the index table needs to be updated, as well as in other scenarios. This application embodiment does not limit this.
[0081] For example, traversing the above X database shards means reading all the database-related information of all database shards.
[0082] For example, generating the index information for the above X database shards means: calculating the spatial address information in each database shard and the data volume information corresponding to the spatial address information in each database shard according to the calculation method corresponding to the data structure of the database, and generating the index information.
[0083] Understandably, the raw spatial address information in each database shard that network devices directly obtain is typically in string format. Strings occupy a significant amount of storage space. Therefore, this raw spatial address information can be converted into other data types that require less storage space by performing calculations. Generally, network devices will calculate the raw spatial address information in a manner corresponding to the database's data structure, thereby changing the data type of the raw spatial address information.
[0084] Example 1: Assume that database shard 1 contains data as base key1 value1 key2 value2, where base is the name of its original spatial address information, key1 value1 is one key-value pair, and key2 value2 is another key-value pair. That is, database shard 1 contains the original spatial address information base and two key-value pairs. Therefore, the amount of data in base in database shard 1 is 2. Assuming that index information needs to be generated based on the data stored in database shard 1, and the database data structure is a hash structure, then refer to... Figure 3 The generation process is as follows: First, generate the spatial address information in the index information: split the original spatial address information "base" into "ba" and "se". Then, hash the first half of the string using the murmurhash3 algorithm to get 0xff3e, and hash the second half of the string using the cityhash algorithm to get 0xe340. Then, perform an XOR operation on the two results to get 0xFF35, which is used as the spatial address information in the index information. Second, generate the data volume information of the spatial address information in the index information: based on the number of key-value pairs mentioned above, the data volume information can be stored in two places.
[0085] It should be noted that the data volume information corresponding to the index table is an array. To save memory usage in the proxy middleware of network devices, the numbers from 0 to 100,000 can be statically allocated as an array. That is, as long as the number of key-value pairs is less than 100,000, pointers to this static array are actually stored. Only when the number is greater than 100,000 will memory be allocated to store the actual number of key-value pairs (i.e., the aforementioned data volume information). Figure 2 As shown, after obtaining the result 0xff35, the corresponding space address information and data volume information are indicated by the stored pointer. Figure 2 The pointer in the array points to a value in the static array.
[0086] For example, the network device summarizes the spatial address information in each database shard and the data volume information corresponding to the spatial address information in each database shard, and finally generates an index table.
[0087] In this way, even if multiple database shards in X database shards contain the same spatial address information, the memory consumption will not be excessive due to the increase in spatial address information. At the same time, the index table can accurately represent the spatial address information and the corresponding data volume information, which facilitates the subsequent execution of read and write commands.
[0088] Optionally, in this embodiment of the application, when the above command is a read command, the data processing method provided in step 302, which splits the above command into Y sub-commands, may include the following steps B1 to B3:
[0089] Step B1: Break down the above command into X sub-read commands.
[0090] Based on step B1 above, after step 303 above, the data processing method provided in this application embodiment further includes the following steps B2 and B3:
[0091] Step B2: Obtain X result data from the above X database shards.
[0092] Step B3: Based on the above X result data, determine the command result of the above command.
[0093] For example, the above X sub-read commands correspond to X database shards.
[0094] For example, splitting the above into X sub-read commands may include: first copying the command content of the read command into X sub-read command contents, then adding X database block addresses to each sub-read command content, and finally merging the above X database block addresses and X sub-read command contents to generate X sub-read commands.
[0095] For example, the correspondence between the X sub-read commands and the X database shards mentioned above means that the command addresses of the X sub-read commands correspond one-to-one with the database shard addresses in the X database shards.
[0096] For example, after the above X database shards execute the above X sub-read commands, X result data will be generated, and the network device can determine the final command result based on the X result data.
[0097] It is understood that, in the embodiments of this application, since the same spatial address information and the corresponding specific key-value pairs may appear in multiple or even all database shards, during the execution of the read command, the read command is directly split into X sub-read commands corresponding to X database shards.
[0098] The following examples use the different read commands mentioned above. Assume that there are X database shards, which are divided into 3 database shards, namely redis1, redis2 and redis3, and the proxy middleware is twemproxy. Here are 4 examples of executing read commands.
[0099] Example 2: Assuming the read command is Hlen, the network device's twemproxy will split Hlen into three sub-read commands, each corresponding to redis1, redis2, and redis3. Then, each of the three sub-read commands is sent to redis1, redis2, and redis3. The length of the corresponding spatial address information (i.e., the X result data mentioned above) is then obtained from redis1, redis2, and redis3. The result data for redis1 is 5, for redis2 it is 6, and for redis3 it is 6. Finally, based on the three result data, twemproxy will perform a combined calculation, merging the three results: 5 + 6 + 6 = 17, and finally sending 17 to the user.
[0100] Example 3: Assuming the read command is Hget, the network device's twemproxy will split Hget into three sub-read commands, each corresponding to redis1, redis2, and redis3. Then, each of the three sub-read commands is sent to redis1, redis2, and redis3. The value (i.e., the X result data mentioned above) corresponding to one of the memory addresses is then retrieved from redis1, redis2, and redis3. The result data for redis1 is 3, the result data for redis2 is 0 (i.e., empty), and the result data for redis3 is 2. Finally, based on the three result data, they are merged and summarized in twemproxy. Twemproxy filters out the empty result data from redis2 and sends the result data 3 from redis1 and the result data 2 from redis3 to the client.
[0101] Example 4: Assuming the read command is Hmget, the network device's twemproxy will split Hmget into three sub-read commands, each corresponding to redis1, redis2, and redis3. Then, each of the three sub-read commands is sent to redis1, redis2, and redis3. In redis1, redis2, and redis3, the corresponding Z spatial address information values (i.e., the X result data mentioned above) are retrieved. The result data for redis1 is value1, for redis2 it is value2, and for redis3 it is value3. Finally, based on the three result data, twemproxy will merge and summarize them, sending the result data value1 from redis1, value3 from redis2, and value3 from redis3 to the client.
[0102] Example 5: Assuming the read command is Hgetall, the network device's twemproxy will split Hgetall into three sub-read commands, each corresponding to redis1, redis2, and redis3. Then, each of the three sub-read commands is sent to redis1, redis2, and redis3. In redis1, redis2, and redis3, the corresponding key / value pairs (i.e., the X result data mentioned above) are retrieved. The result data for redis1 is key1 / value1, for redis2 it's key2 / value2, and for redis3 it's key3 / value3. Finally, based on these three result data, twemproxy merges and summarizes them, sending the result data (key1 / value1 from redis1, key2 / value2 from redis2, and key3 / value3 from redis3) to the client.
[0103] In this way, by directly splitting the read command into X sub-read commands corresponding to X database shards, the information required by the read command can be retrieved quickly and comprehensively, avoiding omissions and improving the efficiency of information retrieval.
[0104] Optionally, in this embodiment of the application, when the above command is a first write command, the first write command is a command to write data in at least two database shards, and the data written by the first write command is a string or an integer. After receiving the command sent by the user terminal in step 301 above, the data processing method provided in this embodiment of the application may include the following steps C1 to C5:
[0105] Step C1: Query the data volume information of each database shard in the X database shards and the Y database shards containing the first space address information of the first write command in the X database shards through the above index table.
[0106] Step C2: Sort the X database shards according to the data volume information in the X database shards mentioned above.
[0107] Step C3: Obtain the sorting order of the X database shards.
[0108] Based on steps C1-C3 above, in step 302 above, the data processing method provided in this application embodiment may include the following step C4:
[0109] Step C4: According to the above sorting order, split the first write command to generate Y sub-first write commands.
[0110] For example, each word first write command in the Y sub-first write commands includes first space address information and at least one data value, and the number of data values in the Y sub-first write commands is negatively correlated with the sorting order of the aforementioned Y database shards.
[0111] Based on C1-C4 above, in step 303 above, the data processing method provided in this application embodiment may include the following step C5:
[0112] Step C5: Based on the data volume information in the Y database shards mentioned above, execute the Y first write commands.
[0113] For example, the data writing method of the first write command is substitution write. The substitution write refers to replacing the numerical content in the original spatial address information with numerical content (e.g., value) other than the spatial address information carried by the first write command.
[0114] It should be noted that if the database shard to which the first write command is written does not contain the spatial address information carried by the first write command, then the write will proceed directly.
[0115] In one example, when the data structure is a hash structure, the first write command mentioned above can be hmset.
[0116] For example, the above data volume information can be referred to the foregoing description, and will not be repeated here.
[0117] For example, the aforementioned first spatial address information refers to the spatial address information corresponding to the first write command. It can be understood that the aforementioned first spatial address information can be spatial address information that exists in all X database shards; it can also be spatial address information that does not exist in the databases of the X database shards; or it can be spatial address information that exists in some database shards (Y database shards) of the X database shards, but not in the remaining database shards.
[0118] For example, the sorting of the X database shards can be done by sorting the X database shards from largest to smallest according to the size of the data information; or by sorting the X database shards from smallest to largest according to the size of the data information. This application embodiment does not limit this.
[0119] It is understandable that, after obtaining the above data volume information and sorting order, when executing the first write command, in order to balance the number of data values in each of the X database shards, the most data needs to be written to the database shard with the fewest data values, and the least data needs to be written to the database shard with the most data values, or even none at all. Based on this writing method that is negatively correlated with the sorting order, the first write command is split into Y sub-first write commands (Y is less than or equal to X; when Y is less than X, it indicates that some database shards have too much data and are not suitable for writing). The number of data values in each sub-first write command is negatively correlated with its sorting order.
[0120] The following example uses the hmset command. Assume there are X database shards, specifically 3 shards: redis1, redis2, and redis3, and the proxy middleware is twemproxy. Here are four examples of executing the hmset command: hmset base key1 / value1, key2 / value2, key3 / value3, key4 / value4, key5 / value5.
[0121] Example 6: Query the data volume information of redis1, redis2, and redis3 in the index table, and the first space address information (base) corresponding to hmset. The data volume information of redis1 is 3, the data volume information of redis2 is 2, and the data volume information of redis3 is 3. Since redis1, redis2, and redis3 do not include the first space address information (base), the sorting order of redis1, redis2, and redis3 according to the data volume information from largest to smallest is redis1, redis3, and redis2. Based on this sorting order, the hmset command is split into three sub-hmset commands: hmset base key1 / value1, key2 / value2, hmset base key3 / value3, key4 / value4, and hmset base key5 / value5. These three sub-hmset commands are then executed in redis1, redis2, and redis3 respectively. After execution, the data volume information of redis1, redis2, and redis3 is 5, 4, and 4 respectively.
[0122] It should be noted that, generally, after executing the Y sub-write commands after splitting, the difference in data volume information between each database shard will not exceed 1.
[0123] Thus, before executing the first write command, the data volume information in each database shard is queried through the index table. Then, the first write command is split according to the data volume information, and data is written to multiple database shards. This ensures that the data volume in the X database shards is always in a relatively balanced state, avoiding problems such as data skew.
[0124] Optionally, in this embodiment of the application, step C5 above may include the following steps D1 and D2:
[0125] Step D1: If the first space address information is included in the P database shards of the above Y database shards, replace the original data value corresponding to the first space address information in the P database shards with at least one of the above data values of the P write commands, where P is a positive integer less than or equal to Y.
[0126] Step D2: Write the first space address information and at least one data value corresponding to the first write command for each of the remaining database shards other than the P database shards mentioned above.
[0127] For example, as described above, since the first write command is executed in an alternative manner, that is, when the database block in which the sub-write command corresponding to the first write command is executed includes the first space address information, the data value of the sub-write command needs to be used to replace the original data value.
[0128] The following example uses the hmset command. Assume there are X database shards, specifically 3 shards: redis1, redis2, and redis3, and the proxy middleware is twemproxy. Here are four examples of executing the hmset command: hmset base key1 / value1, key2 / value2, key3 / value3, key4 / value4, key5 / value5.
[0129] Example 6: Query the data volume information of redis1, redis2, and redis3 in the index table, and the first space address information (base) corresponding to hmset. The data volume information of redis1 is 3, the data volume information of redis2 is 2, and the data volume information of redis3 is 3. redis1 includes the first space address information (base), while redis2 and redis3 do not. Therefore, sort redis1, redis2, and redis3 according to the data volume information from largest to smallest in the following order: redis1, redis3, and redis2. Based on this sorting order, the hmset command is split into three sub-hmset commands: hmset base key1 / value1, key2 / value2, key3 / value3, hmset base key4 / value4, and hmset base key5 / value5. Execute these three sub-hmset commands in redis1, redis2, and redis3 respectively, resulting in data volumes of 3, 3, and 4.
[0130] Optionally, in this embodiment of the application, after step C5, the data processing method provided in this embodiment of the application may include the following steps E1 and E2:
[0131] Step E1: Calculate the first index information of the above Y database shards in the index table according to the first preset algorithm.
[0132] Step E2: Based on the first index information mentioned above, update the index information of the Y database shards in the index table.
[0133] For example, the first preset algorithm described above corresponds to the data structure of the database sharding described above.
[0134] In one example, where the data structure of the database sharding is a hash structure, the first preset algorithm mentioned above is a hash algorithm.
[0135] It is understandable that after the first write command writes new data to Y database shards, the index information of the Y databases will change. Therefore, the first index information can be obtained by calculating the first index information in the Y database shards and the index table can be updated in a timely manner.
[0136] Furthermore, the method for calculating the first index information in steps E1 and E2 above can refer to the method for generating index information in steps A1 to A2 above, and will not be repeated here.
[0137] Thus, after new data is written via the first write command, the index information (i.e., spatial address information and the corresponding data volume information) in each database shard may change. Therefore, the above method can be used to modify the index table in a timely manner, so as to execute commands more accurately and quickly in the database shards in the future.
[0138] Optionally, in this embodiment of the application, when the above command is a second write command, the second write command includes second space address information and an integer data value. In step 302 above, the data processing method provided in this embodiment of the application may include the following steps F1 and F2:
[0139] Step F1: Generate X query commands based on the second write command above.
[0140] Step F2: Using the above X query commands, query the first database shard among the above X database shards.
[0141] Based on steps F1 and F2 above, in step 303 above, the data processing method provided in this application embodiment may include the following step F3:
[0142] Step F3: If the first database shard is included among the X database shards, execute the second write command in the first database shard.
[0143] For example, the above X query commands correspond to the above X database shards.
[0144] For example, the first database shard mentioned above includes the second space address information of the second write command mentioned above.
[0145] For example, the above X query commands are used to query the second spatial address information in the above X database shards. That is, the X query commands are used to query whether the second spatial address information exists in the above X database shards.
[0146] In one example, the above X query commands can be generated by the network device based on the second write command, and each query command can be used to query the second space address information in X database shards.
[0147] For example, the data writing method of the second write command is either superimposed writing or decremental writing. Superimposed writing means that the numerical content (e.g., integer value) other than the spatial address information carried by the second write command is superimposed on the numerical content in the original spatial address information. Decremental writing means that the numerical content (e.g., integer value) other than the spatial address information carried by the second write command is decremented from the numerical content in the original spatial address information.
[0148] It should be noted that if the database shard to which the second write command is written does not contain the second space address information carried by the second write command, then write directly. If write directly, the second write command can be used without further writing. For details, please refer to the subsequent steps G1-G5.
[0149] In one example, when the data structure is a hash structure, the second write command mentioned above can be either hincrby or hdecrby.
[0150] Understandably, since the second write command writes data in an additive or decremental manner, generally, we first check whether the X database shards contain the second space address corresponding to the second write command. If the second space address exists, the second write command is executed directly in the database shard corresponding to the second space address (i.e., the first database shard mentioned above).
[0151] The following example uses the `hincrby` command. Assume there are X database shards, specifically three shards: `redis1`, `redis2`, and `redis3`, and the proxy middleware is `twemproxy`. Here are four examples of executing the `hincrby` command: `hincrby base key value`. The `value` must be an integer, not a string.
[0152] Example 7: After receiving the hincrby command, the network device will generate three hexists commands (i.e., the query commands mentioned above). These three hexists commands are used to query the key in the hincrby command (i.e., the second spatial address information mentioned above), and send the three hexists commands to the corresponding redis1, redis2, and redis3 respectively. Redis3 includes the key in the hincrby command. If redis is the first database shard mentioned above, then the value will be superimposed with the value corresponding to the key in redis3 (i.e., the second write command mentioned above will be executed).
[0153] Therefore, before executing the second write command, X query commands are generated based on the second write command to query the first database containing the second space address information in X database shards, and then the second write command is executed in the first database. In this way, by generating X query commands based on the second write command, the first database from which the second write command can be executed can be found in a timely manner, ensuring the accuracy and reliability of the execution of the second write command.
[0154] The system queries the data volume information in each database shard, then splits the first write command based on the data volume information and writes data to multiple database shards, thereby ensuring that the data volume in the X database shards is always in a relatively balanced state and avoiding problems such as data skew.
[0155] Optionally, in this embodiment of the application, if the aforementioned second spatial address information is not included in the aforementioned X database shards, the data processing method provided in this embodiment of the application may further include the following steps G1 to G5:
[0156] Step G1: Query the data volume information of X database shards through the above index table.
[0157] Step G2: Sort the X database shards according to the data volume information of the X database shards mentioned above.
[0158] Step G3: Obtain the sorting order of the X database shards.
[0159] Step G4: Select the second database shard with the smallest amount of data among the X database shards.
[0160] Step G5: Write the second space address information and the integer data value into the second database fragment.
[0161] For example, the sorting order described above is the same as described above, and will not be repeated here.
[0162] It is understandable that, since none of the X database shards contain the aforementioned second space address information, it is necessary to write the second space address information and the corresponding integer data value to one of the X database shards. In order to ensure the balance of data volume among the database shards, the first write command is needed to directly write data to the database shard with the smallest data volume among the X database shards.
[0163] In one embodiment, during steps G1-G5 above, when the data structure is a hash structure, the hset command can be used to write data.
[0164] The following example uses the `hincrby` command. Assume there are X database shards, specifically three shards: `redis1`, `redis2`, and `redis3`, and the proxy middleware is `twemproxy`. Here are four examples of executing the `hincrby` command: `hincrby base key value`. The `value` must be an integer, not a string.
[0165] Example 8: After receiving the `hincrby` command, the network device will generate three `hexists` commands (i.e., the query commands mentioned above). These three `hexists` commands are used to query the key (i.e., the second spatial address information mentioned above) in the `hincrby` command. The network device will then send these three `hexists` commands to the corresponding `redis1`, `redis2`, and `redis3`, respectively. Since `redis1`, `redis2`, and `redis3` do not contain the key from the `hincrby` command, the network device will query the data volume information of the three database shards using the index table mentioned above. Based on the data volume information, `redis1`, `redis2`, and `redis3` will be sorted in descending order as `redis1`, `redis3`, and `redis2`. The network device will then determine that `redis2` is the database shard with the smallest data volume. Therefore, in `redis2`, the `hset` command (`hset base key value`) will be executed, writing the `basekey value` corresponding to the `hincrby` command.
[0166] In this way, if none of the X database shards contain the aforementioned second space address information, data can be written to the database shard with the smallest data volume among the X database shards by obtaining the sorting order, thereby ensuring data balance and avoiding data skew problems.
[0167] Optionally, in this embodiment of the application, after step G5 above, the data processing method provided in this embodiment of the application may further include the following steps H1 and H2:
[0168] Step H1: Calculate the second index information of the second database shard in the index table according to the second preset algorithm.
[0169] Step H2: Update the index information of the second database shard in the index table according to the second index information mentioned above;
[0170] For example, the second preset algorithm mentioned above corresponds to the data information of the database shards mentioned above.
[0171] In one example, when the data structure of the database sharding is a hash structure, the second preset algorithm mentioned above is a hash algorithm.
[0172] It is understandable that after the second write command writes new data into Y database shards, it will cause changes to the index information of the Y databases. Therefore, the second index information can be obtained by calculating the second index information in the Y database shards and the index table can be updated in a timely manner.
[0173] Furthermore, the method for calculating the second index information in steps H1 and H2 can refer to the method for generating index information in steps A1 to A2, which will not be repeated here.
[0174] Thus, after new data is written via the second write command, the index information (i.e., spatial address information and the corresponding data volume information) in a certain database shard may change. Therefore, the above method can be used to modify the index table in a timely manner so that commands can be executed more accurately and quickly in the database shards in the future.
[0175] Figure 4 This is a schematic diagram illustrating a first possible structure for implementing the data processing apparatus provided in the embodiments of this application. For example... Figure 4 As shown, the data processing device includes: a receiving module 601, configured to receive commands sent by a user terminal when the database architecture corresponding to the database includes X database shards and a proxy middleware, the commands including: a read command or a write command, where X is an integer greater than 1; an execution module 602, configured to split the command according to the command content received by the receiving module 601, generating Y sub-commands, the Y sub-commands corresponding to Y databases in the X database shards, the Y databases storing data information related to the Y sub-commands, where Y is less than or equal to X, and Y is an integer; the execution module is further configured to execute the Y sub-commands in the Y database shards.
[0176] In a database architecture comprising X database shards and a proxy middleware, the data processing device receives read or write commands. Based on the command content, it splits the command into Y sub-commands corresponding to Y databases within the X database shards (the Y databases store data related to the Y sub-commands). Finally, it executes the Y sub-commands across the Y database shards. This command splitting eliminates the need for centralized execution in a single database shard; instead, commands are executed across multiple shards. For write commands, data can be distributed across different databases, preventing data skew issues associated with storing data in a single shard and improving the efficiency of using multiple database shards.
[0177] Optionally, the database includes an index table, which includes index information. The index information includes: spatial address information of each of the X database shards and data volume information of each spatial address information in each database shard; the data format of the spatial address information corresponds to the data format in the X database shards.
[0178] Optionally, the data processing device further includes: a traversal module for traversing the X database shards; a generation module for generating index information for the X database shards; and the generation module is further configured to generate an index table for the X database shards based on the index information.
[0179] Optionally, the data processing apparatus further includes: when the command is a read command, the execution module 602 is specifically used to split the command into X sub-read commands, the X sub-read commands corresponding to X database shards; an acquisition module is used to acquire X result data from the X database shards; and a determination module is used to determine the command result of the command based on the X result data.
[0180] Optionally, when the above command is a first write command, the first write command is a command that writes data in at least two database shards, and the data written by the first write command is a string or an integer. The data processing device further includes: a query module, used to query the data volume information of each of the X database shards and the Y database shards containing the first spatial address information of the first write command through the index table; a sorting module, used to sort the X database shards according to the data volume information of the X database shards; an acquisition module, used to acquire the sorting order of the X database shards; and an execution module 602, specifically used to split the first write command according to the sorting order to generate Y sub-first write commands, each of the Y sub-first write commands including the first spatial address information and at least one data value, the number of data values of the Y sub-first write commands being negatively correlated with the sorting order of the Y database shards; the execution module is also specifically used to execute the Y sub-first write commands according to the data volume information of the Y database shards.
[0181] Optionally, the execution module 602 is specifically used to replace the original data value corresponding to the first space address information in the P database shards with the above-mentioned at least one data value of the P write commands when the first space address information is included in the P database shards of the above-mentioned Y database shards, where P is a positive integer less than or equal to Y; the execution module is also specifically used to write the first space address information and at least one data value corresponding to YP sub-first write commands into the remaining database shards other than the above-mentioned P database shards.
[0182] Optionally, the data processing device further includes: a calculation module, configured to calculate the first index information of the Y database shards in the index table according to a first preset algorithm; and an update module, configured to update the index information of the Y database shards in the index table according to the first index information; wherein the first preset algorithm corresponds to the data structure of the database shards.
[0183] Optionally, when the above command is a second write command, the second write command includes second spatial address information and an integer data value. The data processing device further includes: the execution module 602, specifically configured to generate X query commands according to the second write command, the X query commands corresponding to the X database shards, and the X query commands used to query the second spatial address information in the X database shards; a query module, configured to use the X query commands to query the first database shard in the X database shards, the first database shard including the second spatial address information of the second write command; and the execution module, specifically configured to execute the second write command in the first database shard when the first database shard is included in the X database shards.
[0184] Optionally, if the aforementioned second spatial address information is not included in the aforementioned X database shards, the aforementioned data processing apparatus further includes: the aforementioned query module, used to query the data volume information of the X database shards through the aforementioned index table; sort the X database shards according to the data volume information of the aforementioned X database shards; the acquisition module, used to acquire the sorting order of the X database shards; filter out the second database shard with the smallest data volume information among the X database shards; and the execution module 602, specifically used to write the aforementioned second spatial address information and the aforementioned integer data value into the aforementioned second database shard.
[0185] Optionally, the data processing device further includes: a calculation module, configured to calculate the second index information of the second database shard in the index table according to a second preset algorithm; and an update module, configured to update the index information of the second database shard in the index table according to the second index information; wherein the second preset algorithm corresponds to the data structure of the database shard.
[0186] This application also provides a network device, including a processor and a communication interface. The processor is configured to receive commands sent by a user terminal when the database architecture includes X database shards and a proxy middleware; split the commands according to their content; and execute the Y sub-commands in the Y database shards. This network device embodiment corresponds to the above-described network device method embodiment. All implementation processes and methods of the above method embodiments can be applied to this network device embodiment and achieve the same technical effects.
[0187] Specifically, embodiments of this application also provide a network device. For example... Figure 5 As shown, the network device 700 includes: an antenna 71, a radio frequency (RF) device 72, a baseband device 73, a processor 74, and a memory 75. The antenna 71 is connected to the RF device 72. In the uplink direction, the RF device 72 receives information through the antenna 71 and transmits the received information to the baseband device 73 for processing. In the downlink direction, the baseband device 73 processes the information to be transmitted and sends it to the RF device 72. The RF device 72 processes the received information and transmits it through the antenna 71.
[0188] The method executed by the network device in the above embodiments can be implemented in the baseband device 73, which includes a baseband processor.
[0189] The baseband device 73 may include, for example, at least one baseband board on which multiple chips are disposed, such as... Figure 5 As shown, one of the chips is, for example, a baseband processor, which is connected to the memory 75 via a bus interface to call the program in the memory 75 and execute the network device operations shown in the above method embodiment.
[0190] The network device may also include a network interface 76, such as a common public radio interface (CPRI).
[0191] Specifically, the network device 700 of this embodiment further includes: instructions or programs stored in a memory 75 and executable on a processor 74, wherein the processor 74 calls the instructions or programs in the memory 75 to execute. Figure 5 The methods executed by each module shown achieve the same technical effect, and to avoid repetition, they will not be described in detail here.
[0192] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0193] The processor is the processor in the terminal described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0194] This invention has been illustrated through several specific embodiments. Those skilled in the art will understand that various modifications and equivalent substitutions can be made to this invention without departing from its scope. Furthermore, various modifications can be made to this invention for specific situations or circumstances without departing from its scope. Therefore, this invention is not limited to the specific embodiments disclosed, but should include all embodiments falling within the scope of the claims.
Claims
1. A data processing method, characterized in that, Applied to network devices, the method includes: In a database architecture comprising X database shards and a proxy middleware, commands sent by the user client are received. These commands include read commands or write commands, where X is an integer greater than 1. The database includes an index table containing index information. This index information includes spatial address information for each of the X database shards and data volume information for each spatial address in each database shard. The data volume information refers to the amount of data at each spatial address in each database shard. The data format of the spatial address information corresponds to the data format in the X database shards. Based on the command content, the command is split into Y sub-commands. The Y sub-commands correspond to Y databases in X database shards. The Y databases store data information related to the Y sub-commands. Y is less than or equal to X and Y is an integer. Execute the Y sub-commands in the Y database shards; Wherein, if the command is a first write command, the first write command is a command that writes data in at least two database shards, and the data written by the first write command is a string or an integer, the method further includes, after receiving the command sent by the user terminal: The index table is used to query the data volume information of each of the X database shards and the Y database shards that contain the first space address information of the first write command; Based on the data volume information in the X database shards, sort the X database shards; Obtain the sorting order of the X database shards; The step of splitting the command into Y sub-commands based on its content includes: According to the sorting order, the first write command is split into Y sub-first write commands. Each word first write command in the Y sub-first write commands includes first space address information and at least one data value. The number of data values in the Y sub-first write commands is negatively correlated with the sorting order of the Y database shards. The execution of the Y sub-commands in the Y database shards includes: Based on the data volume information in the Y database shards, execute Y sub-first write commands.
2. The method according to claim 1, characterized in that, Before receiving the command sent by the user terminal, the method further includes: Traverse the X database shards; Generate index information for the X database shards; Based on the index information, an index table is generated for the X database shards.
3. The method according to claim 1 or 2, characterized in that, When the command is a read command, the step of splitting the command into Y sub-commands includes: The command is split into X sub-read commands, and the X sub-read commands correspond to X database shards; After executing the Y sub-commands in the Y database shards, the method further includes: Obtain X result data from the X database shards; Based on the X result data, determine the command result of the command.
4. The method according to claim 1, characterized in that, The step of executing Y sub-first write commands based on the data volume information in the Y database shards includes: When the first spatial address information is included in the P database shards of the Y database shards, the original data value corresponding to the first spatial address information in the P database shards is replaced by the data value of the P write commands, where P is a positive integer less than or equal to Y. Write YP first space address information and at least one data value corresponding to the first write command to the remaining database shards other than the P database shards.
5. The method according to claim 1, characterized in that, After executing Y sub-first write commands based on the data volume information in the Y database shards, the method further includes: The first index information of the Y database shards in the index table is calculated according to the first preset algorithm; Update the index information of the Y database shards in the index table based on the first index information; The first preset algorithm corresponds to the data structure of the database sharding.
6. The method according to claim 1, characterized in that, When the command is a second write command, the second write command includes second space address information and an integer data value. The step of splitting the command into Y sub-commands based on its content includes: Based on the second write command, X query commands are generated, each corresponding to one of the X database shards, and the X query commands are used to query the second spatial address information in the X database shards; Using the X query commands, query the first database shard among the X database shards, where the first database shard includes the second space address information of the second write command; The execution of the Y sub-commands in the Y database shards includes: If the first database shard is included among the X database shards, the second write command is executed in the first database shard.
7. The method according to claim 6, characterized in that, If the second spatial address information is not included in the X database shards, the method further includes: The index table is used to query the data volume information of X database shards; Based on the data volume information in the X database shards, the X database shards are sorted. Obtain the sorting order of the X database shards; Select the second database shard with the smallest amount of data among the X database shards; Write the second space address information and the integer data value into the second database fragment.
8. The method according to claim 7, characterized in that, After writing the second spatial address information and the integer data value into the second database shard, the method further includes: The second index information of the second database shard in the index table is calculated according to the second preset algorithm; Update the index information in the index table of the second database shard according to the second index information; The second preset algorithm corresponds to the data structure of the database sharding.
9. A data processing apparatus, characterized in that, The device includes: The receiving module is configured to receive commands sent by the user terminal when the database architecture includes X database shards and a proxy middleware. The commands include read commands or write commands, where X is an integer greater than 1. The database includes an index table, which contains index information. The index information includes spatial address information for each of the X database shards and data volume information for each spatial address in each database shard. The data volume information refers to the amount of data at each spatial address in each database shard. The data format of the spatial address information corresponds to the data format in the X database shards. The execution module is used to split the command according to the command content received by the receiving module, and generate Y sub-commands. The Y sub-commands correspond to Y databases in X database shards. The Y databases store data information related to the Y sub-commands. Y is less than or equal to X and Y is an integer. The execution module is also used to execute the Y sub-commands in the Y database shards; Wherein, if the command is a first write command, the first write command is a command to write data in at least two database shards, and the data written by the first write command is a string or an integer, the device further includes, after receiving the command sent by the user terminal: The query module is used to query the data volume information of each of the X database shards and the Y database shards containing the first space address information of the first write command through the index table; The sorting module is used to sort the X database shards according to the data volume information in the X database shards; The acquisition module is used to acquire the sorting order of the X database shards; The execution module is specifically used to: split the first write command according to the sorting order to generate Y sub-first write commands, each of the Y sub-first write commands including first space address information and at least one data value, and the number of data values of the Y sub-first write commands is negatively correlated with the sorting order of the Y database shards; The execution module is further specifically used to: execute Y sub-first write commands based on the data volume information in the Y database shards.
10. A network device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the data processing method as described in any one of claims 1 to 8.
11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1-8.