Data management method and apparatus

Abstract

The present disclosure relates to a data management method and device, belonging to the field of big data analysis, which comprises: obtaining table data and historical query records corresponding to the table data; determining a shard key and a range index according to the historical query records; pre-sharding the table data according to the shard key to obtain at least two shard replica sets for sharded storage; obtaining a query request sent by an application program for the table data, and calling a query interface according to the query request to route the query request to the corresponding shard replica set; obtaining a query result corresponding to the query request from the shard replica set according to the range index, and sending the query result to the application program. Thus, the present disclosure is based on the shard technology to more reasonably shard and store the table data, so as to improve the data query response speed, enable users to more quickly obtain the query result, and ensure the accuracy, flexibility and reliability in the data query process.

Description

Technical Field

[0001] This disclosure relates to the field of big data data analysis, and in particular to a data management method and apparatus. Background Technology

[0002] With the rapid development of the mobile internet, more and more activities are taking place online, which in turn increases the demand for functions that allow users to query various metrics and data.

[0003] In related technologies, to achieve data query functionality, Hadoop Database (HBase) is typically used to store data, and enterprise search application servers (Solr) are used to build indexes, or a remote dictionary service (Redis) is used for caching to provide fast queries. However, existing data query processes often suffer from high system complexity, high maintenance costs, and insufficient flexibility in supporting multi-condition queries. Therefore, improving the accuracy, flexibility, and reliability of data query processes has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides a data management method and apparatus to at least solve the problems in related technologies that fail to ensure accuracy, flexibility, and reliability during the data query process. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a data management method is provided, comprising: acquiring table data and historical query records corresponding to the table data; determining a sharding key and a range index based on the historical query records; pre-sharding the table data according to the sharding key to obtain at least two shard replica sets for sharded storage; acquiring a query request for the table data sent by an application, and calling a query interface according to the query request to route the query request to the corresponding shard replica set; obtaining a query result corresponding to the query request from the shard replica set according to the range index, and sending the query result to the application.

[0006] According to one embodiment of this disclosure, obtaining the data volume of the table data includes: obtaining at least one candidate business vertical and an estimated data volume corresponding to each candidate business vertical; obtaining the business vertical corresponding to the table data, and obtaining the data volume of the table data based on the business vertical and the estimated data volume.

[0007] According to one embodiment of this disclosure, obtaining the data volume of the table data includes: obtaining at least one candidate business vertical and an estimated data volume corresponding to each candidate business vertical; obtaining the business vertical corresponding to the table data, and obtaining the data volume of the table data based on the business vertical and the estimated data volume.

[0008] According to one embodiment of this disclosure, determining the sharding key and range index based on the historical query records includes: obtaining the query frequency corresponding to each query condition in the table data based on the historical query records; using the first query condition with the highest query frequency as the sharding key; using the remaining query conditions as second query conditions; and obtaining the range index based on the second query conditions.

[0009] According to one embodiment of this disclosure, the step of pre-sharding the table data according to the sharding key includes: obtaining a pre-set sharding interval; obtaining an initial number of shards based on the data volume and the sharding interval; performing hash processing on the table data according to the sharding key; and pre-sharding the hashed table data according to the initial number of shards.

[0010] According to one embodiment of this disclosure, obtaining the range index based on the second query condition includes: sorting the second query condition according to the query frequency; selecting the second query condition within a preset sorting range; and obtaining the range index based on the second query condition within the preset sorting range.

[0011] According to one embodiment of this disclosure, the step of calling a query interface based on the query request to route the query request to the corresponding shard replica set includes: extracting at least one query condition from the query request; calling a router and routing the query request to the corresponding shard replica set based on the query condition.

[0012] According to one embodiment of this disclosure, the method further includes: obtaining a target subset in the sharded replica set based on the range index and the query conditions, and routing the query request to the target subset.

[0013] According to one embodiment of this disclosure, the method further includes: in response to detecting that the table data update condition is met, obtaining the table data to be updated, and performing pre-sharding and sharded storage on the table data to be updated.

[0014] According to a second aspect of the present disclosure, a data management apparatus is provided, the apparatus comprising: a first acquisition unit configured to acquire table data and historical query records corresponding to the table data; a determination unit configured to determine a sharding key and a range index based on the historical query records; a pre-sharding unit configured to pre-shard the table data according to the sharding key to acquire at least two shard replica sets for sharded storage; a second acquisition unit configured to acquire a query request for the table data sent by an application, and call a query interface according to the query request to route the query request to the corresponding shard replica set; and a sending acquisition unit configured to acquire a query result corresponding to the query request from the shard replica set according to the range index, and send the query result to the application.

[0015] According to one embodiment of this disclosure, the first acquisition unit is further configured to perform: acquiring the data volume of the table data; determining a sharding strategy based on the data volume; and configuring at least one master node, slave node, and arbitrator node as nodes of the sharded replica set.

[0016] According to one embodiment of this disclosure, the first acquisition unit is further configured to perform: acquiring at least one candidate business vertical and an estimated data volume corresponding to each candidate business vertical; acquiring the business vertical corresponding to the table data, and acquiring the data volume of the table data based on the business vertical and the estimated data volume.

[0017] According to one embodiment of this disclosure, the determining unit is further configured to perform: obtaining the query frequency corresponding to each query condition in the table data based on the historical query records; using the first query condition with the highest query frequency as the sharding key; using the remaining query conditions as second query conditions, and obtaining the range index based on the second query conditions.

[0018] According to one embodiment of this disclosure, the pre-sharding unit is further configured to perform: obtaining a pre-defined sharding interval; obtaining an initial number of shards based on the data volume and the sharding interval; hashing the table data according to the sharding key; and pre-sharding the hashed table data according to the initial number of shards.

[0019] According to one embodiment of this disclosure, the determining unit is further configured to perform: sorting the second query conditions according to the query frequency; selecting the second query conditions within a preset sorting range; and obtaining the range index based on the second query conditions within the preset sorting range.

[0020] According to one embodiment of this disclosure, the second acquisition unit is further configured to perform: extracting at least one query condition from the query request; invoking a router; and routing the query request to the corresponding shard replica set according to the query condition.

[0021] According to one embodiment of this disclosure, the management device further includes: an update module configured to, in response to detecting that the table data update conditions are met, acquire the table data to be updated, and pre-shard and shard the table data to be updated.

[0022] According to a third aspect of the present disclosure, a data management system is provided, including: a sharded cluster, a configuration server replica set, a router, and a data management device as described in the second aspect of the present disclosure.

[0023] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the data management method as described in the first aspect of the present disclosure.

[0024] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of a data management device / electronic device / server, enables the data management device / electronic device / server to perform the data management method as described in the first aspect of the present disclosure.

[0025] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the data management method described in the first aspect of the present disclosure.

[0026] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0027] This disclosure obtains table data and corresponding historical query records, determines the sharding key and range index based on the historical query records, pre-shards the table data according to the sharding key to obtain at least two sharded replica sets for sharded storage, then obtains query requests for the table data sent by the application, calls the query interface according to the query request to route the query request to the corresponding sharded replica set, and finally retrieves the query result corresponding to the query request from the sharded replica set according to the range index and sends the query result to the application. Therefore, this disclosure, based on sharding technology, more rationally shards and stores table data, improving data query response speed and enabling users to obtain query results more quickly, while ensuring accuracy, flexibility, and reliability in the data query process.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a schematic diagram according to the first embodiment of the present disclosure.

[0031] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure.

[0032] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure.

[0033] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure.

[0034] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure.

[0035] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure.

[0036] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure.

[0037] Figure 8 This is a block diagram illustrating a data management system according to an exemplary embodiment.

[0038] Figure 9 This is a block diagram illustrating another data management system according to an exemplary embodiment.

[0039] Figure 10 This is a schematic diagram illustrating a data management system according to an exemplary embodiment.

[0040] Figure 11 This is a block diagram illustrating a data management device according to an exemplary embodiment.

[0041] Figure 12 This is a block diagram illustrating another data management device according to an exemplary embodiment.

[0042] Figure 13 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0044] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0045] Figure 1 This is a flowchart illustrating a data management method according to an exemplary embodiment. It should be noted that the data management method of this disclosure is executed by a data management device. The data management method of this disclosure can be executed by the data management device of this disclosure, which can specifically be a hardware device or software within a hardware device. The hardware device includes, for example, a terminal device or a server.

[0046] like Figure 1 As shown, the data management method proposed in this embodiment includes the following steps:

[0047] In step 101, the table data and the corresponding historical query records are obtained.

[0048] Tabular data refers to any data to be stored that is recorded in tabular form, such as marketing management indicator data, comprehensive business indicator data, etc.

[0049] Among them, historical query records refer to query records for table data. Historical query records include at least the following: query condition parameters and corresponding query frequency. Among them, the query condition parameters include at least one of the following parameters: mobile phone number, ID card number, username, etc.

[0050] It should be noted that this disclosure does not limit the specific method for obtaining table data, and the appropriate method can be selected according to the actual situation. For example, the data structure of the target database table in the database can be obtained first, and table data can be formed based on the data structure. Furthermore, after obtaining the data table, historical query operation instructions triggered by the user can be obtained based on the data table, and then at least one historical query record for the target table data can be formed based on the historical query operation instructions, thereby obtaining the historical query record corresponding to the table data.

[0051] For example, after obtaining table data A, the corresponding historical query records a, b, c, etc. can be retrieved based on the user's query operation command.

[0052] In step 102, the sharding key and range index are determined based on the historical query records.

[0053] The shard key, also known as the shard key, is an index that identifies the cluster shard corresponding to each piece of data. In other words, based on the shard key, data can be written to the matching cluster shard.

[0054] Range indexes are used in databases to speed up table queries.

[0055] It should be noted that this disclosure does not limit the specific method for establishing range indexes, and the appropriate method can be selected according to the actual situation.

[0056] Optionally, other condition parameters besides the condition parameter corresponding to the sharding key can be obtained, and a range index can be built based on the other condition parameters.

[0057] Optionally, other condition parameters besides those corresponding to the sharding key can be obtained, and target remaining condition parameters can be selected from these parameters according to a prediction filtering strategy. A range index can then be established based on these target remaining condition parameters. These target remaining condition parameters can be user-specified parameters or default parameters.

[0058] It should be noted that this disclosure does not limit the specific method for determining the sharding key, and it can be selected according to the actual situation.

[0059] One possible approach is to use the most frequently queried condition as the sharding key. For example, if the most frequently queried condition (parameter) is a mobile phone number, then the mobile phone number could be used as the sharding key.

[0060] As another possible approach, sharding keys can be pre-defined based on different business verticals. For example, for query services such as account details inquiries, ID card numbers or other identification document numbers can be used as sharding keys.

[0061] In step 103, the table data is pre-sharded according to the sharding key to obtain at least two sharded replica sets for sharded storage.

[0062] Pre-sharding refers to a sharding method, where sharding refers to the process of splitting data and distributing it across different machines.

[0063] A sharded replica set refers to a group of processes that synchronize data across multiple machines. Sharded replica sets provide data redundancy, extend data availability, and store data on multiple servers to avoid data loss caused by a single server.

[0064] In this embodiment of the disclosure, after obtaining the sharding key, the table data can be pre-sharded according to the sharding key, and then at least two sharded replica sets can be obtained for sharded storage.

[0065] For example, if the sharding key is a mobile phone number and the table data size is 1000 records, in this case, the 1000 records can be pre-sharded to obtain 5 shard replica sets for sharded storage. In this way, each shard replica set corresponds to 200 records.

[0066] It should be noted that this disclosure does not limit the setting of the number of shards obtained by partitioning table data. For example, a default value can be preset based on attributes such as storage space, and the default value can be used as the number of shards; or, for example, the number of shards can be dynamically determined based on the size of the table data.

[0067] In step 104, the query request for table data sent by the application is obtained, and the query interface is called according to the query request to route the query request to the corresponding shard replica set.

[0068] In this context, an application (APP) refers to any application with query functionality. Generally, an application is a computer program designed to perform one or more specific tasks. It runs in user mode, can interact with the user, and has a visual user interface. For example, it could be a banking application or a shopping application.

[0069] It should be noted that, in this embodiment of the disclosure, relevant personnel (such as users) can send query requests for table data by performing various operations.

[0070] Optionally, a query request for table data can be sent by triggering a target button on the application's interactive interface; alternatively, a query request for table data can be sent by inputting a voice command in a target area on the application's interactive interface.

[0071] Accordingly, query requests sent by relevant personnel through various means can be obtained.

[0072] It should be noted that after obtaining a query request, it can be preprocessed to obtain the corresponding Structured Query Language (SQL) statement, so that the query interface can be invoked according to the query request. SQL is a database query and programming language used to store and retrieve data from a database, and to query, update, and manage the database.

[0073] Furthermore, after calling the query interface, the query request can be routed to the corresponding sharded replica set.

[0074] In step 105, the query results corresponding to the query request are obtained from the sharded replica set according to the range index, and the query results are sent to the application.

[0075] In this embodiment of the disclosure, after routing the query request to the corresponding sharded replica set, the query result corresponding to the query request can be obtained from the sharded replica set according to the range index, and the query result can be sent to the application.

[0076] It should be noted that during the query process, since the data after sharding is distributed out of order, the query results corresponding to the query request can be obtained from the sharded replica set based on the range index.

[0077] Correspondingly, after the query results are sent to the application, users can directly obtain the query results for that request on the application's display interface; users can also click the target button on the application and then obtain the query results for that query request.

[0078] This disclosure obtains table data and corresponding historical query records, determines the sharding key and range index based on the historical query records, pre-shards the table data according to the sharding key to obtain at least two sharded replica sets for sharded storage, then obtains query requests for the table data sent by the application, calls the query interface according to the query request to route the query request to the corresponding sharded replica set, and finally retrieves the query result corresponding to the query request from the sharded replica set according to the range index and sends the query result to the application. Therefore, this disclosure, based on sharding technology, more rationally shards and stores table data, improving data query response speed and enabling users to obtain query results more quickly, while ensuring accuracy, flexibility, and reliability in the data query process.

[0079] It should be noted that the data management method provided in this disclosure is applicable to various database architectures with sharded storage capabilities. The MONGODB architecture will be used as an example below to explain the MONGODB architecture of this disclosure.

[0080] Figure 2 This is a flowchart illustrating another data management method according to an exemplary embodiment.

[0081] like Figure 2 As shown, the data management method proposed in this embodiment includes the following steps:

[0082] It should be noted that before performing step 101 to obtain table data, a MONGODB framework can be built. The specific process includes the following steps S201 to S202.

[0083] Among them, using the MONGODB architecture for data management can significantly improve the user experience. Since the user's learning cost is extremely low, there is no need to understand the sharding of the stored data. The corresponding components in the MONGODB architecture can directly route the query requests sent by the user to the corresponding shard replica set for querying.

[0084] In step 201, the amount of data in the table is obtained.

[0085] It should be noted that when building the MONGODB architecture, the amount of table data can be obtained based on the business vertical and the estimated data volume.

[0086] As one possible implementation, such as Figure 3 As shown, based on the above embodiment, the specific process of obtaining the amount of table data in step S201 includes the following steps:

[0087] In step 301, at least one candidate business vertical and the estimated data volume corresponding to each candidate business vertical are obtained.

[0088] The candidate business categories refer to the application scenarios of the tabular data, which can be set according to the actual situation. For example, business categories can be set to include at least one of the following: navigation, shopping, instant messaging, banking services, etc.

[0089] In step 302, the business vertical corresponding to the table data is obtained, and the data volume of the table data is obtained based on the business vertical and the estimated data volume.

[0090] In this embodiment of the disclosure, after obtaining the business category and estimated quantity corresponding to the table data, the data volume of the table data can be obtained based on the business category and estimated data volume. The business category can be any candidate business category.

[0091] For example, if the business category is banking services and the estimated data volume is 30G, the data volume of the table data can be 40G (not less than the estimated data volume) based on the business category and the estimated data volume.

[0092] In step 202, a sharding strategy is determined based on the amount of data, and at least one master node, slave node, and arbitrator node are configured as nodes of the sharded replica set.

[0093] The sharding strategy refers to configuring the hard drive and memory, as well as the number of machines, based on the amount of data.

[0094] It should be noted that after determining the sharding strategy, at least one master node and one slave node can be configured. For example, one master node 1 and one slave node 1 can be configured; or one master node 1, one slave node 1 and one slave node 2 can be configured, etc.

[0095] Among them, the arbiter node is a special type of node. The arbiter node itself does not store data; its main function is to determine which slave node will be promoted to master after the master node fails. For example, when master node 1 fails, the arbiter node can promote slave node 1 to master, thus ensuring continued business operations. In step 203, the table data and the corresponding historical query records are obtained.

[0096] It should be noted that step 203 is the same as step 101 in the above embodiment, and will not be repeated here.

[0097] In step 204, the sharding key and range index are determined based on the historical query records.

[0098] As one possible implementation, such as Figure 4 As shown, based on the above embodiment, the specific process of determining the sharding key and range index according to historical query records in step S204 includes the following steps:

[0099] In step 401, the query frequency corresponding to each query condition in the table data is obtained based on the historical query records.

[0100] In this embodiment of the disclosure, after obtaining historical query records, the query frequency corresponding to each query condition in the table data can be obtained.

[0101] For example, the query condition for retrieving table data is a mobile phone number, with a query frequency of 100,000 times; the query condition for retrieving table data is a name, with a query frequency of 90,000 times.

[0102] In step 402, the first query condition with the highest query frequency is used as the sharding key.

[0103] In this embodiment of the disclosure, after obtaining the query frequency of each query condition, the query frequencies can be compared, and the first query condition with the highest query frequency can be used as the sharding key.

[0104] For example, the query condition is a mobile phone number, and the query frequency is 100,000 times; the query condition is a name, and the query frequency is 90,000 times; then the mobile phone number with the highest query frequency is used as the sharding key.

[0105] In step 403, the remaining query conditions are used as the second query conditions, and the range index is obtained based on the second query conditions.

[0106] As one possible implementation, such as Figure 5 As shown, based on the above embodiment, the specific process of obtaining the range index according to the second query condition in step S403 includes the following steps:

[0107] In step 501, the second query conditions are sorted according to their query frequency.

[0108] In this embodiment of the disclosure, after obtaining the remaining query conditions of the table data, the second query conditions can be sorted according to the query frequency, for example, in descending order.

[0109] In step 502, a second query condition within a preset sorting range is selected, and a range index is obtained based on the second query condition within the preset sorting range.

[0110] It should be noted that this disclosure does not limit the specific method for selecting the second query condition within the preset sorting range, and the selection can be made according to the actual situation. For example, the top three query conditions with the highest query frequency can be selected as the second query condition; or, for example, the top five query conditions with the highest query frequency can be selected as the second query condition.

[0111] In this embodiment of the disclosure, after obtaining the second query condition, a range index can be obtained based on the second query condition that is within a preset sorting range.

[0112] In step 205, the table data is pre-sharded according to the sharding key to obtain at least two sharded replica sets for sharded storage.

[0113] As one possible implementation, such as Figure 6 As shown, based on the above embodiment, the specific process of pre-sharding the table data according to the sharding key in step S205 includes the following steps:

[0114] In step 601, the pre-defined segmentation interval is obtained.

[0115] It should be noted that, in this embodiment of the disclosure, a pre-defined segmentation range can be obtained by performing various operations. For example, a pre-defined segmentation range can be obtained by triggering a target button on the interactive interface.

[0116] In step 602, the initial number of shards is obtained based on the data volume and sharding interval.

[0117] In this embodiment of the disclosure, after obtaining the data volume and the fragmentation range, the initial number of fragments can be obtained.

[0118] For example, if the amount of data acquired is 30GB and there are 5 partitioned intervals, then the initial number of partitions is 6.

[0119] In step 603, the table data is hashed according to the sharding key, and the hashed table data is pre-sharded according to the initial sharding number.

[0120] Hash processing refers to transforming an input of arbitrary length into an output of fixed length using a hash algorithm.

[0121] In this embodiment of the disclosure, after hashing the table data according to the sharding key to make the data dispersed, the hashed table data is pre-sharded according to the initial sharding number.

[0122] In step 206, the query request for table data sent by the application is obtained, and the query interface is called according to the query request to route the query request to the corresponding shard replica set.

[0123] As one possible implementation, such as Figure 7 As shown, based on the above embodiment, the specific process of calling the query interface according to the query request in step S206 to route the query request to the corresponding shard replica set includes the following steps:

[0124] In step 701, at least one query condition is extracted from the query request.

[0125] It should be noted that after obtaining the query request, at least one query condition can be extracted from the query request, such as a mobile phone number; or a name, etc.

[0126] In step 702, the router is invoked, and the query request is routed to the corresponding shard replica set according to the query conditions.

[0127] It provides an interface between client applications and sharded clusters.

[0128] It should be noted that the sharded replica set only contains a portion of the table data, so a router, such as the mongos router, can be invoked to route the query request to the corresponding sharded replica set based on the query conditions.

[0129] In step 703, the target subset in the sharded replica set is obtained based on the range index and query conditions, and the query request is routed to the target subset.

[0130] It should be noted that after routing the query request to the corresponding sharded replica set based on the query conditions, the target subset in the sharded replica set can be obtained based on the range index and the query conditions, and the query request can be routed to the target subset.

[0131] For example, after routing the query request to the corresponding sharded replica set A based on the query conditions, the target subset A1 in sharded replica set A is obtained based on the range index and the query conditions, and the query request is routed to the target subset A1.

[0132] In step 207, the query results corresponding to the query request are obtained from the sharded replica set according to the range index, and the query results are sent to the application.

[0133] It should be noted that step 207 is the same as step 105 in the above embodiment, and will not be repeated here.

[0134] This disclosure utilizes the MONGODB architecture for data management, significantly improving user experience and query response speed. It also enables sharding for different business verticals, enhancing the flexibility of the data query process while ensuring accuracy.

[0135] Furthermore, in this disclosure, after the query results are sent to the application, the table data can be updated according to the update conditions.

[0136] As one possible implementation, when the table data update condition is detected, the table data to be updated is obtained, and the table data to be updated is pre-sharded and sharded for storage.

[0137] It should be noted that this disclosure does not limit the specific method for obtaining the updated conditions of the table data, and the appropriate method can be selected according to the actual situation.

[0138] Optionally, the table data can be updated at a preset time interval. For example, the preset time interval is 5 days, meaning that the table data needs to be updated every 5 days.

[0139] Optionally, a data volume threshold can be set to update the table data. For example, when the data volume threshold is 100,000 rows, that is, when the amount of data in the table exceeds 100,000 rows, the data table needs to be updated.

[0140] It should be noted that, in order to improve the data writing speed during the process of updating table data, multiple processes can be started to write concurrently.

[0141] Accordingly, after obtaining the table data to be updated, it is necessary to pre-shard and shard the table data to be updated.

[0142] This disclosure, upon detecting that a table data update condition is met, retrieves the table data to be updated and performs pre-sharding and sharded storage on the data. Therefore, this disclosure supports rapid batch updates of data and real-time queries under different conditions, improving the speed and reliability of the data query process.

[0143] Figure 8 A data management system is illustrated according to an exemplary embodiment.

[0144] like Figure 8 As shown, the data management system 20 includes: a sharded cluster 21, a configuration server replica set 22, a router 23, and a data management device 24 as described in the first aspect.

[0145] It should be noted that sharded clusters are operations that store data on multiple machines, and mainly consist of query routing mongos (routers), shards, and configuration servers.

[0146] The data management system constructed in this disclosure, through sharding, configuration server replica sets, routers, and data management devices, enables data querying and sends query results to applications. Therefore, this disclosure supports real-time data querying and rapid response, improving the accuracy and reliability of the data query process.

[0147] Figure 9 A data management system is illustrated according to an exemplary embodiment.

[0148] like Figure 9 As shown, the data management system 20 includes: a sharded cluster 21, a configuration server replica set 22, a router 23, and a data management device 24 as described in the first aspect.

[0149] The sharded cluster 21 is configured to execute a process comprising at least two sharded replica sets, each of which is configured with at least one master node, a slave node, and an arbitrator node.

[0150] It should be noted that a master node is a service (instance) that can store data, index, and search; a slave node replicates the data from the master node to other servers; and an arbiter node is a special type of node that does not store data itself. Its main function is to determine which slave node will be promoted to master if the master node becomes ineffective.

[0151] For example, such as Figure 10 The data management system diagram shown illustrates that after user A inputs a query request, user A does not need to know about the sharded storage within the system. The router directly routes the query request to the configuration server replica set based on the query conditions. Then, based on the range index and query conditions, it obtains the target subset from the configuration server replica set and routes the query request to the target subset. Based on the range index, it retrieves the query result corresponding to the query request from the configuration server replica set and sends the query result to the application. Finally, the user can obtain the query result for the query request in the application's display interface.

[0152] The data management system constructed in this disclosure comprises a sharded cluster, a configuration server replica set, a router, and a data management device 24. The sharded cluster includes at least two sharded replica sets, and each sharded replica set is configured with at least one master node, a slave node, and an arbitrator node.

[0153] Therefore, this disclosure leverages the features of MongoDB to quickly create collections based on business needs and quantity characteristics, supporting rapid batch updates of data and real-time queries with different conditions. Furthermore, based on MongoDB's inherent high-efficiency caching performance, query response speed can be further improved by increasing MongoDB memory in specific business scenarios. It can also support greater concurrent access demands through flexible horizontal scaling of sharded clusters and expansion of slave nodes in replica sets, improving the accuracy, flexibility, and reliability of data querying.

[0154] Figure 11 This is a block diagram illustrating a data management device according to an exemplary embodiment.

[0155] like Figure 11 As shown, the data management device 1100 includes: a first acquisition unit 11, a determination unit 12, a pre-segmentation unit 13, a second acquisition unit 14, and a transmission acquisition unit 15.

[0156] The first acquisition unit 11 is configured to acquire the amount of data in the table; determine a sharding strategy based on the amount of data, and configure at least one master node, slave node, and arbitrator node as nodes in the sharded replica set;

[0157] The determining unit 12 is configured to perform the following operations: based on the historical query records, obtain the query frequency corresponding to each query condition in the table data; use the first query condition with the highest query frequency as the sharding key; use the remaining query conditions as the second query conditions; and obtain the range index based on the second query conditions.

[0158] The pre-sharding unit 13 is configured to perform the following actions: obtaining a pre-defined sharding interval; obtaining an initial number of shards based on the data volume and the sharding interval; hashing the table data according to the sharding key; and pre-sharding the hashed table data according to the initial number of shards.

[0159] The determining unit 12 is configured to sort the second query conditions according to the query frequency; select the second query conditions within a preset sorting range; and obtain the range index based on the second query conditions within the preset sorting range.

[0160] The second acquisition unit 14 is configured to extract at least one query condition from the query request; invoke the router; and route the query request to the corresponding shard replica set according to the query condition.

[0161] This disclosure obtains table data and corresponding historical query records, determines the sharding key and range index based on the historical query records, pre-shards the table data according to the sharding key to obtain at least two sharded replica sets for sharded storage, then obtains query requests for the table data sent by the application, calls the query interface according to the query request to route the query request to the corresponding sharded replica set, and finally retrieves the query result corresponding to the query request from the sharded replica set according to the range index and sends the query result to the application. Therefore, this disclosure, based on sharding technology, more rationally shards and stores table data, improving data query response speed and enabling users to obtain query results more quickly, while ensuring accuracy, flexibility, and reliability in the data query process.

[0162] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0163] Figure 12 This is a block diagram illustrating another data management device according to an exemplary embodiment.

[0164] like Figure 12 As shown, the data management device 1100 includes: a first acquisition unit 11, a determination unit 12, a pre-segmentation unit 13, a second acquisition unit 14, a transmission acquisition unit 15, and an update unit 16.

[0165] The first acquisition unit 11 is configured to acquire the amount of data in the table; determine a sharding strategy based on the amount of data, and configure at least one master node, slave node, and arbitrator node as nodes in the sharded replica set;

[0166] The determining unit 12 is configured to perform the following operations: based on the historical query records, obtain the query frequency corresponding to each query condition in the table data; use the first query condition with the highest query frequency as the sharding key; use the remaining query conditions as the second query conditions; and obtain the range index based on the second query conditions.

[0167] The pre-sharding unit 13 is configured to perform the following actions: obtaining a pre-defined sharding interval; obtaining an initial number of shards based on the data volume and the sharding interval; hashing the table data according to the sharding key; and pre-sharding the hashed table data according to the initial number of shards.

[0168] The determining unit 12 is configured to sort the second query conditions according to the query frequency; select the second query conditions within a preset sorting range; and obtain the range index based on the second query conditions within the preset sorting range.

[0169] The second acquisition unit 14 is configured to extract at least one query condition from the query request; invoke the router; and route the query request to the corresponding shard replica set according to the query condition.

[0170] The update unit 16 is configured to, in response to detecting that the table data update conditions are met, obtain the table data to be updated, and perform pre-sharding and sharding storage on the table data to be updated.

[0171] This disclosure obtains table data and corresponding historical query records, determines the sharding key and range index based on the historical query records, pre-shards the table data according to the sharding key to obtain at least two shard replica sets for sharded storage, then obtains query requests for the table data sent by the application, calls the query interface according to the query request to route the query request to the corresponding shard replica set, finally obtains the query result corresponding to the query request from the shard replica set according to the range index, and sends the query result to the application, and finally, when the table data update condition is detected, pre-shards and shards the updated table data for storage. Therefore, this disclosure, based on sharding technology, more rationally shards and stores table data, improving data query response speed, enabling users to obtain query results more quickly, supporting rapid batch updates of data and real-time queries under different query conditions, while ensuring accuracy, flexibility, and reliability in the data query process.

[0172] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0173] To achieve the above embodiments, this disclosure also provides an electronic device, such as... Figure 13 As shown, the electronic device 8000 includes: a processor 801; and one or more memories 802 for storing executable instructions of the processor 801; wherein the processor 801 is configured to execute the data management method described in the above embodiments. The processor 801 and the memories 802 are connected via a communication bus.

[0174] In an exemplary embodiment, this disclosure also provides a computer-readable storage medium including instructions, such as a memory 802 including instructions, which can be executed by a processor 801 of a data management device to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0175] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, characterized in that the computer program, when executed by a processor, implements the data management method described in the above embodiments.

[0176] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0177] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data management method, characterized in that, include: Retrieve the table data and the corresponding historical query records; Based on the historical query records, obtain the query frequency corresponding to each query condition in the table data; Use the most frequently queried first query condition as the sharding key; The remaining query conditions are used as second query conditions, and the second query conditions are sorted according to their query frequency. Select the second query condition that is within the preset sorting range, and obtain the range index based on the second query condition that is within the preset sorting range; The table data is pre-sharded according to the sharding key to obtain at least two sharded replica sets for sharded storage, and the sharded replica sets contain only a portion of the table data. Obtain the query request sent by the application for the table data, and call the query interface according to the query request to route the query request to the corresponding shard replica set; Based on the range index, the query result corresponding to the query request is obtained from the sharded replica set, and the query result is sent to the application. In response to the detection that the table data update conditions are met, the table data to be updated is obtained, and the table data to be updated is pre-sharded and sharded for storage.

2. The data management method according to claim 1, characterized in that, Before obtaining the table data, the process also includes: Obtain the amount of data in the table; The sharding strategy is determined based on the data volume, and at least one master node, slave node, and arbitrator node are configured as nodes in the sharded replica set.

3. The data management method according to claim 2, characterized in that, The amount of data obtained from the table includes: Obtain at least one candidate business vertical category and the estimated data volume corresponding to each candidate business vertical category; Obtain the business vertical category corresponding to the table data, and obtain the data volume of the table data based on the business vertical category and the estimated data volume.

4. The data management method according to claim 1, characterized in that, The step of pre-sharding the table data according to the sharding key includes: Get the pre-defined segmentation range; Based on the data volume and the fragmentation interval, obtain the initial number of fragments; The table data is hashed according to the sharding key, and the hashed table data is pre-sharded according to the initial sharding number.

5. The data management method according to claim 1, characterized in that, The step of calling the query interface according to the query request to route the query request to the corresponding shard replica set includes: Extract at least one query condition from the query request; Invoke the router and route the query request to the corresponding shard replica set according to the query conditions.

6. The data management method according to claim 5, characterized in that, Also includes: Based on the range index and the query conditions, obtain the target subset in the sharded replica set, and route the query request to the target subset.

7. A data management device, characterized in that, include: The first acquisition unit is configured to acquire table data and the historical query records corresponding to the table data. The determining unit is configured to perform the operation of obtaining the query frequency corresponding to each query condition in the table data based on the historical query records; Use the most frequently queried first query condition as the sharding key; The remaining query conditions are used as second query conditions, and the second query conditions are sorted according to their query frequency. Select the second query condition that is within the preset sorting range, and obtain the range index based on the second query condition that is within the preset sorting range; The pre-sharding unit is configured to perform pre-sharding of the table data according to the sharding key to obtain at least two sharded replica sets for sharded storage. The second acquisition unit is configured to execute a query request for the table data sent by the acquisition application, and to call a query interface according to the query request to route the query request to the corresponding sharded replica set, wherein the sharded replica set contains only a portion of the table data. The sending and obtaining unit is configured to perform the following: obtain the query result corresponding to the query request from the sharded replica set according to the range index, and send the query result to the application. Update module: In response to the detection that the table data update conditions are met, the module obtains the table data to be updated and performs pre-sharding and sharding storage on the table data to be updated.

8. The management device according to claim 7, wherein, The first acquisition unit is further configured to: Obtain the amount of data in the table; The sharding strategy is determined based on the data volume, and at least one master node, slave node, and arbitrator node are configured as nodes in the sharded replica set.

9. The management device according to claim 8, wherein, The first acquisition unit is further configured to: Obtain at least one candidate business vertical category and the estimated data volume corresponding to each candidate business vertical category; Obtain the business vertical category corresponding to the table data, and obtain the data volume of the table data based on the business vertical category and the estimated data volume.

10. The management device according to claim 7, wherein, The pre-segmentation unit is also used for: Get the pre-defined segmentation range; Based on the data volume and the fragmentation interval, obtain the initial number of fragments; The table data is hashed according to the sharding key, and the hashed table data is pre-sharded according to the initial sharding number.

11. The management device according to claim 7, wherein, The second acquisition unit is further configured to: Extract at least one query condition from the query request; Invoke the router and route the query request to the corresponding shard replica set according to the query conditions.

12. The management device according to claim 11, wherein, The second acquisition unit is further configured to: Based on the range index and the query conditions, obtain the target subset in the sharded replica set, and route the query request to the target subset.

13. A data management system, characterized in that, include: Sharded clusters, configuration server replica sets, routers, and the data management apparatus as described in claims 7-12.

14. The data management system according to claim 13, characterized in that, The sharded cluster includes at least two sharded replica sets, and each sharded replica set is configured with at least one master node, a slave node, and an arbitrator node.

15. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data management method as described in any one of claims 1-6.

16. A computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor, implements the data management method as described in any one of claims 1-6.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data management method according to any one of claims 1-6.

Images