A method and device for data management

Abstract

The present invention discloses a method and device for data management, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a query request for a metadata table, the query request including a query input parameter; determining a group identifier and a shard identifier of target data corresponding to the query input parameter according to the query input parameter and a constructed secondary index model; and determining a data storage range of the target data according to the group identifier and the shard identifier. This implementation constructs a secondary index model through a metadata table for retrieval, thereby improving the efficiency of data retrieval and user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data management. Background Art

[0002] ES (ElasticSearch, a distributed, full-text search engine) supports distribution, uses reverse indexes, and provides a rich search API. It can be used as a storage tool for massive data and is widely used in massive data retrieval, aggregation, log analysis and other businesses in the Internet field.

[0003] In the prior art, ES is generally used as a storage engine for data synchronization based on the database to solve the problem of list search caused by database sharding. However, when the data volume is large, the business search scope is wide, and the query scenario is complex, the retrieval efficiency of ES is low and it cannot efficiently support and adapt to business needs. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for data management, which can realize rapid retrieval of data, improve retrieval efficiency, and enhance user experience.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a data management method is provided, comprising:

[0006] Obtaining a query request for a metadata table, wherein the query request includes query input parameters;

[0007] Determine the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model;

[0008] The data storage range of the target data is determined according to the grouping identifier and the fragmentation identifier.

[0009] Optionally, before determining the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model, the method further includes:

[0010] Filtering out index fields from various fields of the metadata table, and constructing a secondary index table according to the index fields;

[0011] The metadata in the metadata table is grouped according to the index fields in the secondary index table, and the metadata in each group is fragmented to construct the secondary index model.

[0012] Optionally, grouping the metadata in the metadata table according to the index field in the secondary index table, and sharding the metadata in each group includes:

[0013] Filter out an index field from the index fields of the secondary index table as a grouping field, and filter out a field from the index fields as a sharding key value;

[0014] Grouping the metadata according to the grouping field to obtain one or more groups;

[0015] Sharding the metadata in each group according to the sharding key value to obtain one or more shards;

[0016] The group identifier of the group and the fragment identifier of the fragment corresponding to each metadata are stored.

[0017] Optionally, after storing the group identifier of the group and the fragment identifier of the fragment corresponding to each metadata, the method further includes:

[0018] The storage path information of each metadata is stored.

[0019] Optionally, before storing the storage path information of each metadata, for any group containing multiple shards, execute:

[0020] Determine that a difference or ratio of the amount of metadata in any two fragments of the group does not exceed a preset threshold,

[0021] When the preset threshold is exceeded, all metadata in the group are hashed and re-sharded, and storage path information of all metadata in the group after re-sharding is stored.

[0022] Optionally, all metadata in the group are hashed and re-sharded, including:

[0023] According to the sharding identifier of the metadata before re-sharding, the metadata is hashed twice to determine the sharding identifier of the metadata after re-sharding, and all metadata in the group are re-sharded according to the result of the secondary hashing.

[0024] Optionally, determining a group identifier and a shard identifier of target data corresponding to the query input parameter includes:

[0025] According to the query input parameter, determine the index field corresponding to the query input parameter from the secondary index model, and determine the grouping field and the shard key value corresponding to the query input parameter according to the index field;

[0026] According to the corresponding grouping field and the sharding key value, a grouping identifier and a sharding identifier of the target data corresponding to the query input parameter are determined.

[0027] According to another aspect of an embodiment of the present invention, there is provided a data management device, comprising:

[0028] An acquisition module is used to acquire a query request for a metadata table, wherein the query request includes query input parameters;

[0029] A query module, which determines the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model;

[0030] A determination module determines a data storage range of the target data according to the grouping identifier and the sharding identifier.

[0031] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0032] one or more processors;

[0033] a storage device for storing one or more programs,

[0034] When the one or more programs are executed by the one or more processors, the one or more processors implement the data management method provided by the present invention.

[0035] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the data management method provided by the present invention is implemented.

[0036] An embodiment of the above invention has the following advantages or beneficial effects: through the secondary index model constructed according to the metadata table, each metadata in the metadata table has a corresponding grouping identifier and sharding identifier, and by obtaining the query input parameter from the query request, combined with the secondary index model, the grouping identifier and sharding identifier of the target data corresponding to the query input parameter are obtained, and then the storage range of the target data is obtained. The data management method of the embodiment of the present invention can solve the problem of low ES retrieval efficiency when the data cardinality is large, the business retrieval range is wide, and the query scenario is complex, realize rapid data retrieval, meet business needs, improve retrieval efficiency, and enhance user experience.

[0037] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0039] Figure 1 is a schematic diagram of the main process of a data management method according to an embodiment of the present invention;

[0040] Figure 2is a schematic diagram of a main process of building a secondary index model according to an embodiment of the present invention;

[0041] Figure 3 is a structural schematic diagram of another data management method according to an embodiment of the present invention;

[0042] Figure 4 is a flow chart of a data management method according to an embodiment of the present invention;

[0043] Figure 5 is a schematic diagram of main modules of a data management device according to an embodiment of the present invention;

[0044] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0045] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0047] Figure 1 is a schematic diagram of the main process of a data management method according to an embodiment of the present invention, such as Figure 1 As shown, the data management method includes:

[0048] Step S101: obtaining a query request for a metadata table, where the query request includes query input parameters;

[0049] Step S102: Determine the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model;

[0050] Step S103: Determine the data storage range of the target data according to the group identifier and the slice identifier.

[0051] The data management method provided in the embodiment of the present invention is to solve the problem that ES cannot achieve fast retrieval when the data base is large, the retrieval range is wide and the query scenario is complex. By expanding the storage and retrieval methods of ES, efficient data retrieval is achieved.

[0052] In an embodiment of the present invention, the query request includes query input parameters, which can be obtained by parsing the query request. The query input parameters can be query fields. In the e-commerce field, the query fields can be order number, order time, product name, bill number and other fields.

[0053] In an embodiment of the present invention, before step 102, it includes: filtering out index fields from various fields of the metadata table, and building a secondary index table based on the index fields; grouping the metadata in the metadata table according to the index fields in the secondary index table, and sharding the metadata in each group to build a secondary index (SecondaryIndex) model.

[0054] By sorting out the retrieval scenarios or historical retrieval situations for metadata in the metadata table, one or more frequently searched fields are selected from various fields in the metadata table as index fields, and a secondary index table is constructed based on the index fields. The index fields can be selected by setting the fields whose retrieval times exceed the preset threshold. The storage range of the corresponding metadata can be queried through one or more index fields. The secondary index table includes index fields and the values ​​of the index fields. Optionally, the index field also includes routing time, and the routing time is used as the index field of the secondary index table so that the metadata can be archived according to time in the future. The routing time can be the order time.

[0055] For example, the fields in the metadata table include: order number, order time, bill number, product code, product name, selling price, tax rate, order account, organization ID and invoice number. By sorting out the metadata retrieval scenarios, the frequently searched fields order number, order account, bill number, organization ID and invoice number are encapsulated as index fields into the secondary index table. Generally, routing time, such as order time, also needs to be encapsulated into the secondary index table to build a secondary index table based on the index fields.

[0056] In an embodiment of the present invention, metadata in a metadata table is grouped according to an index field in a secondary index table, and metadata in each group is sharded, including: selecting an index field from the index field of the secondary index table as a grouping field, selecting a field from the index field as a sharding key value; grouping metadata according to the grouping field to obtain one or more groups; sharding metadata in each group according to the sharding key value to obtain one or more shards; and storing a group identifier of the group where each metadata is located and a shard identifier of the shard where it is located.

[0057] In an optional implementation of the embodiment of the present invention, after storing the group identifier of the group and the fragment identifier of the fragment corresponding to each metadata, the method further includes: storing storage path information of each metadata.

[0058] From the index fields in the secondary index table, filter out a grouping field and a sharding key value, that is, the grouping field is one of the index fields in the secondary index table, and the sharding key value is also a field in the secondary index table. Optionally, the filtering rules can be determined according to the business retrieval and query scenarios, and a reasonable mapping file can be generated by combing the business "read" scenarios to make the filtered grouping fields and sharding key values ​​as suitable as possible for all retrieval fields. It is also possible to perform reasonable data splitting based on the growth rate of existing data and future business development, and splitting can be done by taking into account both retrieval efficiency and retrieval scope to determine the appropriate grouping field for archiving data. Metadata can be obtained based on the sharding key value. For example, you can select routing time as the grouping field to archive metadata, which can be archived by year, month, etc.; when determining the shard key value, if the business query scenario only searches by "order number", you can select "order number" as the shard key value. When the business query scenario is more complex, you can select a field with a larger data dimension as the shard key value. For example, in the field of Internet e-commerce, users often search for data through multiple dimensions such as "order number", "order PIN", and "invoice number". In this case, you can select "organization ID" (a common attribute for all data) as the shard key value to facilitate querying the shard where the data is stored.

[0059] According to the filtered grouping fields, the metadata in the metadata table can be grouped to obtain one or more groups, each group has a corresponding grouping identifier (such as ES.index#0, ES.index#1, ..., ES.index#M, M≥0), the grouping identifier can be determined according to the grouping field, and then the metadata in each group is sharded according to the sharding key value (partitionkey) to obtain one or more shards, each shard has a corresponding sharding identifier (such as Shard0, Shard1, Shard2, ..., ShardN, N≥0), one group corresponds to one or more shards, the grouping identifier and sharding identifier corresponding to each metadata are stored, and the storage path information of each metadata is stored. Among them, the storage path information is the path information of the metadata from the grouping identifier to the sharding identifier, that is, the information of which group and which shard the metadata is stored in, so that when the metadata is retrieved, the shard where the metadata is located is quickly located for access. The group where the metadata is located is the storage structure of the outer index of the secondary index model, and the shard where the metadata is located is the storage structure inside the ES of the secondary index model.

[0060] like Figure 2As shown, a schematic diagram of the main process of building a secondary index model is provided, and the secondary index model includes an ES data storage engine and an ES cluster. The ES data storage model includes: filtering out the routing time (such as order time) from the index field of the secondary index table as the grouping field. If the metadata table is order data, the order time can be used as the index field, so that the order data can be archived according to the order time; then an index field (such as organization ID) is filtered out from the secondary index table as the sharding key value, and the determined grouping field and sharding key value are written into the metadata table and the secondary index table. For example, the order ID, order time and organization ID can be written into the metadata table and the secondary index table. The ES cluster includes: grouping metadata according to the grouping field, for example, archiving metadata by year according to the order time, and the grouping identifiers can be index_2019, index_2020 and index_2021, that is, obtaining the value corresponding to the order time, such as the grouping identifier corresponding to the metadata with the order time in 2019 is index_2019; sharding the metadata in each group according to the institution ID, each group corresponds to multiple shards, and the shard identifiers can be Shard0, Shard1, Shard2,..., ShardN, where N≥0, so that metadata with different institution IDs may correspond to different shard identifiers; storing the group identifier and shard identifier corresponding to each piece of metadata and the storage path information, so that when the metadata is retrieved later, the shard where the metadata is located can be quickly accessed according to the storage path information. For example, if an ES index includes 16 Shards, after storing the path information, each time a metadata search request is made, the shard corresponding to the metadata can be directly located without accessing the 16 Shards, reducing the access times by 15 times. In addition, the search range is theoretically 1 / 16 of the previous one, achieving the purpose of fast search and improving search efficiency.

[0061] In an embodiment of the present invention, before storing the storage path information of each piece of metadata, for any group containing multiple shards, the following is performed: determining that the difference or ratio of the data volume of the metadata in any two shards of the group does not exceed a preset threshold; when the preset threshold is exceeded, all metadata in the group are hashed and re-sharded, and the storage path information of all metadata in the group after re-sharding is stored.

[0062] When a group contains multiple shards, there may be a situation where the amount of metadata data in different shards is uneven, and the uneven distribution of data volume may reduce the retrieval efficiency. In this case, the data in the group can be hashed and then re-sharded to make the distribution of the data volume tend to be uniform. It can be determined whether the difference or ratio of the amount of metadata data in any two shards in the group does not exceed the preset threshold. If it does not exceed the preset threshold, there is no need to re-shard. If it exceeds the preset threshold, all metadata in the group are hashed and re-sharded. The preset threshold can be set according to the business scenario or business needs. For example, the ratio of the amount of metadata data in any two shards cannot exceed the preset threshold (50). If it exceeds 50, re-sharding is required.

[0063] In an embodiment of the present invention, all metadata in a group are hashed and then re-sharded, including: performing a secondary hash on the metadata according to a sharding identifier of the metadata before re-sharding, determining a sharding identifier of the metadata after re-sharding, and re-sharding all metadata in the group according to a result of the secondary hashing.

[0064] After all metadata in the group are hashed, the metadata can be hashed again to re-shard all metadata in the group, determine the shard where the metadata is located after re-sharding, and store the storage path information of the metadata after re-sharding to facilitate quick positioning in subsequent retrieval.

[0065] After hashing all metadata in the group, re-sharding is performed through secondary hashing. By analyzing all metadata in the group, the secondary hash key value (SecondHashKey) and hash width (HashRange) are determined. The secondary hash key value is the field of re-sharding, and the hash width is the width of the metadata hash after re-sharding. The secondary hash key value defaults to ES's _id, and a random UUID can be selected. The hash width is determined according to the distribution of the amount of data sharded by the shard key value. After secondary hashing, the data density of the shard is as close as possible, that is, the amount of data in each shard is evenly distributed. When the data distribution is more even, the hash width is larger, and the retrieval performance is greater, so it is necessary to balance data distribution and hash width.

[0066] According to the secondary hash key value and hash width, as well as the shard identifier of the shard where the new metadata is located after resharding, the shard identifier of the shard where the metadata is located after resharding can be determined. The shard identifier ShardA of the shard where the metadata is located after resharding can be obtained by the following formula: ShardA = hash(_partitionkey)%numPrimaryShards+hash(_SecondHashkey)%HashRange, where hash(_partitionkey)%numPrimaryShards is the shard where the metadata is located before resharding, numPrimaryShards is the number of shards of the group before resharding, and hash(_SecondHashkey)%HashRange is the offset of the metadata.

[0067] For example, when the metadata in a group is divided into 8 Shards (Shard0, Shard1, Shard2, ..., Shard7) according to the institution ID as the shard key value, that is, numPrimaryShards = 8, due to the large difference in the order volume of each institution, the metadata is unevenly distributed in each Shard, and UUID is selected as the secondary hash key value with a hash width of 3. All metadata in the group are hashed and redivided into 3 Shards. If the shard where the metadata is located before re-sharding is Shard2, then hash(_partitionkey)%8=Shard2, offset=hash(_SecondHashkey)%3 is a random number between 0-2, then it can be determined that the shard where the metadata is located after re-sharding is Shard2+offset, that is, the shard location where the metadata is stored.

[0068] By hashing the data in the group and then re-sharding, the data distribution in each shard after re-sharding can be made more uniform than before re-sharding, and the number of shards is appropriately reduced, which can improve the efficiency of retrieval.

[0069] In the embodiment of the present invention, Figure 3 As shown, determining the grouping identifier and sharding identifier of the target data corresponding to the query input parameter includes:

[0070] Step S301: according to the query input parameter, determine the index field corresponding to the query input parameter from the secondary index model;

[0071] Step S302: Determine the grouping field and sharding key value corresponding to the query input parameter according to the index field;

[0072] Step S303: Determine the group identifier and shard identifier of the target data corresponding to the query input parameter according to the corresponding group field and shard key value.

[0073] According to the query input parameter, the index field corresponding to the query input parameter is determined from the secondary index model, and then the grouping field (such as the order time) and the sharding key value (such as the organization ID) are determined. According to the query input parameter, and the grouping field and sharding key value corresponding to the query input parameter, it can be obtained in which shard of which group the target data is stored, that is, the grouping identifier and sharding identifier of the target data are determined, and the grouping identifier and sharding identifier of the target data are encapsulated into Routinginfo (an entity that encapsulates the grouping identifier and sharding identifier), and then the data storage range of the target data is determined according to the grouping identifier and sharding identifier. If the secondary index model is grouped by year and then sharded by organization ID, after the order time and organization ID are determined according to the query input parameter, by aggregating the order time, it can be determined in which group it is in, and by aggregating the organization ID, it can be determined in which shard it is in.

[0074] After receiving the query request, the query input parameter (QueryParam) is identified through the default initialization strategy to determine whether the query input parameter contains the grouping field and sharding key value in the secondary index table. If so, the grouping identifier and sharding identifier corresponding to the query input parameter are determined based on the grouping field and sharding key value, and the grouping identifier and sharding identifier are encapsulated into Routinginfo. The data storage range of the target data is determined from the metadata table based on the grouping identifier and sharding identifier.

[0075] In an embodiment of the present invention, in order to improve the retrieval efficiency, the number of shard key values ​​determined according to the query input parameters is set, that is, a set threshold is set. However, when the number of determined shard key values ​​exceeds the set threshold, part of the data retrieval will be lost. Therefore, in combination with the business scenario, a balance is made between user experience and retrieval efficiency. The user's retrieval range can be set at the front end, such as not being able to search across institutions; or, after determining the grouping fields and shard key values ​​corresponding to the query input parameters, and before determining the grouping identifier and shard identifier of the target data corresponding to the query input parameters, it includes: determining whether the number of shard key values ​​exceeds the set threshold. If not, the target data is retrieved from the metadata table according to the grouping identifier and shard identifier; if exceeded, the shards in Routinginfo are reset, and multi-Shard or full-Shard retrieval is set to avoid data loss and improve user experience.

[0076] like Figure 4The flowchart of a method for data management according to an embodiment of the present invention is shown, wherein the secondary index model includes M groups, and the group identifiers are ES.index#0, ES.index#1, ..., ES.index#i, ..., ES.index#M, and ES.index#i includes N shards, and the shard identifiers are Shard0, Shard1, ..., Shardi, ..., ShardN. When a user sends a Query (query request) to the application layer (such as a client), wherein the query request includes a query input parameter, the application layer calls the ES Client (ES client), and according to the secondary index model, determines that the group identifier corresponding to the query input parameter is ES.index#i, and the corresponding shard identifier is Shardj, and according to the group identifier and shard identifier corresponding to the query input parameter, determines the data storage range of the target data corresponding to the query input parameter, that is, determines that the target data is stored in Shardj of ES.index#i, and then obtains the target data from the metadata table according to the data storage range, and returns the target data to the user, completing the query retrieval process.

[0077] The data management method provided by the embodiment of the present invention determines the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter by constructing a secondary index model, and then determines the data storage range of the target data. The method provided by the embodiment of the present invention can improve the efficiency of data retrieval and thus improve the user experience in the case of large data cardinality, wide retrieval range and complex retrieval scenarios.

[0078] According to another aspect of the embodiment of the present invention, Figure 5 As shown, a data management device 500 is provided, comprising:

[0079] The acquisition module 501 acquires a query request for a metadata table, where the query request includes query input parameters;

[0080] The query module 502 determines the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model;

[0081] The determination module 503 determines the data storage range of the target data according to the grouping identifier and the fragmentation identifier.

[0082] In an embodiment of the present invention, the data management device 500 also includes: a construction module, which filters out index fields from various fields of the metadata table and constructs a secondary index table based on the index fields; groups the metadata in the metadata table according to the index fields in the secondary index table, and slices the metadata in each group to construct a secondary index model.

[0083] In an embodiment of the present invention, the construction module is further used to: select an index field from the index fields of the secondary index table as a grouping field, and select a field from the index fields as a sharding key value; group the metadata according to the grouping field to obtain one or more groups; shard the metadata in each group according to the sharding key value to obtain one or more shards; store the group identifier of the group where each metadata is located and the shard identifier of the shard where it is located.

[0084] In the embodiment of the present invention, the construction module is further used to: store storage path information of each metadata.

[0085] In an embodiment of the present invention, the construction module is further used to: before storing the storage path information of each metadata, for any group containing multiple shards, perform: determining that the difference or ratio of the data volume of the metadata in any two shards of the group does not exceed a preset threshold; when the preset threshold is exceeded, hashing all the metadata in the group and re-sharding them, and storing the storage path information of all the metadata in the group after re-sharding.

[0086] In an embodiment of the present invention, the construction module is further used to: perform secondary hashing on the metadata according to the sharding identifier of the metadata before re-sharding, determine the sharding identifier of the metadata after re-sharding, and re-shard all metadata in the group according to the result of the secondary hashing.

[0087] In an embodiment of the present invention, the query module 502 is further used to: determine the index field corresponding to the query input parameter from the secondary index model according to the query input parameter, and determine the grouping field and sharding key value corresponding to the query input parameter according to the index field; determine the grouping identifier and sharding identifier of the target data corresponding to the query input parameter according to the corresponding grouping field and sharding key value.

[0088] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data management method of an embodiment of the present invention.

[0089] Another aspect of the embodiments of the present invention provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the data management method of the embodiments of the present invention is implemented.

[0090] Figure 6 An exemplary system architecture 600 is shown to which the data management method or data management apparatus according to the embodiment of the present invention can be applied.

[0091] like Figure 6As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0092] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0093] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0094] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process received query requests and other data, and feed back target data (such as product information - only an example) corresponding to the query request to the terminal device.

[0095] It should be noted that the data management method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data management device is generally set in the server 605 .

[0096] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0097] Reference below Figure 7 Schematic diagram of the structure. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0098] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0100] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0101] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a query module and a determination module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the acquisition module may also be described as a "module for acquiring a query request for a metadata table".

[0104] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining a query request for a metadata table, the query request includes a query input parameter; determining a group identifier and a shard identifier of target data corresponding to the query input parameter based on the query input parameter and the constructed secondary index model; determining a data storage range of the target data based on the group identifier and the shard identifier.

[0105] According to the technical solution of the embodiment of the present invention, through the secondary index model constructed according to the metadata table, each metadata in the metadata table has a corresponding grouping identifier and sharding identifier, and by obtaining the query input parameter from the query request, combined with the secondary index model, the grouping identifier and sharding identifier of the target data corresponding to the query input parameter are obtained, and then the storage range of the target data is obtained. The data management method of the embodiment of the present invention can solve the problem of low ES retrieval efficiency when the data cardinality is large, the business retrieval range is wide, and the query scenario is complex, and realize rapid data retrieval, meet business needs, improve retrieval efficiency, and enhance user experience.

[0106] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data management, characterized in that: include: Obtaining a query request for a metadata table, wherein the query request includes query input parameters; Determine the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model; Determine the data storage range of the target data according to the group identifier and the fragment identifier; Among them, before determining the grouping identifier and sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model, it also includes: screening out index fields from various fields of the metadata table, and constructing a secondary index table according to the index fields; grouping the metadata in the metadata table according to the index fields in the secondary index table, and sharding the metadata in each group to construct the secondary index model.

2. The method according to claim 1, characterized in that The metadata in the metadata table is grouped according to the index field in the secondary index table, and the metadata in each group is fragmented, including: Filter out an index field from the index fields of the secondary index table as a grouping field, and filter out a field from the index fields as a sharding key value; Grouping the metadata according to the grouping field to obtain one or more groups; Sharding the metadata in each group according to the sharding key value to obtain one or more shards; The group identifier of the group and the fragment identifier of the fragment corresponding to each metadata are stored.

3. The method according to claim 2, characterized in that After storing the group identifier of the group and the fragment identifier of the fragment corresponding to each metadata, the method further includes: The storage path information of each metadata is stored.

4. The method according to claim 3, characterized in that Before storing the storage path information for each piece of metadata, for any group containing multiple shards, execute: Determine that a difference or ratio of the data amounts of the metadata in any two fragments of the group does not exceed a preset threshold, When the preset threshold is exceeded, all metadata in the group are hashed and re-sharded, and storage path information of all metadata in the group after re-sharding is stored.

5. The method according to claim 4, characterized in that All metadata in the group is hashed and re-sharded, including: According to the sharding identifier of the metadata before re-sharding, the metadata is hashed twice to determine the sharding identifier of the metadata after re-sharding, and all metadata in the group are re-sharded according to the result of the secondary hashing.

6. The method according to claim 2, characterized in that Determining the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter includes: According to the query input parameter, determine the index field corresponding to the query input parameter from the secondary index model, and determine the grouping field and the shard key value corresponding to the query input parameter according to the index field; According to the corresponding grouping field and the sharding key value, a grouping identifier and a sharding identifier of the target data corresponding to the query input parameter are determined.

7. A data management device, characterized in that: include: An acquisition module is used to acquire a query request for a metadata table, wherein the query request includes query input parameters; A query module, which determines the grouping identifier and the sharding identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model; A determination module, determining a data storage range of the target data according to the grouping identifier and the sharding identifier; The query module is further used to: before determining the grouping identifier and the shard identifier of the target data corresponding to the query input parameter according to the query input parameter and the constructed secondary index model, filter out the index field from each field of the metadata table, and construct the secondary index table according to the index field; The metadata in the metadata table is grouped according to the index fields in the secondary index table, and the metadata in each group is fragmented to construct the secondary index model.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images