Data search method, device and electronic equipment

Abstract

The application provides a data search method and device and electronic equipment, and relates to the technical field of data processing. The method comprises the following steps: obtaining a data search instruction, wherein the data search instruction comprises a search condition; obtaining a plurality of index fields of a preset database, each index field comprising index content and a data identifier matched with the index content; determining, according to the search condition and the index content of the plurality of index fields, an index field corresponding to the index content matched with the search condition as a target index field; and obtaining, according to a target data identifier in the target index field, target data corresponding to the target data identifier from the preset database. The application can optimize the search query performance of a wide table in a relational database through the index mode.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data search method, apparatus, and electronic device. Background Technology

[0002] Wide tables are database tables that link together metrics, dimensions, and attributes related to a business theme. In big data warehouse applications, wide table models are widely used due to their advantages such as simple structure, easy-to-understand model, distributed data storage, and efficient access.

[0003] To meet the search and query needs of enterprises with large amounts of data, distributed computing and querying are usually carried out using data warehouses such as HBase, Hive, Presto, and Hudi, which are big data systems.

[0004] However, due to the low performance of search queries on large, wide tables in relational databases, it is difficult to optimize search query performance through indexes. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a data search method, apparatus, and electronic device, so as to optimize the search performance of wide tables in relational databases through indexing.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a data search method, the method comprising:

[0008] Obtain data search instructions, wherein the data search instructions include: search criteria;

[0009] Obtain multiple index fields from a preset database, each index field including: index content, and a data identifier that matches the index content;

[0010] Based on the search criteria and the index content of the multiple index fields, the index field corresponding to the index content that matches the search criteria is determined as the target index field;

[0011] Based on the target data identifier in the target index field, the target data corresponding to the target data identifier is obtained from the preset database.

[0012] Optionally, obtaining multiple index fields from a preset database includes:

[0013] Based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents;

[0014] Based on the data identifiers of the business content corresponding to the same index content, determine the set of data identifiers corresponding to each index content;

[0015] Based on the multiple index contents and the data identifier set corresponding to the multiple index contents, the multiple index fields are generated respectively.

[0016] Optionally, based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents, including:

[0017] If the type of the business field is an enumeration type, multiple enumeration business contents are numerically processed to obtain at least one first index value;

[0018] The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes:

[0019] Based on the data identifier of the enumerated business content corresponding to each first index value, determine the first data identifier set corresponding to each first index value;

[0020] The index field corresponding to each first index value includes: each first index value and a first data identifier set corresponding to each first index value.

[0021] Optionally, determining the first set of data identifiers corresponding to the first index value based on the data identifiers of the enumerated service content corresponding to the first index value includes:

[0022] Based on the number of enumerations corresponding to the business fields of the enumeration type, a hash transformation is performed on the data identifiers of all enumeration business contents corresponding to the first index value to determine the first data identifier set.

[0023] Optionally, based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents, including:

[0024] If the type of the business field is numeric, multiple numeric business contents are processed into data blocks to obtain multiple first index data blocks;

[0025] The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes:

[0026] Based on the data identifier of the numerical service content corresponding to each first index data block, determine the second data identifier set corresponding to each first index data block;

[0027] The index fields corresponding to the plurality of first index data blocks include: the plurality of first index data blocks and the second data identifier set corresponding to each first index data block.

[0028] Optionally, based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents, including:

[0029] If the business field is of type text, multiple text business contents are segmented to obtain multiple index words;

[0030] The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes:

[0031] Based on the data identifier of the text business content corresponding to each index term, determine the third data identifier set corresponding to each index term;

[0032] The index field corresponding to each index term includes: each index term and the third data identifier set corresponding to each index term.

[0033] Optionally, based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents, including:

[0034] If the business field is of type date, then the multiple date business contents should be split separately.

[0035] The split date business content is numerically processed to obtain at least one second index value;

[0036] The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes:

[0037] Based on the data identifier of the date business content corresponding to each second index value, determine the fourth data identifier set corresponding to each second index value;

[0038] The index field corresponding to each second index value includes: each second index value and the fourth data identifier set corresponding to each second index value.

[0039] Optionally, based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents, including:

[0040] If the type of the business field is date, multiple date business contents are processed into data blocks to obtain multiple second index data blocks;

[0041] The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes:

[0042] Based on the data identifier of the date business content corresponding to each second index data block, determine the fifth data identifier set corresponding to each second index data block;

[0043] The index fields corresponding to the plurality of second index data blocks include: the plurality of second index data blocks and the fifth data identifier set corresponding to each second index data block.

[0044] Secondly, embodiments of this application also provide a data search device, the device comprising:

[0045] The search instruction acquisition module is used to acquire data search instructions, which include: search conditions;

[0046] The index field acquisition module is used to acquire multiple index fields of a preset database. Each index field includes: index content and a data identifier that matches the index content.

[0047] The index field matching module is used to determine the index field corresponding to the search content that matches the search conditions as the target index field based on the search conditions and the index content of the multiple index fields.

[0048] The data search module is used to retrieve the target data corresponding to the target data identifier from the preset database based on the target data identifier in the target index field.

[0049] Optionally, the index field retrieval module includes:

[0050] The index content generation unit is used to preprocess the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents;

[0051] The data identifier determination unit is used to determine the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content.

[0052] The index field generation unit is used to generate the multiple index fields respectively based on the multiple index contents and the data identifier set corresponding to the multiple index contents.

[0053] Optionally, the index content generation unit is specifically used to perform numerical processing on multiple enumerated business contents if the type of the business field is an enumeration type, to obtain at least one first index value.

[0054] The data identifier determination unit is specifically used to determine the first data identifier set corresponding to each first index value based on the data identifier of the enumerated business content corresponding to each first index value; wherein, the index field corresponding to each first index value includes: each first index value and the first data identifier set corresponding to each first index value.

[0055] Optionally, the data identifier determination unit is specifically used to perform a hash transformation on the data identifiers of all enumerated business contents corresponding to the first index value according to the enumeration quantity corresponding to the business field of the enumeration type, and determine the first data identifier set.

[0056] Optionally, the index content generation unit is specifically used to process multiple numeric business contents into multiple first index data blocks if the type of the business field is a numeric type.

[0057] The data identifier determination unit is specifically used to determine the second data identifier set corresponding to each first index data block based on the data identifier of the numerical service content corresponding to each first index data block; wherein, the index fields corresponding to the plurality of first index data blocks include: the plurality of first index data blocks and the second data identifier set corresponding to each first index data block.

[0058] Optionally, the index content generation unit is specifically used to perform word segmentation on multiple text business contents to obtain multiple index words if the type of the business field is text.

[0059] The data identifier determination unit is specifically used to determine the third data identifier set corresponding to each index term based on the data identifier of the text business content corresponding to each index term; wherein, the index field corresponding to each index term includes: each index term and the third data identifier set corresponding to each index term.

[0060] Optionally, the index content generation unit is specifically used to split multiple date business contents separately if the type of the business field is date type; and to perform numerical processing on the split date business contents to obtain at least one second index value;

[0061] The data identifier determination unit is specifically used to determine the fourth data identifier set corresponding to each second index value based on the data identifier of the date business content corresponding to each second index value; wherein, the index field corresponding to each second index value includes: each second index value and the fourth data identifier set corresponding to each second index value.

[0062] Optionally, the index content generation unit is specifically used to process multiple date business contents into multiple second index data blocks if the type of the business field is date type;

[0063] The data identifier determination unit is specifically used to determine the fifth data identifier set corresponding to each second index data block based on the data identifier of the date business content corresponding to each second index data block; wherein, the index fields corresponding to the plurality of second index data blocks include: the plurality of second index data blocks and the fifth data identifier set corresponding to each second index data block.

[0064] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the data search method as described in any of the first aspects.

[0065] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data processing method as described in any of the first aspects.

[0066] The beneficial effects of this application are:

[0067] The data search method, apparatus, and electronic device provided in this application determine the target index field corresponding to the index content matching the search conditions based on the index content in the index field and the data identifier that matches the index content. Then, based on the data identifier in the target search field, the corresponding data is retrieved from the database. Since the index field stores the index content and data identifier corresponding to the data, compared with composite indexes, there is no need to consider the order of search query conditions or the syntax of SQL statements, avoiding full table searches due to index failure. Furthermore, the index field, including the index content and the data identifier that matches the index content, can basically cover all search conditions, avoiding full table searches due to the index's inability to cover search conditions, thus improving search efficiency. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 Flowchart of the data search method provided in the embodiments of this application Figure 1 ;

[0070] Figure 2 Flowchart of the data search method provided in the embodiments of this application Figure 2 ;

[0071] Figure 3 Flowchart of the data search method provided in the embodiments of this application Figure 3 ;

[0072] Figure 4 This is a diagram illustrating the index field corresponding to the first index value.

[0073] Figure 5 Flowchart of the data search method provided in the embodiments of this application Figure 4 ;

[0074] Figure 6 Flowchart of the data search method provided in the embodiments of this application Figure 5 ;

[0075] Figure 7 Flowchart of the data search method provided in the embodiments of this application Figure 6 ;

[0076] Figure 8 Flowchart of the data search method provided in the embodiments of this application Figure 7 ;

[0077] Figure 9 A schematic diagram illustrating the principle of generating index fields provided in this application embodiment;

[0078] Figure 10 A schematic diagram of the structure of the data search device provided in the embodiments of this application;

[0079] Figure 11 A schematic diagram of an electronic device provided in an embodiment of the application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0081] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0082] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0084] Existing technologies for index queries on large, wide tables in relational databases primarily utilize leftmost matching of composite indexes. This involves constructing composite indexes on the query condition fields and placing the fields of the query range at the end of the composite index. Furthermore, optimization of the Structured Query Language (SQL) is necessary to ensure that the database query engine's query plan applies the indexes as much as possible, avoiding full table searches and reducing query costs.

[0085] However, the leftmost matching of a composite index has the following drawbacks:

[0086] First, because users design numerous search conditions when performing data searches, it is impossible to guarantee that indexes will be created for all search condition fields. Furthermore, search conditions are dynamic, and the condition fields used in each search query are different. This can cause the database query engine to be unable to use indexed fields for indexed queries and instead perform a full table search. It is impossible to guarantee that all queries can avoid the drawbacks of a full table search, resulting in low search efficiency in some search scenarios.

[0087] Secondly, composite indexes only support range queries on a single field, and the range query condition field must be placed at the end of the composite index and the SQL statement. It is necessary to fully consider each field of the WHERE condition and the ORDER BY sorting field, making SQL statement optimization more difficult. When the query condition contains two or more range query conditions, the composite index will become ineffective, causing the query to require a full table scan.

[0088] Please refer to Figure 1 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the method may include:

[0089] S10: Obtain data search instructions, which include search criteria.

[0090] In this embodiment, the preset data management system consists of a preset data management client and a preset database. The preset data management client serves as the front end and can provide users with a data search and query server. The preset database is a relational database, which uses multiple wide tables with relationships to store data.

[0091] Users input search criteria in a preset data client to generate data search instructions based on the search criteria. The search criteria may include one or more search criteria. For example, the search criteria may include at least one of the following: "Top percentile" search criteria, "TopN" search criteria, "like" fuzzy search criteria, "=" exact search criteria, "!=" non-search criteria, ">", ">=", "<", "<=" range search criteria, "[], [)", "(], "()" interval search criteria, "in" including search criteria, "not in" not including search criteria, etc.

[0092] S20: Obtain multiple index fields from a preset database. Each index field includes: index content and a data identifier that matches the index content.

[0093] In this embodiment, an index field is created in advance for the data stored in the data wide table of a preset database. The index field consists of index content and data identifiers that match the index content. The index content corresponds to the data stored in the data wide table, and the data identifiers correspond to the list identifiers of the data in the data wide table. Specifically, the data identifiers that match the index content are the set of list identifiers of the same data in the data wide table.

[0094] S30: Based on the search criteria and the index content of multiple index fields, determine the index field corresponding to the index content that matches the search criteria as the target index field.

[0095] In this embodiment, the index field corresponding to the index content that meets the search conditions is determined as the target index field based on the index content in multiple index fields.

[0096] In some embodiments, if the search criteria include multiple criteria, multiple target index fields corresponding to the search content that satisfies the multiple search criteria can be determined based on the index content in the multiple index fields.

[0097] S40: Based on the target data identifier in the target index field, retrieve the target data corresponding to the target data identifier from the preset database.

[0098] In this embodiment, an SQL statement is generated based on the target data identifier in the target index field. The SQL statement is then used to retrieve the target data corresponding to the target data identifier from a preset database, so as to display the target data on a preset data management client.

[0099] For example, the SQL statement can be represented as follows:

[0100] select c1,c2,c3,…from tableT where id in(i1,i2,…,iN)

[0101] In some embodiments, if the search criteria include multiple search criteria, the data search instruction further includes: the relationship between the multiple search criteria, such as an AND relationship to indicate the intersection of the results corresponding to the multiple search criteria, an OR relationship to indicate the union of the results corresponding to the multiple search criteria, and, based on the relationship between the multiple search criteria, the intersection or union of the target data identifiers in the multiple target search fields is performed and then duplicates are removed to obtain the target data identifier result set result_list_all{id1,id2,…}.

[0102] In some possible implementations, if the data search instruction includes pagination to retrieve the Top N search criteria, the total number (totalNum) is determined by statistically analyzing the result set result_list_all{id1,id2,…} of the target data identifier. If the total number (totalNum) is less than or equal to the Top N, the corresponding SQL statement is directly generated based on the result set result_list_all{id1,id2,…} of the target data identifier.

[0103] If the total number of items (totalNum) is greater than the number of items (TopN), then it is necessary to iterate through the result set result_list_all{id1,id2,…} of the target data identifier in order to obtain the result set result_list_top_n{id1,id2,…,idN} corresponding to the number of items (TopN).

[0104] In some implementations, if the target data identifier result set result_list_all{id1,id2,…} is very large, iterating through it to obtain the Top N result set may be slow, requiring optimization of the sorting algorithm. This can be achieved by splitting the target data identifier result set result_list_all{id1,id2,…} into multiple result subsets, performing multi-threaded hash sorting on these subsets to determine the Top N results for each thread, and finally, having the main thread integrate the Top N results from each thread to obtain the final result set result_list_top_n{id1,id2,…,idN}.

[0105] It should be noted that the data identifiers in the wide data table are clustered indexes. SQL statements are used to query the index, and the database query engine reads the target data corresponding to the data identifier based on the disk data location. The target data is then returned to the preset data management client for display by the user.

[0106] Based on the result set result_list_top_n{id1,id2,…,idN} corresponding to TopN, when querying data from the database through the database query engine, only a limited number of N target data entries need to be queried, and the query of the target data can be completed through at most N IO reads.

[0107] In some embodiments, only the list identifier is used as an index in the wide data table, and the index fields corresponding to the above data are all stored in memory. When the search conditions do not include the identifier search conditions, the data search method provided in this embodiment can achieve pure memory query, and disk I / O can achieve zero overhead.

[0108] The data search method provided in the above embodiments determines the target index field corresponding to the index content that matches the search conditions based on the index content in the index field and the data identifier that matches the index content. Then, it retrieves the corresponding data from the database according to the data identifier in the target search field. Since the index field stores the index content and data identifier corresponding to the data, compared with composite indexes, it eliminates the need to consider the order of search query conditions or the syntax of SQL statements, avoiding full table searches due to index failure. Furthermore, the index field, which includes the index content and the data identifier that matches the index content, can essentially cover all search conditions, preventing full table searches due to the index's inability to cover search conditions and improving search efficiency.

[0109] Furthermore, the data search method provided in this embodiment only requires the use of a database search engine and in-memory index, without the need for additional reliance on Redis caching, Elasticsearch search, HBase storage, database sharding plugins, or big data distributed query computing frameworks, which can effectively reduce the resource occupancy rate and operation and maintenance costs of the basic platform.

[0110] The following describes one possible implementation method for obtaining the index field as described above, with reference to an example.

[0111] Please refer to Figure 2 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 2 ,like Figure 2 As shown, the process of obtaining multiple index fields of a preset database in S20 above may include:

[0112] S201: Based on the types of multiple business fields in the data wide table of the preset database, preprocess the business content corresponding to each business field to obtain multiple index contents.

[0113] S202: Determine the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content.

[0114] S203: Generate multiple index fields based on multiple index contents and the data identifier set corresponding to the multiple index contents.

[0115] In this embodiment, there are multiple business fields in the data wide table. Each business field has a corresponding field type or field attribute. The type of business content recorded in each business field matches the field type or field data corresponding to each business field.

[0116] To index the business content in the wide data table, it is necessary to generate index content corresponding to the business content. Specifically, for each business field type, the business content is preprocessed using a preprocessing method corresponding to the business field type to obtain the corresponding index content.

[0117] Among them, for different types of business fields, there may be the same business content that is preprocessed into the same index content, or different business content that is preprocessed into the same index content.

[0118] Based on the index content corresponding to the business content, identify the business content with the same index content, generate a set of data identifiers for the data identifiers of the business content with the same index content, and generate an index field based on the index content and the set of data identifiers.

[0119] The data search method provided in the above embodiments preprocesses the business content of different business fields into corresponding index content based on the types of different business fields. It constructs index fields based on the data identifier set of the business content corresponding to the same index content and the index content, so that all search conditions can match the target search field in the index fields, thereby improving the efficiency of data search query and eliminating the need to consider the order of search conditions and the optimization of SQL statements.

[0120] In one possible implementation, please refer to Figure 3 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 3 ,like Figure 3 As shown, the process of S201 above, which preprocesses the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents, may include:

[0121] S211: If the type of the business field is an enumeration type, perform numerical processing on multiple enumeration business contents to obtain at least one first index value.

[0122] In this embodiment, the enumeration type is used to indicate that a business field has only a few possible contents. Each enumerated business content is one of the possible contents. Since the enumerated business content has only a limited number of possible contents, it can be numerically processed and represented by a first index value. The number of first index values ​​is consistent with the number of possible contents of the enumerated business content.

[0123] For example, enumerated business fields can be gender, investment type, risk level, province / city, education level, etc. The enumerated business content for gender is male and female, and the first index value obtained through numerical processing can be 1 and 2 (other numerical identifiers can also be used). The enumerated business content for investment type can be A, B, C, and the first index value obtained through numerical processing can be 1, 2, 3.

[0124] It should be noted that the index values ​​used when converting the enumerated business content corresponding to the business fields of each enumeration type to numerical values ​​must be different in order to distinguish between different business fields.

[0125] The process of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content in S202 above may include:

[0126] S212: Determine the first data identifier set corresponding to each first index value based on the data identifier of the enumerated business content corresponding to each first index value.

[0127] The index field corresponding to each first index value includes: each first index value and the first data identifier set corresponding to each first index value.

[0128] In this embodiment, after numerical processing of the enumerated business content, for each first index value, a first data identifier set consisting of the data identifiers of the enumerated business content corresponding to each first index value is determined. Based on the first index value and the first data identifier set corresponding to the first index value, an index field corresponding to each first index value is generated.

[0129] For example, if the first index value is 1, then the first data identifier set includes: all data identifiers in the data wide table whose gender is male.

[0130] The data search method provided in the above embodiments converts enumerated business content into a first index value, and generates an index field based on the first index value and the first data identifier set of the enumerated business content corresponding to the first index value. This allows the matching first data identifier set to be determined directly from the corresponding index field when the search conditions include enumerated business content, thereby improving data search efficiency. Furthermore, converting enumerated business content into a first index value to construct the index field can effectively reduce the memory usage of the original content.

[0131] In some embodiments, the process of determining the first data identifier set corresponding to each first index value based on the data identifier of the enumerated service content corresponding to each first index value in S212 may include:

[0132] Based on the number of enumerations corresponding to the business fields of the enumeration type, perform a hash transformation on the data identifiers of all enumeration business contents corresponding to the first index value to determine the first data identifier set.

[0133] In this embodiment, the enumeration count is used to indicate the number of possible contents contained in the business field. For example, the enumeration count of the gender business field is 2, and the enumeration count of the provincial administrative region business field is 34.

[0134] Based on the number of enumerations of different business fields, the data identifiers of the enumerated business content corresponding to the first index value are hashed to construct the first data identifier set after hash transformation.

[0135] In some embodiments, if the number of enumerations is less than a preset threshold, a type A index is used to perform a HashMap index on the first data identifier set. The hash-transformed first data identifier set is represented as a bit array, where the length of the bit value is the data size of the wide table. Representing the data identifier in int format as a bit array can reduce the memory usage of the index field and improve the computational efficiency of intersection, union, and difference when performing data searches. For example, the preset threshold can be 32.

[0136] For example, please refer to Figure 4 This is a diagram illustrating the index field corresponding to the first index value, such as... Figure 4 As shown, the index field corresponding to the first index value includes the first index value key and the hash-transformed first data identifier set value. The value represents the data identifier of the enumeration business content corresponding to the first index value key in the data wide table in the form of a bit array. For example, if the fifth bit in the bit array is 0, it means that the first index value corresponding to the enumeration business content with data identifier 5 in the data wide table is not the first index value corresponding to the index field. If the seventh bit in the bit array is 1, it means that the first index value corresponding to the enumeration business content with data identifier 7 in the data wide table is the first index value corresponding to the index field.

[0137] In some possible implementations, when the number of data sets corresponding to each enumeration variable is equal to or close to the average data volume in the wide table, if a bit array is directly used to generate the index field, the memory occupied by all index fields corresponding to that business field will be the number of enumerations multiplied by the data volume in the wide table, resulting in severe memory waste. For example, if the data volume in the wide table is 100 million, the number of enumerations is 10, and the number of data sets for each enumeration variable is 10 million, then the memory occupied by all index fields will be 10 * 100 million bits, or 1 billion bits.

[0138] Based on this, the first data identifier set corresponding to the index field can be represented by an auxiliary bit value plus an optimized bit array. The auxiliary bit value records the data identifier of the enumerated business content corresponding to the first index value in the wide table, and the optimized bit value includes the number of 1s corresponding to the data set of the first index value. For example, if the auxiliary bit array is [100, 200, 300] and the optimized bit array is [1, 1, 1], multiplying the auxiliary bit value and the optimized bit array determines that the data identifiers with the first index value are 100, 200, and 300.

[0139] In some embodiments, when the number of preset sets corresponding to the enumeration variable is less than or equal to the preset set number threshold, using a bit array to store the first data identifier set will result in a huge waste of memory. Therefore, an int value can be used to identify the first data identifier set to save memory.

[0140] For example, with an enumeration value of 1, the first data identifier set has only three data identifiers: {1, 500w, 1000w}. Since the largest data identifier is 1000w, it requires 1000w bits of memory, which is approximately 37M of memory space. Therefore, the first data identifier set can be directly identified by an int array [1, 500w, 1000w]. The length of this int array is 3, which only requires 96 bits of memory space.

[0141] In other embodiments, if the number of enumerations is greater than or equal to a preset number threshold, a B-type index method is used to perform a HashSet transformation on the first data identifier set, and the data identifiers in the first data identifier set are transformed by HashSet and stored in the order of hash values. When performing a search query, the time advantage of the time complexity O(1) of HashSet search can be fully utilized.

[0142] The data search method provided in the above embodiments performs a hash transformation on the data identifiers based on the number of enumerations to obtain a first set of data identifiers. This can reduce the memory usage of the index fields and improve search efficiency.

[0143] In one possible implementation, please refer to Figure 5 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 4 ,like Figure 5 As shown, the process of S201 above, which preprocesses the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents, may include:

[0144] S213: If the business field is of numeric type, perform data block processing on multiple numeric business contents to obtain multiple first index data blocks.

[0145] In this embodiment, the numeric type is used to indicate that the content of the business field is represented by a numerical value, such as holding share, holding market value, management scale, and other business fields. To improve search efficiency, the numeric business content in the numeric type business field can be sorted numerically before data block processing is performed. That is, the sorted numeric business content is divided into equal intervals to obtain multiple first index data blocks, each index data block including at least one numeric business content.

[0146] In some embodiments, the data block processing method can be as follows: the sorted numerical business content is divided into multiple equal intervals to obtain multiple first index data blocks. For example, the sorted numerical business content is divided into two intervals to obtain two first index data blocks. Then, the numerical business content in the two first index data blocks is divided into two intervals respectively, and each first index data block also obtains two first index data blocks. This process is repeated to obtain multiple first index data blocks.

[0147] It should be noted that, in this embodiment, the business fields of the numeric type do not include the business fields of the date type.

[0148] The process of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content in S202 above may include:

[0149] S214: Determine the second set of data identifiers corresponding to each first index data block based on the data identifier of the numerical business content corresponding to each first index data block.

[0150] The index fields corresponding to the multiple first index data blocks include: multiple first index data blocks and a set of second data identifiers corresponding to each first index data block.

[0151] In this embodiment, after processing the numerical business content into data blocks, for each first index data block, a second data identifier set consisting of data identifiers of the numerical business content corresponding to each first index data block is determined. Based on the first index data block and the second data identifier set corresponding to the first index data block, a C-type indexing method is used to generate multiple index fields corresponding to the first index data block.

[0152] In some embodiments, the index fields corresponding to multiple first index data blocks are represented by a B+ tree structure. The multiple first index data blocks form a B+ tree according to the level when the interval is divided. Each first index data block in the B+ tree corresponds to a non-leaf node. The content of the non-leaf node is the block number and the numerical range corresponding to the first index data block. The content of the leaf nodes under the multiple first index data blocks at the last level is the second data identifier set. The index fields formed by the B+ tree have the characteristics of small size and can be fully loaded into memory for application.

[0153] During data search, a binary search is performed on the data search conditions based on the B+ tree until the first index data block corresponding to the last level of the data search conditions is determined from the B+ tree. The first index data block and the second data identifier set under the first index data block constitute the target index field.

[0154] The data search method provided in the above embodiments can improve search efficiency by processing numerical business content into data blocks to obtain a first index data block and a corresponding second data identifier set. When performing data search based on the index field composed of the first index data block and the corresponding second data identifier set, the search efficiency can be improved.

[0155] In one possible implementation, please refer to Figure 6 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 5 ,like Figure 6 As shown, the process of S201 above, which preprocesses the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents, may include:

[0156] S215: If the business field is of type text, perform word segmentation on multiple text business contents to obtain multiple index terms.

[0157] The process of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content in S202 above may include:

[0158] S216: Determine the third set of data identifiers corresponding to each index term based on the data identifier of the text business content corresponding to each index term.

[0159] The index fields corresponding to each index term include: each index term and a set of third data identifiers corresponding to each index term.

[0160] In this embodiment, the text type is used to indicate that the content of the business field is represented by a text string, such as name, fund product name, fund manager name, company name, etc., but does not include the text business content in the enumerated type business fields.

[0161] Using the D-type indexing method, multiple text business contents are first grouped. Based on the data identifiers corresponding to the same text business contents and the corresponding text business contents, a text business contents array is constructed. For example, the text business contents array can be represented as: [{text business contents 1:{data identifier set 1}},{text business contents 2:{data identifier set 2}}]. The text business contents in each text business contents array are segmented to obtain multiple index words corresponding to each text business contents. Based on the third data identifier set corresponding to the same index word and the corresponding index word, an index field corresponding to each index word is constructed. The index field can be represented as: [{segmentation 1:{data identifier set 3}},{segmentation 2:{data identifier set 4}}].

[0162] In some embodiments, after segmenting the text business content, a word set model can be used to determine the similar index words corresponding to each index word, and an index field can be constructed based on the index words, similar index words, and a third data identifier set.

[0163] In one possible implementation, after generating index terms, the index terms can be used as enumerated business content. Through numerical processing, the index terms can be converted into index values, and then the index fields corresponding to the text business content can be constructed using the A-type index method. On the one hand, converting index terms into index values ​​can reduce memory usage, and on the other hand, it can improve search efficiency.

[0164] The data search method provided in the above embodiments can improve search efficiency by performing word segmentation on text business content to obtain index words and corresponding third data identifier sets. When performing data search based on the index field composed of index words and corresponding third data identifier sets, the search efficiency can be improved.

[0165] In one possible implementation, please refer to Figure 7 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 6 ,like Figure 7 As shown, the process of S201 above, which preprocesses the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents, may include:

[0166] S217: If the business field is of type date, split the multiple date business contents separately.

[0167] S218: Perform numerical processing on the split date business content to obtain at least one second index value.

[0168] The process of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content in S202 above may include:

[0169] S219: Determine the fourth data identifier set corresponding to each second index value based on the data identifier of the date business content corresponding to each second index value.

[0170] The index field corresponding to each second index value includes: each second index value and the fourth data identifier set corresponding to each second index value.

[0171] In this embodiment, the date type is used to indicate that the content of the business field records the date and / or time. By splitting the date business content, users can search for data by year, month, day, etc.

[0172] The split year, month, and day are quantified to obtain the second index value. The quantification process can refer to the process of quantifying the enumerated business content in S211-S212 above. After quantifying the split date business content as the enumerated business content, for each second index value, a fourth data identifier set consisting of the data identifier of the date business content corresponding to each second index value is determined. Based on the second index value and the fourth data identifier set corresponding to the second index value, an index field corresponding to each second index value is generated.

[0173] The data search method provided in the above embodiments converts date business content into a second index value, and generates an index field based on the fourth data identifier set of the date business content corresponding to the second index value and the second index value. This allows the matching fourth data identifier set to be determined directly from the corresponding index field when searching data in units of year, month or day, thereby improving data search efficiency.

[0174] In one possible implementation, please refer to Figure 8 The following is a flowchart illustrating the data search method provided in the embodiments of this application. Figure 7 ,like Figure 8 As shown, the process of S201 above, which preprocesses the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database to obtain multiple index contents, may include:

[0175] S220: If the business field is of type date, perform data block processing on multiple date business contents to obtain multiple second index data blocks.

[0176] The process of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content in S202 above may include:

[0177] S221: Determine the fifth data identifier set corresponding to each second index data block based on the data identifier of the date business content corresponding to each second index data block.

[0178] The index fields corresponding to the multiple second index data blocks include: multiple second index data blocks and a fifth data identifier set corresponding to each second index data block.

[0179] In this embodiment, the date service content is processed into data blocks to obtain a second index database. The data block processing process can refer to the process of processing the date service content into data blocks in S213-S214 described above. For each second index data block, a fifth data identifier set consisting of the data identifiers of the date service content corresponding to each second index data block is determined. Based on the second index data block and the fifth data identifier set corresponding to the second index data block, an index field corresponding to each second index data block is generated.

[0180] The data search method provided in the above embodiments converts date business content into a second index data block, and generates an index field based on the fifth data identifier set of the date business content corresponding to the second index data block and the second index data block. This allows the matching fifth data identifier set to be determined directly from the corresponding index field when performing a data search for a specific date, thereby improving data search efficiency.

[0181] Based on the above examples of generating index fields according to the type of business fields, please refer to... Figure 9 This is a block diagram illustrating the principle of generating index fields according to an embodiment of this application, as shown below. Figure 9 As shown, enumerated business fields are converted into indexed values, numeric business fields are converted into indexed data blocks, text business fields are converted into indexed terms and indexed values, and date business fields are converted into indexed values ​​and indexed data blocks. For indexed values, a type A or type B index is used depending on the number of enumerations. For indexed data blocks, a type C index is used. For indexed terms, a type D index is used.

[0182] The data search method provided in the above embodiments designs different indexing methods for different types of business fields. By combining the advantages of various indexes, the efficiency of data search can be fully optimized, and the memory space occupied by the index can also be effectively optimized.

[0183] Based on the above method embodiments, this application also provides a data search device. Please refer to... Figure 10 This is a schematic diagram of the structure of the data search device provided in the embodiments of this application, as shown below. Figure 10 As shown, the device may include:

[0184] The search instruction acquisition module 10 is used to acquire data search instructions, which include: search conditions;

[0185] The index field acquisition module 20 is used to acquire multiple index fields of a preset database. Each index field includes: index content and a data identifier that matches the index content.

[0186] The index field matching module 30 is used to determine the index field corresponding to the search content that matches the search conditions as the target index field based on the search conditions and the index content of multiple index fields.

[0187] The data search module 40 is used to retrieve the target data corresponding to the target data identifier from a preset database based on the target data identifier in the target index field.

[0188] Optionally, the index field retrieval module 20 includes:

[0189] The index content generation unit is used to preprocess the business content corresponding to each business field according to the type of multiple business fields in the data wide table of the preset database, so as to obtain multiple index contents;

[0190] The data identifier determination unit is used to determine the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content.

[0191] The index field generation unit is used to generate multiple index fields based on multiple index contents and the data identifier set corresponding to the multiple index contents.

[0192] Optionally, the index content generation unit is specifically used to perform numerical processing on multiple enumerated business contents if the type of the business field is an enumeration type, to obtain at least one first index value.

[0193] The data identifier determination unit is specifically used to determine the first data identifier set corresponding to each first index value based on the data identifier of the enumerated business content corresponding to each first index value; wherein, the index field corresponding to each first index value includes: each first index value and the first data identifier set corresponding to each first index value.

[0194] Optionally, the data identifier determination unit is specifically used to perform a hash transformation on the data identifiers of all enumerated business contents corresponding to the first index value based on the enumeration quantity corresponding to the business field of the enumeration type, and to determine the first data identifier set.

[0195] Optionally, the index content generation unit is specifically used to process multiple numeric business contents into multiple first index data blocks if the type of the business field is numeric.

[0196] The data identifier determination unit is specifically used to determine the second data identifier set corresponding to each first index data block based on the data identifier of the numerical business content corresponding to each first index data block; wherein, the index fields corresponding to the multiple first index data blocks include: multiple first index data blocks and the second data identifier set corresponding to each first index data block.

[0197] Optionally, the index content generation unit is specifically used to perform word segmentation on multiple text business contents to obtain multiple index terms if the type of the business field is text.

[0198] The data identifier determination unit is specifically used to determine the third data identifier set corresponding to each index term based on the data identifier of the text business content corresponding to each index term; wherein, the index field corresponding to each index term includes: each index term and the third data identifier set corresponding to each index term.

[0199] Optionally, the index content generation unit is specifically used to split multiple date business contents separately if the business field is of date type; and to perform numerical processing on the split date business contents to obtain at least one second index value.

[0200] The data identifier determination unit is specifically used to determine the fourth data identifier set corresponding to each second index value based on the data identifier of the date business content corresponding to each second index value; wherein, the index field corresponding to each second index value includes: each second index value and the fourth data identifier set corresponding to each second index value.

[0201] Optionally, the index content generation unit is specifically used to process multiple date business contents into multiple second index data blocks if the type of the business field is date.

[0202] The data identifier determination unit is specifically used to determine the fifth data identifier set corresponding to each second index data block based on the data identifier of the date business content corresponding to each second index data block; wherein, the index fields corresponding to multiple second index data blocks include: multiple second index data blocks and the fifth data identifier set corresponding to each second index data block.

[0203] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0204] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0205] Please refer to Figure 11 This is a schematic diagram of the electronic device provided in the application embodiment, such as... Figure 11 As shown, the electronic device 100 includes a processor 101, a storage medium 102, and a bus. The storage medium 102 stores program instructions executable by the processor 101. When the electronic device 100 is running, the processor 101 communicates with the storage medium 102 via the bus, and the processor 101 executes the program instructions to perform the above-described method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0206] Optionally, the present invention also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the above-described method embodiments. The specific implementation and technical effects are similar and will not be repeated here.

[0207] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0210] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0211] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data search method characterized by, The method includes: Obtain data search instructions, wherein the data search instructions include: search criteria; Obtain multiple index fields from a preset database, each index field including: index content, and a data identifier that matches the index content; Based on the search criteria and the index content of the multiple index fields, the index field corresponding to the index content that matches the search criteria is determined as the target index field; Based on the target data identifier in the target index field, the target data corresponding to the target data identifier is obtained from the preset database; The step of obtaining multiple index fields from the preset database includes: If multiple business fields in the data wide table of the preset database are of enumeration type, the multiple enumeration business contents are numerically processed to obtain at least one first index value. Based on the number of enumerations corresponding to the business fields of the enumeration type, perform a hash transformation on the data identifiers of all enumeration business contents corresponding to the first index value to determine the first data identifier set. The index field corresponding to each first index value includes: each first index value and a first data identifier set corresponding to each first index value; If the number of enumerations is less than a preset threshold, the first data identifier set is indexed by a Hash Map and the hash-transformed first data identifier set is represented in the form of a bit array. If the number of enumerations is greater than a preset threshold, the data identifiers in the first data identifier set are transformed by HashSet, and the transformed data identifiers are stored in the order of hash values. If the number of data sets corresponding to each enumeration variable is equal to the average amount of data in the wide table, the first data identifier set corresponding to the index field is represented by an auxiliary bit value + an optimized bit array. The auxiliary bit value is used to record the data identifier of the enumeration business content corresponding to the first index value in the wide data table. The optimized bit value includes the number of 1s in the data set corresponding to the first index value. The data identifier with the first index value is determined by multiplying the auxiliary bit value and the optimized bit array. If the number of data sets corresponding to each enumerated variable is less than or equal to a preset set number threshold, the first data identifier set is represented by an int value.

2. The method as described in claim 1, characterized in that, The step of obtaining multiple index fields from the preset database includes: Based on the types of multiple business fields in the data wide table of the preset database, the business content corresponding to each business field is preprocessed to obtain multiple index contents; Based on the data identifiers of the business content corresponding to the same index content, determine the set of data identifiers corresponding to each index content; Based on the multiple index contents and the data identifier set corresponding to the multiple index contents, the multiple index fields are generated respectively.

3. The method as described in claim 2, characterized in that, The step involves preprocessing the business content corresponding to each business field based on its type in the data wide table of the preset database, resulting in multiple index contents, including: If the type of the business field is numeric, multiple numeric business contents are processed into data blocks to obtain multiple first index data blocks; The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes: Based on the data identifier of the numerical service content corresponding to each first index data block, determine the second data identifier set corresponding to each first index data block; The index fields corresponding to the plurality of first index data blocks include: the plurality of first index data blocks and the second data identifier set corresponding to each first index data block.

4. The method as described in claim 2, characterized in that, The step involves preprocessing the business content corresponding to each business field based on its type in the data wide table of the preset database, resulting in multiple index contents, including: If the business field is of type text, multiple text business contents are segmented to obtain multiple index words; The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes: Based on the data identifier of the text business content corresponding to each index term, determine the third data identifier set corresponding to each index term; The index field corresponding to each index term includes: each index term and the third data identifier set corresponding to each index term.

5. The method as described in claim 2, characterized in that, The step involves preprocessing the business content corresponding to each business field based on its type in the data wide table of the preset database, resulting in multiple index contents, including: If the business field is of type date, then the multiple date business contents should be split separately. The split date business content is numerically processed to obtain at least one second index value; The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes: Based on the data identifier of the date business content corresponding to each second index value, determine the fourth data identifier set corresponding to each second index value; The index field corresponding to each second index value includes: each second index value and the fourth data identifier set corresponding to each second index value.

6. The method as described in claim 2, characterized in that, The step involves preprocessing the business content corresponding to each business field based on its type in the data wide table of the preset database, resulting in multiple index contents, including: If the type of the business field is date, multiple date business contents are processed into data blocks to obtain multiple second index data blocks; The step of determining the set of data identifiers corresponding to each index content based on the data identifiers of the business content corresponding to the same index content includes: Based on the data identifier of the date business content corresponding to each second index data block, determine the fifth data identifier set corresponding to each second index data block; The index fields corresponding to the plurality of second index data blocks include: the plurality of second index data blocks and the fifth data identifier set corresponding to each second index data block.

7. A data search device, characterized in that, The data search device is used to perform the data search method as described in any one of claims 1 to 6, the device comprising: The search instruction acquisition module is used to acquire data search instructions, which include: search conditions; The index field acquisition module is used to acquire multiple index fields of a preset database. Each index field includes: index content and a data identifier that matches the index content. The index field matching module is used to determine the index field corresponding to the search content that matches the search conditions as the target index field based on the search conditions and the index content of the multiple index fields. The data search module is used to retrieve the target data corresponding to the target data identifier from the preset database based on the target data identifier in the target index field.

8. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the data search method as described in any one of claims 1 to 6.

Images