Data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and storage medium, wherein the data processing method comprises: generating and storing indexes corresponding to each first data in at least two first data in a storage engine; wherein each index comprises a first key, a second key and at least one third key; the first key represents the corresponding first data; the second key represents the identity of the corresponding first data; the indexes corresponding to the at least two first data are sorted according to the key value of the at least one third key in the storage engine; at least one first index corresponding to a first query condition is found in the storage engine; the key value of the third key of each first index in the at least one first index matches the key value of the third key indicated in the first query condition; a query result is returned; the query result comprises the key value of the second key of each first index in the at least one first index.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Data indexing technology involves creating indexes within a data system to help users quickly locate stored data. However, implementing data indexing within a native data system requires custom development for each system, resulting in low development efficiency. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium to at least solve the problem of low development efficiency in implementing data indexing within the native data system in related technologies.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, the method comprising:

[0006] An index corresponding to each of at least two first data sets is generated and stored in a storage engine; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents the identifier of the corresponding first data set; the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key; at least one first index corresponding to a first query condition is found in the storage engine; the key value of the third key of each of the at least one first indexes matches the key value of the third key indicated in the first query condition;

[0007] Return the query results; the query results include the key value of the second key of each of the at least one first index.

[0008] In the above scheme, the at least one third key includes at least one of the following:

[0009] The key of the data table that stores the data corresponding to the first data;

[0010] The key that represents the logical partition corresponding to the first data;

[0011] The key that represents the attribute corresponding to the first data.

[0012] In the above scheme, the key value format of the first key is the same as that of the second key; a separator is set between the first key and the second key.

[0013] In the above scheme, before generating and storing the index corresponding to each of the at least two first data in the storage engine, the method further includes:

[0014] The at least two first data items are escaped to obtain the key value of the corresponding first key.

[0015] In the above scheme, when finding at least one first index corresponding to the first query condition in the storage engine, the method includes:

[0016] If a first index is determined, locate the previous or next index of the determined first index in the storage engine;

[0017] If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, the located index is determined to be a first index;

[0018] The search ends if the value of the third key in the located index does not match the value of the third key indicated in the first query condition.

[0019] In the above scheme, the first query condition is further used to indicate the key value of the first key; determining the index to be located as a first index includes:

[0020] If the key value of the first key in the located index matches the key value of the first key indicated in the first query condition, the located index is determined to be a first index.

[0021] In the above scheme, before generating and storing the index corresponding to each of the at least two first data in the storage engine, the method further includes:

[0022] The corresponding first data is processed according to the set rules to obtain the key value of the first key.

[0023] In the above scheme, the step of processing the corresponding first data according to a set rule to obtain the key value of the first key includes:

[0024] Reverse the order of each character in the corresponding first data to obtain the key value of the first key;

[0025] And / or,

[0026] The key value of the first key is obtained based on at least two characters in the corresponding first data.

[0027] This application also provides a data processing apparatus, including:

[0028] A generation unit is configured to generate and store in a storage engine an index corresponding to each of at least two first data sets; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents the identifier of the corresponding first data set; and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key.

[0029] A lookup unit is configured to find at least one first index in the storage engine corresponding to a first query condition; the key value of the third key of each of the at least one first index matches the key value of the third key indicated in the first query condition.

[0030] The result unit is used to return query results; the query results include the key value of the second key of each of the at least one first index.

[0031] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0032] The processor is used to execute the steps of the above-described data processing method when running the computer program.

[0033] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described data processing method.

[0034] In this embodiment, an index corresponding to each of at least two first data sets is generated and stored in a storage engine within the data system. Each index includes a first key, a second key, and at least one third key, and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the corresponding at least one third key. When searching for data, at least one first index is found in the storage engine that matches the key value of the corresponding third key with the key value of the third key indicated in the first query condition, and the key value of the second key of each of the at least one first index is returned as the query result. This allows data querying to be implemented within the data system. The above data processing scheme is universally applicable to various data systems. When indexing data, there is no need to access an external indexing system, nor is it necessary to develop different data indexing functions for different native data systems, thereby improving the development efficiency of the data system. Attached Figure Description

[0035] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0036] Figure 2A schematic diagram illustrating the format of an index setting provided in an embodiment of this application;

[0037] Figure 3 A schematic diagram illustrating an index sorting method provided in an embodiment of this application;

[0038] Figure 4 A schematic diagram illustrating an index lookup provided in an embodiment of this application;

[0039] Figure 5 A schematic diagram of an index provided for an application embodiment of this application;

[0040] Figure 6 A schematic diagram illustrating yet another type of index provided for an application embodiment of this application;

[0041] Figure 7 A schematic diagram illustrating another index provided for an application embodiment of this application;

[0042] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0043] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] With the advent of the big data era, storage systems now handle massive amounts of data. However, users often struggle to locate specific data within this vast amount of data. For example, objects and files are two of the most common forms of unstructured data storage. Searching through hundreds of millions of objects and tens of millions of files in a data system can take anywhere from tens of minutes to hours, placing a significant burden on users' ability to effectively utilize the data.

[0045] Data indexing technology involves creating indexes within a data system to help users quickly locate stored data. When searching through files, users don't need to traverse all the data, significantly reducing search time and improving data availability. One approach is to store the indexing system within the native data system. For example, MySQL's InnoDB storage engine implements hash and B-tree indexes based on the native storage structure. However, because the indexing system directly involves the storage structure, implementing data indexing within the native data system requires custom development for each system, resulting in low development efficiency.

[0046] Based on this, in various embodiments of this application, at least two indexes corresponding to each of the first data sets are generated and stored in a storage engine within the data system. Each index includes a first key, a second key, and at least one third key, and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the corresponding at least one third key. When searching for data, at least one first index is found in the storage engine that matches the key value of the corresponding third key with the key value of the third key indicated in the first query condition, and the key value of the second key of each of the at least one first index is returned as the query result. In this way, data querying can be implemented within the data system. The above data processing scheme is universally applicable to various data systems. When performing data indexing, there is no need to access an external indexing system, nor is it necessary to develop different data indexing functions for different native data systems, thereby improving the development efficiency of the data system.

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Figure 1 This is a schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application. The executing entity can be an electronic device. Figure 1 The data processing methods shown include:

[0049] Step 101: Generate and store in the storage engine an index corresponding to each of the first data in at least two sets of first data.

[0050] Each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data; the second key represents the identifier of the corresponding first data; and the indexes corresponding to the at least two first data are sorted in the storage engine according to the key value of the at least one third key.

[0051] In this embodiment, an index corresponding to each of at least two first data items is generated by an electronic device and stored in a storage engine within the data system. The index has a defined format, and each index includes a first key, a second key, and at least one third key. The indexes corresponding to the at least two first data items are sorted in the storage engine according to the key value of the corresponding at least one third key. The first key represents the corresponding first data item itself. The second key represents the identifier of the first data item pointed to by the index; it can be a unique identifier pointing to the first data item. This identifier can be used to indicate the storage location of the first data item, such as an inode in a file system or a primary key in a Structured Query Language server database (SQL). Here, the storage engine can be a key-value store engine (KV) or RocksDB, a single-machine KV key-value store engine implemented based on a Log-Structured Merge (LSM) tree, supporting point lookup interfaces such as Put and Get, as well as continuous lookup interfaces provided by iterators. The storage engine can be located within the data system.

[0052] In one embodiment, the key value format of the first key is the same as that of the second key; a separator is provided between the first key and the second key.

[0053] Here, the key value format of the first key is the same as that of the second key, which can be bytes. A separator is set between the first key and the second key. The separator can be a special character with an ASCII code value of 0 to 31, that is, a single bit of a non-printable character. This special character can be called an INDEX SPLITTER. In this way, the first key and the second key of each index in bytes format can be distinguished.

[0054] In one embodiment, before generating and storing the index corresponding to each of the at least two first data in the storage engine, the method further includes:

[0055] The at least two first data items are escaped to obtain the key value of the corresponding first key.

[0056] Before generating and storing the index corresponding to each of the at least two first data points in the storage engine within the data system, each first data point is escaped to obtain the key value of the first key corresponding to each first data point. Here, escaping can be achieved by defining ASCII characters that cannot be displayed, such as \0, \t, \n, etc., by adding a "\" before specific letters.

[0057] In this way, when the key value of the first key includes arbitrary binary data (printable and non-printable characters), the first key and the second key in bytes format of each index can be distinguished.

[0058] In one embodiment, the at least one third key includes at least one of the following:

[0059] The key of the data table that stores the data corresponding to the first data;

[0060] The key that represents the logical partition corresponding to the first data;

[0061] The key that represents the attribute corresponding to the first data.

[0062] At least one third key can be any combination of the following: including only the key representing the data table storing the corresponding first data; including only the key representing the logical partition corresponding to the first data; including only the key representing the attribute corresponding to the first data; including both the key representing the data table storing the corresponding first data and the key representing the logical partition corresponding to the first data; including both the key representing the data table storing the corresponding first data and the key representing the attribute corresponding to the first data; including both the key representing the logical partition corresponding to the first data and the key representing the attribute corresponding to the first data; including the key representing the data table storing the corresponding first data, the key representing the logical partition corresponding to the first data, and the key representing the attribute corresponding to the first data. Here, each key can be represented as a field. In this way, by setting the format, the index can be stored contiguously in the storage engine, which can improve the efficiency of finding indexes that match the search conditions.

[0063] like Figure 2 The diagram illustrates a typical index configuration format. An index consists of a primary key, a secondary key, and three tertiary keys, arranged according to a defined format. The key name, field name, field type, and explanation for each key are shown below:

[0064]

[0065] like Figure 3 The diagram illustrates how indexes are sorted according to the key values ​​of their corresponding third keys. The indexes are arranged according to the key values ​​of third key 1 (region_1, region_2, region_3, etc.). Within the index of third key 1 with the key value of region_1, they are further sorted according to the key values ​​of third key 2, and then further sorted according to the key values ​​of third key 3 (name, owner). They can also be further sorted according to the key values ​​of the first key. In this way, within the same Region, indexes with the same Column and Field stored in the storage engine are lexicographically consecutive. This contiguous storage of indexes in the storage engine improves the efficiency of finding indexes that match the search criteria.

[0066] Step 102: Locate at least one first index in the storage engine corresponding to the first query condition.

[0067] In this context, the key value of the third key in each of the at least one first index matches the key value of the third key indicated in the first query condition.

[0068] Based on the key value of the third key indicated by the first query condition, a search is performed in the storage engine to find at least one first index. Here, the key value of the third key corresponding to the found first index matches the key value of the third key indicated by the first query condition. The first query condition can be set to include only the key value of at least one third key corresponding to the index to be searched, or it can be set to include both the key value of at least one third key corresponding to the index to be searched and some or all of the key values ​​of the first key corresponding to the index to be searched.

[0069] For example, the first query condition can be set to the third key having the values ​​of region_1, mds, and owner. Alternatively, the first query condition can be set to the third key having the values ​​of region_1, mds, and owner, and the first key having the value of root.

[0070] Step 103: Return the query results.

[0071] The query result includes the key value of the second key of each of the at least one first index.

[0072] Based on finding at least one first index in the storage engine corresponding to the first query condition, the query results are returned, including the key value of the second key of each of the at least one first index.

[0073] An index corresponding to each of at least two first data sets is generated and stored in a storage engine within the data system. Each index includes a first key, a second key, and at least one third key, and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the corresponding at least one third key. When searching for data, at least one first index is found in the storage engine whose corresponding third key value matches the third key value indicated in the first query condition, and the second key value of each of the at least one first index is returned as the query result. This allows data querying to be performed within the data system. The above data processing scheme is universally applicable to various data systems. When indexing data, there is no need to access an external indexing system, nor is it necessary to develop different data indexing functions for different native data systems, thereby improving the development efficiency of the data system.

[0074] In one embodiment, when finding at least one first index corresponding to a first query condition in the storage engine, the method includes:

[0075] If a first index is determined, locate the previous or next index of the determined first index in the storage engine;

[0076] If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, the located index is determined to be a first index.

[0077] The search ends if the value of the third key in the located index does not match the value of the third key indicated in the first query condition.

[0078] Based on the key value of the third key indicated in the first query condition, a search is performed in the storage engine. If a first index is identified, the search continues to the previous or next index of that first index within the storage engine. If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, that located index is also identified as a first index, and the search continues to the previous or next index of the most recently identified first index within the storage engine. If the key value of the third key of the located index does not match the key value of the third key indicated in the first query condition, the search ends. Here, the query results can be returned after the search ends. If the previous search targeted the previous index of the first index, the current search will also target the previous index of the first index; if the previous search targeted the next index of the first index, the current search will also target the next index of the first index. This sequential search of the indexes stored sequentially in the storage engine improves the efficiency of finding indexes that meet the conditions.

[0079] In practical applications, if a user needs to find all data of type 'name' in the 'file' table under 'region_1', they can use an iterator to locate the data and determine the first index. For example... Figure 4The illustrated index lookup diagram shows that the first query condition is set to the key value of the third key: region_1, file, name. This means the index is on data of type name in the file table under region_1. The search uses an iterator to locate the first index and continues searching in the storage engine to the next index after the most recently determined first index (i.e., the second index). It checks if the second index satisfies the first query condition. Since the second index also satisfies the first query condition, it is designated as the first index. The search then continues to the next index after the most recently determined first index (i.e., the third index) and checks if the third index satisfies the first query condition. This process continues until the sixth index is reached, at which point the search ends because the sixth index does not satisfy the first query condition.

[0080] In one embodiment, the first query condition is further used to indicate the key value of the first key; determining the index to be located as a first index includes:

[0081] If the key value of the first key in the located index matches the key value of the first key indicated in the first query condition, the located index is determined to be a first index.

[0082] If the value of the third key of the located index matches the value of the third key indicated in the first query condition, then it is determined whether the value of the first key of the located index matches the value of the first key indicated in the first query condition. If they match, the located index is determined to be a first index. In this way, the native data system can realize the function of looking up the index in the storage engine based on the indexed data.

[0083] In practical applications, if a user needs to find all data belonging to the root user, they can use an iterator to locate the data and determine the first index. For example... Figure 4 The illustrated index lookup diagram shows that the first query condition is set to the key values ​​of the third key being region_1, file, and owner, and the key value of the first key being root. The iterator locates the sixth index, and then continues locating the next index after the most recently determined first index in the storage engine (i.e., the seventh index). It checks if the seventh index satisfies the first query condition. Since the seventh index also satisfies the first query condition, it is designated as the first index. The search then locates the next index after the most recently determined first index in the storage engine (i.e., the eighth index), and checks if the eighth index satisfies the first query condition. Since the eighth index does not satisfy the first query condition, the lookup ends.

[0084] In one embodiment, before generating and storing the index corresponding to each of the at least two first data in the storage engine, the method further includes:

[0085] The corresponding first data is processed according to the set rules to obtain the key value of the first key.

[0086] Before generating and storing the index corresponding to each of the at least two first data points in the storage engine within the data system, the corresponding first data points are processed according to predefined rules to obtain the key value of the first key. This processing of the first data points enables the native data system to support multiple index lookup methods, making its index lookups universally applicable to various indexing scenarios, thereby improving the development efficiency of the data system.

[0087] In one embodiment, processing the corresponding first data according to a set rule to obtain the key value of the first key includes:

[0088] Reverse the order of each character in the corresponding first data to obtain the key value of the first key;

[0089] And / or,

[0090] The key value of the first key is obtained based on at least two characters in the corresponding first data.

[0091] Here, the first set of data refers to at least two characters. These can be two consecutive characters, or the last character and the first character, such as "4l" in "life4". The first set of data can be processed according to the set rules. This can be by reversing the order of each character, by using at least two characters, or by a combination of both. This processing of the first set of data enables the native data system to support multiple search and indexing methods, making its index lookup universally applicable to various indexing scenarios and improving the development efficiency of the data system.

[0092] like Figures 5 to 7 The diagram shows the index.

[0093] Figure 5 This diagram illustrates a method of reversing the order of the first data, such as processing "life4" into "4efil". This allows searching for files ending in "le4" starting from the last character. The search can use "4ef" as the first query condition, which will match the fourth index. In this way, the storage engine's index can be used to implement a reverse index function.

[0094] Figure 6This diagram illustrates a method of processing the first data using a round-robin index. For example, "life4" is processed as "file4$", "ile4$f", "le4$fi", "e4$fil", and "4$file". The number of corresponding results corresponds to the length of the string. The "$" character is used as a separator to indicate the end of the string. When searching, using "ile" as the first query condition will match the second index. In this way, the storage engine's index can be used for fuzzy searching of intermediate fields like "ile", achieving the round-robin index function.

[0095] Figure 7 This diagram illustrates a method of processing the first data using k-grams. For example, "life4" is processed as "fil", "ile", "le4", "e4$f", and "4$fi". The $ character is used as a separator to indicate the end of the string. When searching, "ile" is used as the first query condition, which will match the second index. After finding the index result, the result needs to be filtered again. In this way, the storage engine's index can be used for fuzzy searching of intermediate fields like "ile", and it saves storage space compared to round-robin indexes, thus achieving the function of k-gram indexes.

[0096] The present application will now be described in further detail with reference to application examples.

[0097] Currently, indexing schemes in existing data systems can be categorized into two types based on the location of the index: natively implemented data indexes and externally implemented data indexes.

[0098] (1) Native implementation of data indexes: This involves implementing usable indexing functionality within the native data system. For example, MySQL's InnoDB data engine implements Hash, B-tree, and other indexing functions based on the native storage structure. Because the indexing system directly involves the storage structure, implementing native data indexes for each data system requires corresponding development, resulting in low development efficiency. Furthermore, the supported search methods are limited.

[0099] (2) External data indexing: An indexing system is deployed outside the data system. The data system synchronizes data to the indexing system synchronously or asynchronously, and the subsequent user search requests are also handled by the indexing system. However, this method suffers from lower operating efficiency and weaker consistency.

[0100] Generally, because cross-system interaction is not required, (1) is more efficient than (2). For the same reason, the indexing function of (1) can have stronger consistency than that of (2). However, implementing the indexing function on an existing data system requires a lot of work and corresponding development for each data system, resulting in low development efficiency. Furthermore, the supported index types are often not comparable to the externally implemented data indexing of the (2) scheme.

[0101] This application provides an inverted index (InvertedIndexes) method for a key-value store. It utilizes a KV key-value storage engine to store the index sequentially and uses iterators for continuous access to achieve lookups. Unlike MySQL's direct index physical storage structure, this solution, based on a KV key-value storage engine, does not require direct indexing of the physical storage structure. The iterator function hides the underlying storage structure while providing an efficient abstraction for continuous access.

[0102] Here, key-value pair storage refers to a storage system where data is organized in key-value pair format and provides Put and Get interfaces. An inverted index system is a technique that speeds up data retrieval by maintaining additional index data. Indexes have defined formats, such as... Figure 2 As shown, based on the defined index format, indexes within the same region are organized in a specific way. Within the same Region (the physical dividing unit of the data table, i.e., the third key 1), indexes with the same Column (i.e., the third key 2) and Field (i.e., the third key 3) stored in the storage engine are lexicographically consecutive. By storing indexes contiguously in the storage engine, the efficiency of finding indexes that match the search criteria can be improved.

[0103] In the storage engine, iterators are used to find the first index. Iterators include operation interfaces such as Seek, SeekForPrev, Next, and Prev. Seek and SeekForPrev are used to quickly locate the index, respectively, to locate data after or before the current index. Next and Prev are used to locate data backward or forward within the index, respectively.

[0104] The set Region, Column, and Field fields, along with some or all of the Data fields (i.e., the first key), constitute the search fields passed to the Seek interface. Then, the index is accessed sequentially through the Next interface. If the index does not meet the query conditions, the search stops. If it does meet the conditions, the ID field (i.e., the second key) corresponding to the index is retrieved and added to the returned results, and then the Next interface is called again.

[0105] by Figure 4In the scenario shown, a user needs to find all data belonging to the root user. `region_1`, `file`, `owner`, and `root` constitute the search fields for the given Seek interface. Seek locates the first piece of data that matches the query criteria. Finding that all fields match the criteria, inode 9 is added to the returned results. After calling Next, the next record also matches the query, so inode 4 is added. The final Next call finds that the Unix string in the `Data` field does not match `root`, ending the search.

[0106] Since the storage engine can be a single-machine key-value storage engine implemented based on an LSM tree, which is an N-level tree structure, and because the index data at each level is ordered, traversing all matching index data by an iterator can be viewed as an N-way data merge. With the aid of Bloom filters, Block Indexes, and other auxiliary sorted string table (SST) data structures, the iterator can perform efficient traversal.

[0107] Furthermore, by setting rules to process the Data field corresponding to the first data, the supported index types include not only ordered indexes but also other index types in the retrieval domain, such as round-robin indexes and k-grams. In this way, by processing the Data field corresponding to the first data, existing native data systems can support multiple index lookup methods, making the index lookup of native data systems universally applicable to various indexing scenarios, thereby improving development efficiency.

[0108] like Figures 5 to 7 The diagram shows the index.

[0109] Figure 5 This diagram illustrates how to reverse the order of the Data field corresponding to the first data entry, such as processing "life4" into "4efil". This allows searching for files ending in "le4" starting from the last character of the Data field. When searching, "4ef" can be used as the first query condition, which will match the fourth index. In this way, reverse indexing can be implemented using the storage engine's indexes.

[0110] Figure 6This diagram illustrates a round-robin indexing process for the Data field corresponding to the first data. For example, "life4" is processed as "file4$", "ile4$f", "le4$fi", "e4$fil", and "4$file". The number of corresponding results corresponds to the length of the string. The "$" character is used as a separator to indicate the end of the string. When searching, using "ile" as the first query condition will match the second index. In this way, the storage engine's index can be used for fuzzy searching of intermediate fields like "ile", achieving the round-robin indexing function.

[0111] Figure 7 This diagram illustrates a k-gram processing method for the Data field corresponding to the first data. For example, "life4" is processed as "fil", "ile", "le4", "e4$f", and "4$fi". The $ character is used as a separator to indicate the end. When searching, "ile" is used as the first query condition, which will match the second index. After finding the index result, the result needs to be filtered again. In this way, the storage engine's index can be used for fuzzy searching of intermediate fields like "ile", and it can save storage space compared to round-robin indexes, thus achieving the function of k-gram index.

[0112] To implement the method of the embodiments of this application, the embodiments of this application also provide a data processing apparatus, such as... Figure 8 As shown, the device includes:

[0113] The generation unit 801 is configured to generate and store in the storage engine an index corresponding to each of at least two first data sets; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents the identifier of the corresponding first data set; and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key.

[0114] The lookup unit 802 is used to find at least one first index corresponding to the first query condition in the storage engine; the key value of the third key of each of the at least one first index matches the key value of the third key indicated in the first query condition.

[0115] Result unit 803 is used to return query results; the query results include the key value of the second key of each of the at least one first index.

[0116] In one embodiment, the at least one third key includes at least one of the following:

[0117] The key of the data table that stores the data corresponding to the first data;

[0118] The key that represents the logical partition corresponding to the first data;

[0119] The key that represents the attribute corresponding to the first data.

[0120] In one embodiment, the key value format of the first key is the same as that of the second key; a separator is provided between the first key and the second key.

[0121] In one embodiment, the apparatus further includes:

[0122] An escape unit is used to escape the at least two first data to obtain the key value of the corresponding first key;

[0123] In one embodiment, the lookup unit 802 is configured to:

[0124] If a first index is determined, locate the previous or next index of the determined first index in the storage engine;

[0125] If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, the located index is determined to be a first index.

[0126] The search ends if the value of the third key in the located index does not match the value of the third key indicated in the first query condition.

[0127] In one embodiment, the first query condition is further used to indicate the key value of the first key; determining the index to be located as a first index includes:

[0128] If the key value of the first key in the index to which the query is located matches the key value of the first key indicated in the first query condition, the index to which the query is located is determined to be a first index.

[0129] In one embodiment, the apparatus further includes:

[0130] The processing unit is used to process the corresponding first data according to a set rule to obtain the key value of the first key.

[0131] In one embodiment, the processing unit is configured to:

[0132] Reverse the order of each character in the corresponding first data to obtain the key value of the first key;

[0133] And / or,

[0134] The key value of the first key is obtained based on at least two consecutive characters in the corresponding first data.

[0135] In practical applications, the generation unit 801, search unit 802, result unit 803, escape unit, and processing unit can be implemented by processors in data processing devices, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0136] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0137] Based on the hardware implementation of the above program modules, and in order to implement the data processing method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 9 As shown, the electronic device 900 includes:

[0138] The communication interface 910 enables information exchange with other devices, such as network devices.

[0139] The processor 920 is connected to the communication interface 910 to enable information interaction with other devices and to execute the methods provided by one or more of the above-described technical solutions when running a computer program. The computer program is stored in the memory 930.

[0140] Specifically, the processor 920 is used for:

[0141] An index corresponding to each of at least two first data sets is generated and stored in a storage engine; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents the identifier of the corresponding first data set; and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key.

[0142] In the storage engine, at least one first index corresponding to the first query condition is found; the key value of the third key of each of the at least one first index matches the key value of the third key indicated in the first query condition.

[0143] Return the query results; the query results include the key value of the second key of each of the at least one first index.

[0144] In one embodiment, the at least one third key includes at least one of the following:

[0145] The key of the data table that stores the data corresponding to the first data;

[0146] The key that represents the logical partition corresponding to the first data;

[0147] The key that represents the attribute corresponding to the first data.

[0148] In one embodiment, the key value format of the first key is the same as that of the second key; a separator is provided between the first key and the second key.

[0149] In one embodiment, the processor 920 is configured to:

[0150] The at least two first data items are escaped to obtain the key value of the corresponding first key.

[0151] In one embodiment, the processor 920 is configured to:

[0152] If a first index is determined, locate the previous or next index of the determined first index in the storage engine;

[0153] If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, the located index is determined to be a first index.

[0154] The search ends if the value of the third key in the located index does not match the value of the third key indicated in the first query condition.

[0155] In one embodiment, the first query condition is further used to indicate the key value of the first key; determining the index to be located as a first index includes:

[0156] If the key value of the first key in the located index matches the key value of the first key indicated in the first query condition, the located index is determined to be a first index.

[0157] In one embodiment, the processor 920 is configured to:

[0158] The corresponding first data is processed according to the set rules to obtain the key value of the first key.

[0159] In one embodiment, the processor 920 is configured to:

[0160] Reverse the order of each character in the corresponding first data to obtain the key value of the first key;

[0161] And / or,

[0162] The key value of the first key is obtained based on at least two consecutive characters in the corresponding first data.

[0163] Of course, in practical applications, the various components in electronic device 900 are coupled together through bus system 940. It can be understood that bus system 940 is used to realize the connection and communication between these components. In addition to a data bus, bus system 940 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 9 The general labeled all buses as Bus System 940.

[0164] The memory 930 in this embodiment is used to store various types of data to support the operation of the electronic device 900. Examples of such data include any computer program used to operate on the electronic device 900.

[0165] It is understood that memory 930 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 930 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0166] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 920. The processor 920 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 920 or by instructions in the form of software. The processor 920 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 920 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 930. The processor 920 reads the program in the memory 930 and combines it with its hardware to complete the steps of the aforementioned method.

[0167] Optionally, when the processor 920 executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0168] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 930 storing a computer program, which can be executed by a processor 920 of an electronic device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0174] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict. Unless otherwise stated and limited, the term "connection" should be interpreted broadly. For example, it can refer to an electrical connection, or the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above term according to the specific circumstances.

[0175] Furthermore, in the examples of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first," "second," and "third" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein.

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0177] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A data processing method, characterized in that, The method includes: An index corresponding to each of at least two first data sets is generated and stored in a storage engine; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents the unique identifier of the corresponding first data set; and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key. In the storage engine, at least one first index corresponding to the first query condition is found; the key value of the third key of each of the at least one first index matches the key value of the third key indicated in the first query condition. Return the query results; the query results include the key value of the second key of each of the at least one first index; Wherein, the at least one third bond includes at least one of the following: The key of the data table that stores the data corresponding to the first data; The key that represents the logical partition corresponding to the first data; The key that represents the attribute corresponding to the first data.

2. The method according to claim 1, characterized in that, The key value format of the first key is the same as that of the second key; a separator is set between the first key and the second key.

3. The method according to claim 1, characterized in that, Before generating and storing the index corresponding to each of the at least two first data points in the storage engine, the method further includes: The at least two first data items are escaped to obtain the key value of the corresponding first key.

4. The method according to any one of claims 1 to 3, characterized in that, When finding at least one first index corresponding to the first query condition in the storage engine, the method includes: If a first index is determined, locate the previous or next index of the determined first index in the storage engine; If the key value of the third key of the located index matches the key value of the third key indicated in the first query condition, the located index is determined to be a first index; The search ends if the value of the third key in the located index does not match the value of the third key indicated in the first query condition.

5. The method according to claim 4, characterized in that, The first query condition is also used to indicate the key value of the first key; determining the index to be located as a first index includes: If the key value of the first key in the located index matches the key value of the first key indicated in the first query condition, the located index is determined to be a first index.

6. The method according to any one of claims 1 to 3, characterized in that, Before generating and storing the index corresponding to each of the at least two first data points in the storage engine, the method further includes: The corresponding first data is processed according to the set rules to obtain the key value of the first key.

7. The method according to claim 6, characterized in that, The step of processing the corresponding first data according to a set rule to obtain the key value of the first key includes: Reverse the order of each character in the corresponding first data to obtain the key value of the first key; And / or, The key value of the first key is obtained based on at least two characters in the corresponding first data.

8. A data processing apparatus, characterized in that, include: A generation unit is configured to generate and store in a storage engine an index corresponding to each of at least two first data sets; wherein each index includes a first key, a second key, and at least one third key; the first key represents the corresponding first data set; the second key represents a unique identifier of the corresponding first data set; and the indexes corresponding to the at least two first data sets are sorted in the storage engine according to the key value of the at least one third key. A lookup unit is configured to find at least one first index in the storage engine corresponding to a first query condition; the key value of the third key of each of the at least one first index matches the key value of the third key indicated in the first query condition. The result unit is used to return query results; the query results include the key value of the second key of each of the at least one first index; Wherein, the at least one third bond includes at least one of the following: The key of the data table that stores the data corresponding to the first data; The key that represents the logical partition corresponding to the first data; The key that represents the attribute corresponding to the first data.

9. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the data processing method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 7.

Images