Data query method, device and equipment, medium and product

Abstract

The invention provides a data query method, device and equipment, a medium and a product, and relates to the technical field of data processing. The method comprises the following steps: receiving a data query request for a target data element sent by an online transaction client; performing preliminary query according to indexes of a plurality of pre-constructed sub-arrays, and determining a target sub-array where the target data element is located; the plurality of sub-arrays form an original array, and the index of one sub-array comprises each data element in the sub-array and the position of each data element in the sub-array; querying again in the target sub-array, and determining the position of the target data element in the original array; and returning a re-query result to the online transaction client. Through the data query method provided by the invention, on the basis of not depending on pre-sorting of an original array, the number of query elements can be reduced, the time complexity of query can be reduced, and efficient real-time query can be realized.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a data query method, device, equipment, medium and product. Background Art

[0002] In current software development, Java array query is a common operation. Although there are many array query solutions, most of them are often seriously affected in query efficiency when processing large-scale data sets. Take traditional linear query as an example, it needs to compare each element in the array until the target element is found or the entire array is traversed. This method performs well when processing small arrays, but the efficiency will be significantly reduced when processing large arrays, and it cannot meet the needs of real-time query.

[0003] In addition, there are some advanced query algorithms, such as binary search, which can improve query efficiency to a certain extent, but they usually require the array to be pre-sorted, which may not always be feasible in practical applications. Summary of the invention

[0004] Embodiments of the present invention provide a data query method, apparatus, device, medium and product to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of an embodiment of the present invention provides a data query method, the method comprising:

[0006] Receiving a data query request for a target data element sent by an online transaction client;

[0007] Performing a preliminary query based on the indexes of a plurality of pre-constructed sub-arrays to determine the target sub-array where the target data element is located; the plurality of sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array;

[0008] Performing a re-query in the target sub-array to determine the position of the target data element in the original array;

[0009] The result of the re-query is returned to the online transaction client.

[0010] Optionally, the method further comprises:

[0011] Determine the sub-array size m based on the data partitioning requirement information;

[0012] Divide the original array of length n into k sub-arrays, each sub-array has a size of m = n / k;

[0013] When a change in data size or access pattern is detected, the subarray size m is re-determined and the original array of length n is re-partitioned.

[0014] Optionally, the method further comprises:

[0015] For each sub-array, generate an index for the sub-array according to the position of the data element in the sub-array;

[0016] For a sub-array, when the same data element exists in the sub-array, determining the latest position of the same data element in the sub-array;

[0017] The latest position of the same data element in the sub-array is saved in the index of the sub-array.

[0018] Optionally, the indexes of the plurality of sub-arrays are stored in a memory in the form of a hash table or an array; performing a preliminary query based on the pre-built indexes of the plurality of sub-arrays to determine the target sub-array where the target data element is located includes:

[0019] According to the target data element, the index of each sub-array is traversed to determine the target sub-array where the target data element is located.

[0020] Optionally, the re-querying in the target sub-array to determine the position of the target data element in the original array includes:

[0021] Comparing each data element in the target sub-array with the target data element respectively until the target data element is found in the target sub-array, and determining a local position of the target data element in the target sub-array;

[0022] The position of the target data element in the original array is determined according to the index of the target sub-array, the sub-array size and the local position.

[0023] Optionally, the method further comprises:

[0024] Determining the update frequency of the original array;

[0025] The step of determining the subarray size m according to the data partitioning requirement information includes:

[0026] When the update frequency is greater than the target frequency, the sub-array size m is determined according to the data partitioning requirement information.

[0027] A second aspect of an embodiment of the present invention provides a data query device, the device comprising:

[0028] A request receiving module, used for receiving a data query request for a target data element sent by an online transaction client;

[0029] A preliminary query module, used to perform preliminary query based on the indexes of a plurality of pre-built sub-arrays to determine the target sub-array where the target data element is located; the plurality of sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array;

[0030] A re-query module, used for re-querying in the target sub-array to determine the position of the target data element in the original array;

[0031] The result returning module is used to return the result of the re-query to the online transaction client.

[0032] Optionally, the device further comprises:

[0033] A data size determination module is used to determine the sub-array size m according to the data partitioning requirement information;

[0034] A first data partitioning module is used to partition an original array of length n into k sub-arrays, each sub-array having a size of m=n / k;

[0035] The second data partitioning module is used to redetermine the sub-array size m and re-partition the original array of length n when a change in data scale or access mode is detected.

[0036] Optionally, the device further comprises:

[0037] An index generation module, for generating an index for each sub-array according to the position of the data element in the sub-array in the sub-array;

[0038] A position determination module, for determining, for a sub-array, when the same data element exists in the sub-array, the latest position of the same data element in the sub-array;

[0039] The position saving module is used to save the latest position of the same data element in the sub-array in the index of the sub-array.

[0040] Optionally, the indexes of the plurality of sub-arrays are stored in a memory in the form of a hash table or an array; the preliminary query module comprises:

[0041] The preliminary query submodule is used to traverse the index of each subarray according to the target data element and determine the target subarray where the target data element is located.

[0042] Optionally, the re-query module includes:

[0043] An element comparison submodule, used for comparing each data element in the target subarray with the target data element respectively, until the target data element is found in the target subarray, and determining a local position of the target data element in the target subarray;

[0044] The re-query submodule is used to determine the position of the target data element in the original array according to the index of the target sub-array, the sub-array size and the local position.

[0045] Optionally, the device further comprises:

[0046] A frequency determination module, used to determine the update frequency of the original array;

[0047] The data size determination module comprises:

[0048] The data size determination submodule is used to determine the subarray size m according to the data partitioning requirement information when the update frequency is greater than the target frequency.

[0049] A third aspect of an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the data query method of the first aspect of the embodiment of the present invention is implemented.

[0050] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data query method of the first aspect of the embodiment of the present invention is implemented.

[0051] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the data query method described in the first aspect when executed by a processor.

[0052] According to the data query method provided by the embodiment of the present invention, first, a data query request for a target data element sent by an online transaction client is received; second, a preliminary query is performed according to the indexes of a plurality of pre-constructed sub-arrays to determine the target sub-array where the target data element is located; wherein the plurality of sub-arrays constitute the original array, and the index of a sub-array includes the individual data elements in the sub-array and the position of each data element in the sub-array; then, a re-query is performed in the target sub-array to determine the position of the target data element in the original array; finally, the result of the re-query is returned to the online transaction client, so as to realize fast and accurate query of the target data element. In this method, by constructing the index by segmenting the original array, that is, dividing the original array into a plurality of sub-arrays and constructing the indexes respectively, it does not rely on the pre-sorting of the original array, performs a preliminary query according to the index, and locates the target data element after the target sub-array for re-query, thereby reducing the number of elements to be queried, reducing the time complexity of the query, reducing the overhead of the re-query, and realizing efficient real-time query. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 is a flow chart of a data query method shown in one embodiment of the present invention;

[0055] Figure 2 This is an example diagram of an array query method based on the MapReduce concept proposed in one embodiment of the present invention;

[0056] Figure 3 is a structural block diagram of a data query device provided by an embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of an electronic device shown in one embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Hereinafter, specific examples of the present solution will be described in more detail with reference to the accompanying drawings.

[0060] refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a data query method according to an embodiment of the present invention. Figure 1 As shown, the data query method of this embodiment may include the following steps:

[0061] Step S11: receiving a data query request for a target data element sent by an online transaction client.

[0062] In this embodiment, the online transaction client can issue a data query request for the target data element to query the position of the target data element in the original array. In this embodiment, a data query request for the target data element sent by the online transaction client can be received, and the data query request carries the target data element and / or the identifier corresponding to the target data element. The original array of this embodiment can be a Java array or other array, which is not limited. In an optional example, the data query method provided by this embodiment can be applied to a data query system or an online transaction server.

[0063] Step S12: Perform a preliminary query based on the pre-constructed indexes of multiple sub-arrays to determine the target sub-array where the target data element is located; the multiple sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array.

[0064] In this embodiment, the target data element and / or the identifier corresponding to the target data element can be determined according to the received data query request, and the original array corresponding to the target data element can be determined based on the target data element and / or the identifier of the target data element. In this embodiment, the indexes of multiple sub-arrays corresponding to the original array are pre-constructed. Among them, multiple sub-arrays constitute the original array, and the index of a sub-array includes: each data element in the sub-array and the position of each data element in the sub-array.

[0065] This embodiment can perform a preliminary query based on the indexes of multiple sub-arrays corresponding to the original array to quickly locate the sub-array where the target data element is located, and determine the target sub-array where the target data element is located, so as to reduce the range of data elements that need to be traversed.

[0066] Step S13: perform another search in the target sub-array to determine the position of the target data element in the original array.

[0067] In this embodiment, after determining the target subarray corresponding to the target data element, a re-query can be performed in the target subarray to determine the position of the target data element in the original array and obtain the result of the re-query. The re-query in this embodiment is based on the accurate query of the target data element in the target subarray to ensure the accuracy of the query.

[0068] Step S14: Returning the result of the re-query to the online transaction client.

[0069] In this embodiment, after the result of the re-query (ie, the position of the target data element in the original array) is obtained, the result of the re-query may be returned to the online transaction client.

[0070] In this embodiment, by dividing the original array into multiple sub-arrays and constructing indexes respectively, it does not rely on the pre-sorting of the original array, performs a preliminary query based on the index, and performs a second query after locating the target data element in the target sub-array. Compared with the traditional linear query, the overall query time is reduced by first narrowing the query range and then performing an accurate match within a small range. In addition, due to the small size of the sub-array, the overhead of the accurate query is also lower, thereby achieving efficient real-time query.

[0071] In combination with the above embodiments, in one implementation, the present invention also provides a data query method. Specifically, in addition to the above steps, the method may also include steps S21 to S23:

[0072] Step S21: Determine the sub-array size m according to the data partitioning requirement information.

[0073] In this embodiment, the sub-array size m can be determined based on the data partitioning requirement information of the original array. The data partitioning requirement information of this embodiment can be the actual requirement of the original array, such as: the number of elements, data type or data distribution. The sub-array sizes corresponding to each sub-array constituting the original array can be the same or different, and there is no limitation on this.

[0074] Step S22: Divide the original array of length n into k sub-arrays, each sub-array having a size of m=n / k.

[0075] In this step, the subarray size corresponding to each subarray is the same as m. Based on the determined subarray size m, the original array of length n can be divided into k subarrays, and the size of each subarray is m=n / k, that is, k=n / m. For example, n=51, m=17, then k=3, and the original array needs to be divided into 3 subarrays, each of which has a size of 17.

[0076] Step S23: when a change in the data scale or access mode is detected, the sub-array size m is re-determined, and the original array of length n is re-divided.

[0077] In this embodiment, when it is detected that the data scale or access mode of the original data has changed, the subarray size m can be re-determined, and the original array of length n can be re-divided to obtain new k subarrays.

[0078] The block strategy (sub-array partitioning strategy) of this embodiment divides the original array into multiple sub-arrays according to the distribution characteristics of the data. By reducing the number of elements queried each time, the time complexity of the query is reduced and the query efficiency is improved. The block size can be flexibly adjusted. By flexibly adjusting the block size, it can adapt to different data scales and access modes to optimize query performance and balance index storage overhead.

[0079] In another optional embodiment, the number of sub-arrays k may be determined based on the data partitioning requirement information (actual requirement, such as the number of elements, data type or data distribution) of the original array, and then the original array of length n may be partitioned based on the size of each sub-array m=n / k to obtain k sub-arrays. When it is detected that the data scale or access mode of the original data has changed, the number of sub-arrays k may be re-determined, the size of each sub-array m=n / k may be re-determined, and the original array of length n may be partitioned again to obtain new k sub-arrays.

[0080] In one example, chunking of a primitive array (such as a Java array) can be accomplished by the following code:

[0081] / / Block function, divide the original array into sub-arrays of specified size

[0082] private void buildChunks(){

[0083] int n = array.length;

[0084] for(int i=0; i <n;i+=chunkSize){

[0085] int end=Math.min(n,i+chunkSize);

[0086] subArrays.add(Arrays.copyOfRange(array,i,end)); / / Each subarray is chunkSize in size

[0087] }

[0088] }

[0089] In this step, assume that the original array used is {4,8,15,16,23,42,15,23,42}, and the block size (i.e. sub-array size) is 3. This original array will be divided into three sub-arrays:

[0090] Subarray 1: {4,8,15}

[0091] Subarray 2: {16,23,42}

[0092] Subarray 3: {15,23,42}.

[0093] In combination with the above embodiment, in another implementation, the embodiment of the present invention further provides a data query method. Specifically, in addition to the above steps, the method may also include steps S31 to S33:

[0094] Step S31: for each sub-array, generate an index for the sub-array according to the position of the data element in the sub-array.

[0095] In this embodiment, in the index building stage, for each sub-array obtained by division, a lightweight index may be generated for the sub-array according to the position of the data element in the sub-array in the sub-array. Specifically, the data elements in the sub-array and their positions in the sub-array may be saved in the index corresponding to the sub-array.

[0096] In an optional embodiment, the indexes of multiple sub-arrays and the positions of data elements in the sub-arrays are incremented starting from 0. For example, the first sub-array includes {4, 8, 15}, where data element 4 is at position 0 in the sub-array, data element 8 is at position 1 in the sub-array, and data element 15 is at position 2 in the sub-array. Then, for the first sub-array, the generated index is 0, and the index of the first sub-array includes: {4=0, 8=1, 15=2}. The second sub-array includes {16, 23, 42}, where data element 16 is at position 0 in the sub-array, data element 23 is at position 1 in the sub-array, and data element 42 is at position 2 in the sub-array. Then, for the second sub-array, the generated index is 1, and the index of the second sub-array includes: {16=0, 23=1, 42=2}, and so on.

[0097] Step S32: for a sub-array, when the same data element exists in the sub-array, determine the latest position of the same data element in the sub-array.

[0098] In this embodiment, for a sub-array, when the same data element exists in the sub-array, the latest position of the same data element in the sub-array can be determined.

[0099] Step S33: Save the latest position of the same data element in the sub-array in the index of the sub-array.

[0100] In this embodiment, after the latest position of the same data element in the sub-array is determined, the latest position of the same data element in the sub-array and the same data element may be stored in the index of the sub-array.

[0101] That is to say, when the same data element exists in the same sub-array, the position of the latter data element in the sub-array will prevail: the position of the previously recorded same data element will be overwritten, and only the latest position of the same data element in the sub-array will be retained. For example, in sub-array A, there are at least data element A, data element B, and data element A. When the position of the first data element A in the array is determined to be B1, data element A and position B1 are saved together in the index of sub-array A; when the second data element A is determined to be the same data element as the first data element A, the position of the second data element A in the sub-array is determined to be B2, and B2 is overwritten with B1, retaining the latest position B2 of data element A in the sub-array. At this time, data element A is saved together with position B2 in the index of sub-array A.

[0102] In this embodiment, the index coverage strategy of the same data element is used to store the latest position of the element, which effectively solves the problem of duplicate value coverage, simplifies index maintenance, reduces the computational complexity of repeated queries, and ensures that the index always points to the latest data location. This method is particularly suitable for application scenarios that require frequent data updates to ensure the accuracy of real-time queries. In addition, the index coverage strategy of this embodiment can build an index without sorting, which reduces the time complexity of index construction. It is more effective in dynamic data and is suitable for dynamically updated data sets. It avoids the complexity of saving all historical positions while maintaining the simplicity of the index and the efficiency of the query.

[0103] In one example, index building can be achieved by the following code:

[0104] / / Build a local index for each sub-array (that is, build the index of the sub-array)

[0105] private void buildLocalIndexes(){

[0106] for(int i=0; i <subArrays.size();i++){

[0107] int[]subArray=subArrays.get(i);

[0108] Map<Integer,Integer> localIndex=new HashMap<>();

[0109] for(int j=0; j <subArray.length;j++){

[0110] localIndex.put(subArray[j],j); / / Build the local index of the subarray and record the position in the subarray

[0111] }

[0112] chunkedIndexes.add(localIndex); / / Save the local index of each sub-array

[0113] }

[0114] }

[0115] When building a local index, each subarray will have a hash table that records the element and its position in the subarray. The following are the subarrays and the corresponding local indexes:

[0116] Subarray 1: {4,8,15}

[0117] Local index: {4=0,8=1,15=2}

[0118] Subarray 2: {16,23,42}

[0119] Local index: {16=0,23=1,42=2}

[0120] Subarray 3: {15,23,42}

[0121] Local index: {15=0, 23=1, 42=2}.

[0122] In combination with the above embodiments, in one implementation, an embodiment of the present invention further provides a data query method. Specifically, in this method, the indexes of multiple sub-arrays are stored in the memory in the form of a hash table or an array, and the "perform a preliminary query based on the pre-constructed indexes of multiple sub-arrays to determine the target sub-array where the target data element is located" in the above step S12 may specifically include step S41:

[0123] Step S41: According to the target data element, traverse the index of each sub-array to determine the target sub-array where the target data element is located.

[0124] In this embodiment, the preliminary query process is supported by the index, and the target sub-array can be quickly locked through the hash map, which greatly shortens the search time. Compared with the linear scan of the entire array, the preliminary query significantly improves the query efficiency, especially in the scenario where the data is unevenly distributed or has a hierarchical structure. Specifically, it can be based on the target data element to traverse the index of each sub-array, and search for the target data element in the index of each sub-array in turn until the target data element is found, and determine the target sub-array where the target data element is located. In another optional example, if the target data element is not found after traversing the index of each sub-array, the search result of "search failed" is output at this time, and the data query is stopped.

[0125] In this embodiment, the target sub-array where the target data element is located can be quickly located through the index, reducing the linear traversal direction; in the case of uneven data distribution, the query efficiency can be significantly improved to adapt to complex data set structures.

[0126]

[0127] For example, query element 23

[0128] Iterate over the local index of each subarray:

[0129] 23 is not found in the local index {4=0,8=1,15=2} of subarray 1.

[0130] 23 is found in the local index {16=0, 23=1, 42=2} of sub-array 2, and it is determined that the target element 23 is located in sub-array 2.

[0131] result:

[0132] A preliminary query determines that 23 is located in subarray 2 (i.e., the second subarray).

[0133] For example, if you query for non-existent element 100

[0134] Iterate over the local index of each subarray:

[0135] 100 is not found in the local index {4=0,8=1,15=2} of subarray 1.

[0136] 100 is not found in the local index {16=0,23=1,42=2} of subarray 2.

[0137] 100 is also not found in the local index {15=0, 23=1, 42=2} of subarray 3.

[0138] Result: The initial query cannot find element 100 and returns -1.

[0139] In combination with the above embodiments, in one implementation, the present invention also provides a data query method. Specifically, in this method, the above step S13 may include steps S51 to S52:

[0140] Step S51: Compare each data element in the target sub-array with the target data element respectively until the target data element is found in the target sub-array, and determine the local position of the target data element in the target sub-array.

[0141] In this embodiment, in the re-query stage, each data element in the target sub-array may be compared with the target data element respectively until the target data element is found in the target sub-array, and the local position of the target data element in the target sub-array is determined, wherein the local position is the position of the target data element in the target sub-array, and the local position increases sequentially starting from 0.

[0142] Step S52: Determine the position of the target data element in the original array according to the index of the target sub-array, the sub-array size and the local position.

[0143] In this embodiment, the position of the target data element in the original array can be determined according to the index of the target sub-array, the sub-array size and the local position of the target data element in the target sub-array. Specifically, the sub-array size of this embodiment is the size of the sub-array obtained by dividing the original array, and in this embodiment, the size of each sub-array is the same.

[0144] In an optional example, the position of the target data element in the original array can be specifically: (index of the target subarray×subarray size)+local position of the target data element in the target subarray, thereby realizing fast and accurate query of the target data element. This embodiment reduces the cost of accurate query and improves the efficiency of real-time query by narrowing the query scope.

[0145] In one example, re-query (precise query) can be implemented by the following code:

[0146] / / Precise query: perform precise search in the locked sub-array

[0147] public int preciseQuery(int target){

[0148] int chunkIndex = preliminaryQuery(target); / / Call preliminary query to get subarray index

[0149] if (chunkIndex!=-1) {

[0150] Map<Integer,Integer> localIndex=chunkedIndexes.get(chunkIndex);

[0151] int localPos = localIndex.get(target); / / Get the position of the target element in the subarray

[0152] return chunkIndex*chunkSize+localPos; / / Return the exact position of the element in the original array

[0153] }

[0154] return -1; / / If not found, return -1

[0155] }

[0156] In sub-array 2, the local index {16=0, 23=1, 42=2} shows that 23 is located at the first position of sub-array 2.

[0157] result:

[0158] The position of 23 in the original array is:

[0159] Original position = (subarray index × subarray size) + local position = (1 × 3) + 1 = 4

[0160] Therefore, 23 is at position 4 in the original array.

[0161] In combination with the above embodiments, in one implementation, the present invention also provides a data query method. Specifically, in the method, in addition to the above steps, step S61 may be included, and the above step S21 may specifically include step S62:

[0162] Step S61: Determine the update frequency of the original array.

[0163] Step S62: when the update frequency is greater than the target frequency, determine the sub-array size m according to the data partitioning requirement information.

[0164] In this embodiment, the update frequency of the original array can be determined, and the update frequency can be compared with the target frequency. The target frequency of this embodiment is a preset minimum update frequency representing frequent updates. The specific value can be set arbitrarily and there is no restriction on this. In this embodiment, when it is determined that the update frequency is greater than the target frequency, it means that the original array is updated frequently. At this time, it is necessary to redetermine the sub-array size m based on the data division requirement information of the original array, and re-divide the sub-array to obtain the newly divided sub-array.

[0165] Figure 2 This is an example diagram of an array query method based on the MapReduce concept proposed in an embodiment of the present invention. The array query method based on the MapReduce concept, based on the MapReduce divide-and-conquer concept, realizes efficient query of large-scale Java arrays through block and covering index construction strategies, significantly improves the real-time query capability of large-scale data sets, shows significant advantages in time and space complexity, is suitable for frequently updated data environments, and has wide application potential. Specifically, Figure 2 As shown, in Figure 2 In the process, first, data segmentation is performed: the original array is divided into multiple smaller sub-arrays according to the actual needs of the original array (such as the number of elements, data type or data distribution); second, index construction is performed: a lightweight index is generated for each sub-array, and the elements in the sub-array and their positions in the sub-array are saved in the index. If there are identical elements in multiple sub-arrays, the original positions will be overwritten, and the latest positions of the identical elements in the sub-array will be retained; then, a preliminary query is performed: the target sub-array where the target element is located is quickly located through a preliminary query to reduce the range of elements that need to be traversed. The preliminary query process is supported by the index, and the sub-array is quickly locked through a hash map; finally, a precise query is performed: one by one query is performed in the target sub-array to ensure the accuracy of the query and to achieve fast and accurate positioning of the target element.

[0166] The related technologies have the problem of low efficiency when processing large-scale data sets: the traditional linear query method is not sufficient to support the requirements of real-time query when the data volume is huge. Although there are advanced algorithms such as binary search, pre-sorting and B-tree, these algorithms have high requirements for data preprocessing, such as pre-sorting of arrays or additional memory overhead. Especially when it is necessary to process frequently updated or dynamically expanded arrays, the cost of maintaining these structures is high.

[0167] The embodiment of the present invention uses the above-mentioned segmented indexing method, which does not rely on array pre-sorting and reduces the complexity of index maintenance, and is particularly suitable for efficient query of dynamic and unstructured data sets. Compared with pre-sorting or similar pre-built B-tree index solutions, the segmented indexing method proposed in this embodiment has significant advantages in both time and space complexity:

[0168] 1. Comparison of time complexity:

[0169] 1.1. Time complexity of pre-sorting

[0170] Time complexity of sorting: For an array of length n, the time complexity of pre-sorting is usually O(nlogn), such as using algorithms such as quick sort or merge sort.

[0171] Time complexity of query: After sorting, binary search can find the target element in O(logn) time.

[0172] 1.2. Time complexity of B-tree index

[0173] Time complexity of building a B-tree: B-tree is a balanced tree whose construction time complexity is O(nlogm), where n is the number of elements and m is the order of the B-tree.

[0174] Time complexity of query: In a B-tree, the time complexity of finding an element is O(logm n).

[0175] 1.3. Time complexity of constructing indexes in segments in this embodiment

[0176] Time complexity of index construction: When constructing the index in segments, the array is divided into k sub-arrays, each of which has a size of m = n / k. The time complexity of constructing the index on each sub-array is O(m), and the total time complexity is O(n). In the absence of sorting requirements, the time complexity of index construction is low.

[0177] Time complexity of query: The query process includes two parts: preliminary query and precise query. The time complexity of preliminary query is O(1) (through hash table or direct positioning), and the time complexity of precise query is O(m). Therefore, the time complexity of the overall query is about O(m), that is, O(n / k).

[0178] Time complexity advantage:

[0179] Segmented index building avoids the high time overhead of full array sorting, especially in dynamic data sets that need to be updated frequently, and can quickly build indexes to respond to query requests. Compared with B-trees, segmented index building is simpler and more direct, reducing balancing operations and complex tree node management operations, and is particularly suitable for scenarios with large data volumes but simple structures.

[0180] 2. Comparison of space complexity

[0181] 2.1. Space complexity of pre-sorting

[0182] Space complexity of sorting: In the case of in-place sorting (such as quick sort), the space complexity is O(1); for non-in-place sorting (such as merge sort), the space complexity is O(n).

[0183] 2.2. Space complexity of B-tree index

[0184] B-tree space complexity: The space complexity of a B-tree depends mainly on the height of the tree and the amount of information stored in the nodes. Generally, the space complexity of a B-tree is O(n), but since each node needs to store multiple child nodes and key-value pairs, the overall space consumption is large, especially when the tree has a high node order.

[0185] 2.3. Space complexity of constructing indexes in segments in this embodiment

[0186] Space complexity of segment index: The space complexity of segment index is O(n / k).

[0187] O(n / k) because an index is generated for each subarray. Indexes are usually stored in hash tables or arrays, which consume less memory. In addition, since no global sorting or tree structure is required, the space overhead of the index is lower than that of B-tree or full array sorting.

[0188] Space complexity advantage:

[0189] The index constructed by segmentation on each sub-array is smaller in size and occupies less space overall, which is particularly suitable for memory-constrained environments. Compared with B-tree, segmentation index avoids the complexity of tree structure and reduces the additional space overhead of tree nodes.

[0190] 2.4 Comprehensive Advantage Analysis

[0191] Flexibility: Segmented index building does not require pre-sorting of arrays, and is suitable for dynamic data sets and real-time query requirements. It avoids the complexity of sorting and tree structure maintenance, and performs well in scenarios where data is frequently updated.

[0192] Time efficiency: Since the query time complexity of segmented index construction depends on the size of the sub-array O(n / k) rather than the size of the entire array, the query efficiency is significantly improved with reasonable block division, especially for large-scale arrays.

[0193] Space efficiency: The space complexity of building indexes by segments is low, especially for large-scale data. It avoids the extra space overhead caused by sorting or B-tree structure and can use memory more efficiently.

[0194] Furthermore, the segmented indexing scheme of the embodiment of the present invention is applicable to large-scale data sets that need to be frequently queried and updated, and performs well in real-time data processing and online trading systems. In comparison, pre-sorting is applicable to static data, and B-tree index is suitable for use in scenarios that require balanced query and insert operations.

[0195] Compared with the above-mentioned related technologies, the embodiments of the present invention have the following main advantages:

[0196] No need for pre-sorting: This solution does not rely on pre-sorting of the array, avoiding the extra overhead caused by sorting, and is suitable for dynamically changing array data; Efficient query: Through block and index construction, the query scope is greatly narrowed, especially when processing large-scale data, the query efficiency can be significantly improved; Low index maintenance cost: Compared with advanced index structures such as B-tree, the index construction and maintenance process of the present invention is simpler and suitable for scenarios with frequent updates; Strong scalability: The solution allows the block size and index strategy to be adjusted according to actual needs, and can flexibly respond to data sets of different sizes and structures.

[0197] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0198] Based on the same inventive concept, an embodiment of the present invention provides a data query device 300. Figure 3 , Figure 3 FIG. 1 is a structural block diagram of a data query device provided by an embodiment of the present invention. Figure 3 As shown, the device 300 includes:

[0199] The request receiving module 301 is used to receive a data query request for a target data element sent by an online transaction client;

[0200] A preliminary query module 302 is used to perform a preliminary query based on the indexes of a plurality of pre-built sub-arrays to determine the target sub-array where the target data element is located; the plurality of sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array;

[0201] A re-query module 303 is used to perform a re-query in the target sub-array to determine the position of the target data element in the original array;

[0202] The result returning module 304 is used to return the result of the re-query to the online transaction client.

[0203] Optionally, the device 300 further includes:

[0204] A data size determination module is used to determine the sub-array size m according to the data partitioning requirement information;

[0205] A first data partitioning module is used to partition an original array of length n into k sub-arrays, each sub-array having a size of m=n / k;

[0206] The second data partitioning module is used to redetermine the sub-array size m and re-partition the original array of length n when a change in data scale or access mode is detected.

[0207] Optionally, the device 300 further includes:

[0208] An index generation module, for generating an index for each sub-array according to the position of the data element in the sub-array in the sub-array;

[0209] A position determination module, for determining, for a sub-array, when the same data element exists in the sub-array, the latest position of the same data element in the sub-array;

[0210] The position saving module is used to save the latest position of the same data element in the sub-array in the index of the sub-array.

[0211] Optionally, the indexes of the plurality of sub-arrays are stored in a memory in the form of a hash table or an array; the preliminary query module 302 comprises:

[0212] The preliminary query submodule is used to traverse the index of each subarray according to the target data element and determine the target subarray where the target data element is located.

[0213] Optionally, the re-query module 303 includes:

[0214] An element comparison submodule, used for comparing each data element in the target subarray with the target data element respectively, until the target data element is found in the target subarray, and determining a local position of the target data element in the target subarray;

[0215] The re-query submodule is used to determine the position of the target data element in the original array according to the index of the target sub-array, the sub-array size and the local position.

[0216] Optionally, the device 300 further includes:

[0217] A frequency determination module, used to determine the update frequency of the original array;

[0218] The data size determination module comprises:

[0219] The data size determination submodule is used to determine the subarray size m according to the data partitioning requirement information when the update frequency is greater than the target frequency.

[0220] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data query method described in any of the above embodiments of the present invention are implemented.

[0221] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 400, such as Figure 4 shown. Figure 4 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device comprises a memory 402, a processor 401 and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the data query method according to any of the above embodiments of the present invention when executing the computer program.

[0222] Based on the same inventive concept, another embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the data query method when executed by a processor.

[0223] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0224] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0225] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0226] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0228] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0229] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0231] The data query method, device, equipment, medium and product provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained in this article by using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data query method, characterized in that: The method comprises: Receiving a data query request for a target data element sent by an online transaction client; Performing a preliminary query based on the indexes of a plurality of pre-constructed sub-arrays to determine the target sub-array where the target data element is located; the plurality of sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array; Performing a re-query in the target sub-array to determine the position of the target data element in the original array; The result of the re-query is returned to the online transaction client.

2. The data query method according to claim 1, characterized in that: The method further comprises: Determine the sub-array size m based on the data partitioning requirement information; Divide the original array of length n into k sub-arrays, each sub-array has a size of m = n / k; When a change in data size or access pattern is detected, the subarray size m is re-determined and the original array of length n is re-partitioned.

3. The data query method according to claim 1, characterized in that: The method further comprises: For each sub-array, generate an index for the sub-array according to the position of the data element in the sub-array; For a sub-array, when the same data element exists in the sub-array, determining the latest position of the same data element in the sub-array; The latest position of the same data element in the sub-array is saved in the index of the sub-array.

4. The data query method according to any one of claims 1 to 3, characterized in that: The indexes of the plurality of sub-arrays are stored in the memory in the form of a hash table or an array; the preliminary query is performed according to the pre-built indexes of the plurality of sub-arrays to determine the target sub-array where the target data element is located, comprising: According to the target data element, the index of each sub-array is traversed to determine the target sub-array where the target data element is located.

5. The data query method according to any one of claims 1 to 3, characterized in that: The re-querying in the target sub-array to determine the position of the target data element in the original array includes: Comparing each data element in the target sub-array with the target data element respectively until the target data element is found in the target sub-array, and determining a local position of the target data element in the target sub-array; The position of the target data element in the original array is determined according to the index of the target sub-array, the sub-array size and the local position.

6. The data query method according to claim 2, characterized in that: The method further comprises: Determining the update frequency of the original array; The step of determining the subarray size m according to the data partitioning requirement information includes: When the update frequency is greater than the target frequency, the sub-array size m is determined according to the data partitioning requirement information.

7. A data query device, characterized in that: The device comprises: A request receiving module, used for receiving a data query request for a target data element sent by an online transaction client; A preliminary query module, used to perform preliminary query according to the indexes of a plurality of pre-built sub-arrays to determine the target sub-array where the target data element is located; the plurality of sub-arrays constitute the original array, and the index of a sub-array includes each data element in the sub-array and the position of each data element in the sub-array; A re-query module, used for re-querying in the target sub-array to determine the position of the target data element in the original array; The result returning module is used to return the result of the re-query to the online transaction client.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the data query method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data query method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the data query method according to any one of claims 1 to 6 is implemented.

Images