Data retrieval method, device, computer equipment and storage medium

Abstract

The present application relates to a data retrieval method, device, computer equipment, storage medium and computer program product. The method includes: obtaining a retrieval request, determining a main dimension and a to-be-processed dimension among multiple retrieval dimensions corresponding to the retrieval request; determining the amount of data corresponding to each main dimension member in the main dimension; constituting the to-be-decomposed dimension with the main dimension members whose data amount meets the threshold condition, selecting one of the retrieval dimensions from the to-be-processed dimension as a secondary dimension, and constituting at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension; if there is a to-be-decomposed dimension whose data amount meets the threshold condition in the newly constituted combined dimension, then selecting another retrieval dimension from the to-be-processed dimension as a secondary dimension to continue decomposing, until the data amount corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition, and finally obtaining a retrieval result corresponding to the retrieval request. The use of this method can improve the efficiency of data retrieval.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of the big data era, it is becoming increasingly difficult to retrieve and query data. In digital systems, data retrieval is the most basic and frequent operation. Rapid data retrieval relies on a large number of index strategies. In order to avoid wasting storage space and overly large indexes, multidimensional databases are usually used to store massive amounts of multidimensional data.

[0003] In traditional technology, multi-dimensional data retrieval can generally be performed using a multi-threaded or distributed retrieval strategy on the database side. However, this retrieval method may cause memory overflow due to uneven data distribution on a single thread, and may also cause database crashes due to excessive data volume retrieved at a single time, thus affecting the efficiency of data retrieval. Summary of the invention

[0004] Based on this, it is necessary to provide a data retrieval method, apparatus, computer device, computer-readable storage medium and computer program product that can improve retrieval efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data retrieval method. The method comprises:

[0006] Obtaining a search request, and determining a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0007] Determine the amount of data corresponding to each main dimension member in the main dimension;

[0008] The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, one of the retrieval dimensions is selected from the dimensions to be processed as the secondary dimension, and at least one new combined dimension is constituted based on each secondary dimension member of the secondary dimension and the dimension to be decomposed;

[0009] If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0010] Based on each combined dimension obtained by the final decomposition, a corresponding search subtask is generated and executed, and based on the execution result of each search subtask, a search result corresponding to the search request is determined.

[0011] In one embodiment, the method further comprises:

[0012] If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, the main dimension member whose data volume does not meet the threshold condition is used as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension;

[0013] If there is a combined dimension whose data volume does not meet the threshold condition among the combined dimensions, the combined dimension whose data volume does not meet the threshold condition is taken as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension;

[0014] The determining of the search result corresponding to the search request based on the execution result of each search subtask includes:

[0015] Based on the execution results of the retrieval subtasks corresponding to the target dimensions and the execution results of the retrieval subtasks corresponding to the combined dimensions obtained by the final decomposition, a retrieval result corresponding to the retrieval request is determined.

[0016] In one of the embodiments, if there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, taking the main dimension member whose data volume does not meet the threshold condition as the target dimension includes:

[0017] If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, the main dimension members corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension;

[0018] If there is a combined dimension whose data volume does not meet the threshold condition among the combined dimensions, taking the combined dimension whose data volume does not meet the threshold condition as the target dimension includes:

[0019] If there are combined dimensions whose data volume does not meet the threshold condition, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, then the combined dimensions corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension.

[0020] In one embodiment, determining a main dimension and a to-be-processed dimension among a plurality of retrieval dimensions corresponding to the retrieval request includes:

[0021] Determine a plurality of search dimensions corresponding to the search request, wherein each search dimension includes at least one dimension member;

[0022] The search dimension including the largest number of dimension members is taken as the main dimension, and the other search dimensions among the multiple search dimensions except the main dimension are taken as dimensions to be processed.

[0023] In one of the embodiments, each search dimension is sorted from large to small according to the number of dimension members included;

[0024] The selecting one of the search dimensions from the dimensions to be processed as the secondary dimension includes:

[0025] According to the sorting order of the dimensions to be processed, a search dimension that has not been combined is selected from the dimensions to be processed as a secondary dimension.

[0026] In one of the embodiments, the number of main dimension members included in the main dimension is not less than the total number of retrieval threads.

[0027] In a second aspect, the present application also provides a data retrieval device. The device comprises:

[0028] An acquisition module, used to acquire a search request and determine a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0029] A determination module, used to determine the amount of data corresponding to each main dimension member in the main dimension;

[0030] A selection module, configured to form a to-be-decomposed dimension from the main dimension members whose data volume meets a threshold condition, select one of the retrieval dimensions from the to-be-processed dimensions as a secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension;

[0031] A return module, configured to return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension if there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0032] The determination module is further used to generate and execute corresponding retrieval subtasks based on each combined dimension finally decomposed, and determine the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask.

[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] Obtaining a search request, and determining a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0035] Determine the amount of data corresponding to each main dimension member in the main dimension;

[0036] The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, one of the retrieval dimensions is selected from the dimensions to be processed as the secondary dimension, and at least one new combined dimension is constituted based on each secondary dimension member of the secondary dimension and the dimension to be decomposed;

[0037] If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0038] Based on each combined dimension obtained by the final decomposition, a corresponding search subtask is generated and executed, and based on the execution result of each search subtask, a search result corresponding to the search request is determined.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Obtaining a search request, and determining a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0041] Determine the amount of data corresponding to each main dimension member in the main dimension;

[0042] The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, one of the retrieval dimensions is selected from the dimensions to be processed as the secondary dimension, and at least one new combined dimension is constituted based on each secondary dimension member of the secondary dimension and the dimension to be decomposed;

[0043] If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0044] Based on each combined dimension obtained by the final decomposition, a corresponding search subtask is generated and executed, and based on the execution result of each search subtask, a search result corresponding to the search request is determined.

[0045] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0046] Obtaining a search request, and determining a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0047] Determine the amount of data corresponding to each main dimension member in the main dimension;

[0048] The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, one of the retrieval dimensions is selected from the dimensions to be processed as the secondary dimension, and at least one new combined dimension is constituted based on each secondary dimension member of the secondary dimension and the dimension to be decomposed;

[0049] If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0050] Based on each combined dimension obtained by the final decomposition, a corresponding search subtask is generated and executed, and based on the execution result of each search subtask, a search result corresponding to the search request is determined.

[0051] The above-mentioned data retrieval method, device, computer equipment, storage medium and computer program product determine the main dimension and the to-be-processed dimension in the multiple retrieval dimensions corresponding to the retrieval request, the main dimension can determine the main scope of the data retrieved this time, and the to-be-processed dimension can assist the secondary dimension division of the main dimension in the retrieval process. Then determine the amount of data corresponding to each main dimension member in the main dimension, so that it can be determined whether the main dimension member needs to be decomposed according to the amount of data corresponding to each main dimension member. The main dimension members whose data volume meets the threshold condition constitute the to-be-decomposed dimension, select one of the retrieval dimensions from the to-be-processed dimension as the secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension, so that the secondary dimension is used to decompose the to-be-decomposed dimension composed of the main dimension members whose data volume meets the threshold condition, that is, the secondary dimension is used to decompose the to-be-decomposed dimension with a large data volume, which can reduce the amount of data contained in the to-be-decomposed dimension, so that the thread executing the corresponding retrieval subtask will not have the risk of memory overflow, jamming or crash. If there are still dimensions to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension, a new secondary dimension is selected to decompose the newly formed combined dimension again, and the judgment iteration is continuously performed until the data volume corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition, so as to ensure that the data volume of each combined dimension obtained will not exceed the performance range of the thread executing the corresponding retrieval subtask. Finally, the corresponding retrieval subtask is generated and executed based on each combined dimension obtained by the final decomposition, and the retrieval result corresponding to the retrieval request is determined based on the execution result of each retrieval subtask. In this way, for the retrieval of multi-dimensional data, the main dimension data in the retrieval request is progressively decomposed by using one or more secondary dimensions, so that the data volume of all combined dimensions obtained in the end is within a completely controllable range, avoiding the problem of long single-node processing time or memory overflow caused by uneven data distribution when each thread executes the retrieval task, and also avoiding the problem of database read and write blocking or downtime caused by excessive I / O and computing processing pressure of the database during a single query, thereby accelerating the efficiency of data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 An application environment diagram of a data retrieval method in an embodiment;

[0053] Figure 2 is a flow chart of a data retrieval method in one embodiment;

[0054] Figure 3 A flowchart showing a summary of the idea of ​​a data retrieval method in one embodiment;

[0055] FIG. 4( a ) is a data structure of single-dimensional statistics in a data retrieval method in one embodiment;

[0056] FIG4( b ) is a data structure of dual-dimensional statistics in a data retrieval method in one embodiment;

[0057] Figure 5 A detailed flow chart of a data retrieval method in one embodiment;

[0058] Figure 6 is a structural block diagram of a data retrieval device in one embodiment;

[0059] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The data retrieval method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data retrieval method mentioned in each embodiment of the present application can be implemented by the terminal and the server separately, or it can be implemented by the terminal and the server in collaboration. Taking the data retrieval method in the present application as an example, the user can obtain a retrieval request through the terminal, determine the main dimension and the dimension to be processed in the multiple retrieval dimensions corresponding to the retrieval request, and determine the data volume corresponding to each main dimension member in the main dimension, and then form the dimension to be decomposed with the main dimension members whose data volume meets the threshold condition, and then select one of the retrieval dimensions from the dimension to be processed as the secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the dimension to be decomposed. If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension, then return to the step of selecting one of the retrieval dimensions from the dimension to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition, and finally generate and execute the corresponding retrieval subtask based on each combined dimension obtained by the final decomposition, and determine the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0062] In one embodiment, Figure 2 As shown, a data retrieval method is provided, which is applied to a computer device (the computer device can be specifically Figure 1 The terminal or server in the example is used to illustrate, including the following steps:

[0063] Step S202: Obtain a search request, and determine a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request.

[0064] Among them, the main dimension is used to define the overall scope of this search, and the dimension to be processed is used to assist the main dimension in performing secondary dimensional division during the search process.

[0065] Specifically, the user may initiate a search request through a computer device, and the search request carries multiple search dimensions of this search query. Then, the computer device may determine the main dimension and the to-be-processed dimension from the multiple search dimensions.

[0066] For example, a computer device receives a search request from a user, and the search request requires obtaining the financial data of N organizations within the company within the fourth quarter of M years. In this case, the department can be used as the main dimension of this search, and the year and quarter can be used as the dimensions to be processed for this search.

[0067] Step S204, determining the amount of data corresponding to each main dimension member in the main dimension.

[0068] The main dimension includes multiple main dimension members, each of which includes multiple data. The data volume can be the total memory of the data included in each main dimension member, or the number of data included in each main dimension member, which is not limited here.

[0069] Specifically, the computer device determines the main dimension members contained in the main dimension according to the main dimension in the retrieval request, and estimates the amount of data contained in each of the main dimension members. The estimate can be a cache after statistics of the amount of data of the dimension member during the last retrieval, or it can be statistics of the data of the dimension member when it is written to the database.

[0070] For example, if the search request is to obtain the financial data of N organizations within the company within the fourth quarter of M years, then the main dimension is the organization, which contains N sub-organizations, and each sub-organization is a main dimension member of the main dimension. In a previous search for the organization's financial data, the amount of financial data contained in each sub-organization has been queried, and the amount of financial data is P1, P2, ..., Pn. The information on the amount of financial data is cached in the computer device. Therefore, for this search for the financial data of the main dimension organization, the amount of data corresponding to each main dimension member sub-organization is P1, P2, ..., Pn.

[0071] Step S206: main dimension members whose data volume meets the threshold condition constitute dimensions to be decomposed.

[0072] Among them, the threshold condition is used to determine how the main dimension members or combined dimensions containing different data volumes should be retrieved. The threshold condition can be that the memory corresponding to the data volume of each main dimension member or combined dimension is greater than the memory of the single thread that executes the retrieval subtask, or the number of data volumes of each main dimension member or combined dimension is greater than the threshold number that can be processed by the single thread that executes the retrieval subtask, or the number of data volumes of each main dimension member or combined dimension is greater than a preset multiple of the threshold number, such as 1.2 times, which is not limited here. Specifically, the computer device forms the main dimension members whose data volumes meet the threshold condition into dimensions to be decomposed.

[0073] Step S208: Select one of the search dimensions from the dimensions to be processed as a secondary dimension, and construct at least one new combined dimension based on each secondary dimension member of the secondary dimension and the dimension to be decomposed.

[0074] Specifically, the computer device may select one of the retrieval dimensions from the dimensions to be processed as a sub-dimension, wherein the sub-dimension includes at least one sub-dimension member, and then each sub-dimension member of the sub-dimension and the dimension to be decomposed respectively form a new combined dimension.

[0075] In one of the embodiments, the retrieval request is to obtain the financial data of N organizations within the company in 2016-2020, each year including 4 quarters, the main dimension is organization, the dimensions to be processed are year and quarter, and the main dimension organization contains A1, A2, ..., An sub-organizations. If the number of data contained in sub-organization A3 is 1,000, and the single thread that executes the corresponding retrieval subtask can process only 500 data, then sub-organization A3 is constituted as the dimension to be decomposed, and then the year is selected from the dimension to be processed as the sub-dimension, and the sub-dimension contains five sub-dimension members of 2016, 2017, 2018, 2019 and 2020, and then sub-organization A3 is respectively combined with 2016, 2017, 2018, 2019 and 2020 to form a new combined dimension.

[0076] Step S210, determine whether there is a dimension to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension. If yes, return to step S208, if not, proceed to step S212.

[0077] Specifically, the computer device estimates the amount of data in each newly constructed combined dimension, determines whether the amount of data in each newly constructed combined dimension meets the threshold condition, takes the new combined dimension whose data amount meets the threshold condition as the dimension to be decomposed, selects an unused retrieval dimension from the dimensions to be processed as the sub-dimension, and continues to construct at least one new combined dimension based on each sub-dimensional member of the sub-dimension and the dimension to be decomposed, until the data amount corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition.

[0078] In one of the embodiments, the computer device estimates the amount of data in the new combined dimension formed by the sub-organization A3 and the year 2018 obtained in the aforementioned embodiment. If the memory corresponding to the amount of data in the new combined dimension is larger than the memory of a single thread executing the retrieval subtask, then the new combined dimension is used as the dimension to be decomposed, and the quarterly dimension is selected from the dimensions to be processed as the secondary dimension. Based on the four secondary dimension members of the quarterly dimension (the first quarter, the second quarter, the third quarter and the fourth quarter) and the dimension to be decomposed (the year 2018 of the A3 sub-organization), four new combined dimensions are formed, namely, the first quarter of 2018 of the A3 organization, the second quarter of 2018 of the A3 organization, the third quarter of 2018 of the A3 organization, and the fourth quarter of 2018 of the A3 organization. The computer device then estimates the amount of data in each new combined dimension, and determines whether the amount of data in each newly formed combined dimension meets the threshold condition, and so on, until the memory corresponding to the amount of data of each combined dimension finally decomposed is smaller than the memory of a single thread executing the retrieval subtask.

[0079] Step S212: Generate and execute corresponding search subtasks based on each combined dimension finally decomposed, and determine the search result corresponding to the search request based on the execution result of each search subtask.

[0080] Each retrieval subtask is executed by a single thread or a single node.

[0081] Specifically, the computer device generates a corresponding retrieval subtask for each combined dimension finally decomposed, and performs a retrieval operation based on each retrieval subtask on a single thread or node. The retrieval results obtained after the retrieval of each retrieval subtask constitute a part of the retrieval results corresponding to the retrieval request.

[0082] In one of the embodiments, when the amount of data contained in each combined dimension finally decomposed is smaller than the amount of data that can be processed by a single thread, a corresponding search subtask is generated for each combined dimension. For example, the combined dimension A3 in the aforementioned embodiment is used to generate a corresponding search subtask for the first quarter of 2018, and a single thread is used to perform a search on the search subtask. The search results obtained constitute a part of the search results corresponding to the search request.

[0083] In the above data retrieval method, firstly, a data retrieval request is obtained, and the main dimension and the to-be-processed dimension in the multiple retrieval dimensions corresponding to the retrieval request are determined. The main dimension can determine the main scope of the data retrieved this time, and the to-be-processed dimension can assist the secondary dimension division of the main dimension in the retrieval process. Then, the amount of data corresponding to each main dimension member in the main dimension is determined, so that it can be determined whether the main dimension member needs to be decomposed according to the amount of data corresponding to each main dimension member. The main dimension members whose data volume meets the threshold condition constitute the to-be-decomposed dimension, and one of the retrieval dimensions is selected from the to-be-processed dimension as the secondary dimension, and at least one new combined dimension is formed based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension, so that the secondary dimension is used to decompose the to-be-decomposed dimension composed of the main dimension members whose data volume meets the threshold condition, that is, the secondary dimension is used to decompose the to-be-decomposed dimension with a large data volume, so as to reduce the amount of data contained in the to-be-decomposed dimension, so that the thread executing the corresponding retrieval subtask will not have the risk of memory overflow, jamming or downtime. If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, a new secondary dimension is selected to decompose the newly constructed combined dimension again, until the data volume corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition, so as to ensure that the data volume of each combined dimension obtained will not exceed the performance range of the thread executing the corresponding retrieval subtask. Finally, the corresponding retrieval subtask is generated and executed based on each combined dimension obtained by the final decomposition, and the retrieval result corresponding to the retrieval request is determined based on the execution result of each retrieval subtask. In this way, for the retrieval of multi-dimensional data, the main dimension data in the retrieval request is progressively decomposed by using one or more secondary dimensions, so that the data volume of all combined dimensions obtained in the end is within a completely controllable range, avoiding the problem of long single-node processing time or memory overflow caused by uneven data distribution when each thread executes the retrieval task, and also avoiding the problem of database read and write blocking or downtime caused by excessive I / O and computing processing pressure of the database during a single query, thereby accelerating the efficiency of data retrieval.

[0084] In one embodiment, the method also includes: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, then the main dimension members whose data volume does not meet the threshold condition are used as target dimensions, and the corresponding retrieval subtasks are generated and executed directly based on the target dimensions; if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, then the combined dimensions whose data volume does not meet the threshold condition are used as target dimensions, and the corresponding retrieval subtasks are generated and executed directly based on the target dimensions; based on the execution results of each retrieval subtask, the retrieval result corresponding to the retrieval request is determined, including: based on the execution results of the retrieval subtasks corresponding to each target dimension, and the execution results of the retrieval subtasks corresponding to each combined dimension obtained by the final decomposition, the retrieval result corresponding to the retrieval request is determined.

[0085] The data volume not meeting the threshold condition may be that the data volume is smaller than the data volume that can be processed by a single thread or a single node executing the retrieval subtask, or that the memory corresponding to the data volume is smaller than the memory of a single thread or a single node executing the retrieval subtask.

[0086] Specifically, if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, the computer device will take the main dimension members whose data volume does not meet the threshold condition as the target dimension, directly generate and execute the corresponding retrieval subtask based on the target dimension. If there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, the computer device will take the combined dimensions whose data volume does not meet the threshold condition as the target dimension, directly generate and execute the corresponding retrieval subtask based on the target dimension. Finally, the execution results of the retrieval subtasks corresponding to each target dimension are merged with the execution results of the retrieval subtasks corresponding to each combined dimension obtained by the final decomposition, and the retrieval results corresponding to the retrieval request are obtained.

[0087] In one embodiment, as described in the above embodiment, if the search request is to obtain the financial data of N organizations within the company in 2016-2020, each year including four quarters, and the main dimension organization contains A1, A2, ..., An sub-organizations, if the data volume of sub-organization A4 is less than the data volume that can be processed by a single thread executing the search sub-task, then the computer device uses sub-organization A4 as the target dimension, directly generates a corresponding search sub-task based on sub-organization A4, and performs the search by a single thread. If the data volume of the new combined dimension composed of sub-organization A3 and the secondary dimension member 2020 is less than the data volume that can be processed by a single thread executing the search sub-task, then the computer device uses the new combined dimension composed of sub-organization A3 and the secondary dimension member 2020 as the target dimension, and performs the search on the target dimension by a single thread. Finally, the execution results of the search sub-tasks corresponding to each target dimension under the main dimension organization are merged with the execution results of the search sub-tasks corresponding to each combined dimension obtained by the final decomposition, and the search results of the financial data of N organizations within the company in 2016-2020, each year including four quarters, are obtained.

[0088] In one of the embodiments, the data retrieval method includes the following steps: a computer device obtains a retrieval request, determines a main dimension and a to-be-processed dimension among a plurality of retrieval dimensions corresponding to the retrieval request; determines the amount of data corresponding to each main dimension member in the main dimension; if there is a main dimension member whose data amount does not meet a threshold condition among the main dimension members, then the main dimension member whose data amount does not meet the threshold condition is used as a target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension; if there is a main dimension member whose data amount meets the threshold condition among the main dimension members, then the main dimension member whose data amount meets the threshold condition is used to form a to-be-decomposed dimension; selects one of the retrieval dimensions from the to-be-processed dimensions as a secondary dimension, At least one new combined dimension is formed based on each secondary dimension member of the secondary dimension and the dimension to be decomposed; whether there is a dimension to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension; if there is a combined dimension whose data volume does not meet the threshold condition in the newly formed combined dimension, the combined dimension whose data volume does not meet the threshold condition is used as the target dimension, and the corresponding retrieval subtask is directly generated and executed based on the target dimension; if there is a dimension to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension, the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension is returned to continue execution until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition. When a single thread or node executes the task of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and forming at least one new combined dimension based on each secondary dimension member of the secondary dimension and the dimension to be decomposed, another thread or node can simultaneously execute the retrieval task of the target dimension in parallel. Based on the execution results of the retrieval subtasks corresponding to each target dimension and the execution results of the retrieval subtasks corresponding to each combined dimension finally decomposed, the retrieval result corresponding to the retrieval request is determined.

[0089] In this embodiment, the main dimension members and combined dimensions whose data volume does not meet the threshold conditions are used as target dimensions to directly generate retrieval subtasks, and the retrieval operation is performed on a single thread or node, ensuring that the data volume of a single retrieval subtask and the memory corresponding to the data volume will not exceed the upper limit of the data volume and memory upper limit that a single thread or node can handle, thereby avoiding problems such as uneven data volume distribution leading to long processing time and memory overflow for a single node, and also avoiding problems such as database read and write blocking and even downtime caused by excessive data processing, thereby accelerating the rate of data retrieval. At the same time, the formation of a new combined dimension and the parallel execution of the target dimension retrieval task also greatly accelerate the efficiency of data retrieval.

[0090] In one embodiment, if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, the main dimension members whose data volume does not meet the threshold condition are taken as the target dimension, including: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, the main dimension members corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension; if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, the combined dimensions whose data volume does not meet the threshold condition are taken as the target dimension, including: if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, the combined dimensions corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension.

[0091] Among them, the work of merging the main dimension members or combined dimensions whose total amount of data does not meet the threshold condition, and the work of retrieving the main dimension members or combined dimensions that meet the threshold condition, both can be executed in parallel through different nodes or threads.

[0092] Specifically, if the sum of the data amounts contained in multiple main dimension members does not meet the threshold condition, the computer device merges these main dimension members to obtain the target dimension. If the sum of the data amounts contained in multiple combined dimensions decomposed by the same secondary dimension does not meet the threshold condition, the computer device merges these combined dimensions to obtain the target dimension.

[0093] In one of the embodiments, the threshold condition is that the memory corresponding to the data volume is greater than 2GB. If the search request is to obtain the financial data of N organizations within the company in 2016-2020, each year includes four quarters, the main dimension is the organization, the dimensions to be processed are the year and the quarter, and the main dimension organization contains A1, A2, ..., An sub-organizations. For example, the total memory of the data volume contained in the main dimension members A5 and A6 is 1.5GB. Obviously, 1.5GB < 2GB, then the computer device merges the main dimension members A5 and A6 to form a target dimension. As described in the above embodiment, if the total memory of the data volume of the combined dimension sub-organization A3 in 2019 and the data volume of the sub-organization A3 in 2020 is 1.8GB, obviously, 1.8GB < 2GB, then the computer device merges the combined dimension sub-organization A3 in 2019 with the combined dimension sub-organization A3 in 2020 to form a target dimension.

[0094] In one of the embodiments, multiple combined dimensions whose data amounts after decomposition of the same dimension do not meet the threshold conditions are recombined to obtain at least one sub-combination. For data sources that support multi-condition OR parallel retrieval queries, one or more sub-combinations can be combined into a total combination based on the fact that the data amount does not meet the threshold conditions. The total combination constitutes the target dimension (the data amount corresponding to the total combination does not meet the threshold conditions), and the computer device uses a single thread or node to complete the retrieval subtask corresponding to the total combination. For data sources that do not support multi-condition OR parallel retrieval queries, each sub-combination can constitute a target dimension, and the retrieval subtask corresponding to each target dimension is executed by a separate thread or node.

[0095] For example, if the threshold condition is that the amount of data that a single thread can process is 5000, if the search request is to obtain the financial data of N organizations within the company in 2016-2020, each year includes four quarters, the main dimension is organization, the dimensions to be processed are year and quarter, and the main dimension organization contains A1, A2, ..., An sub-organizations. According to the threshold condition, multiple combined dimensions are obtained, including: the amount of data of sub-organization A7 in the first quarter of 2016 is 1000, the amount of data of sub-organization A7 in the second quarter of 2016 is 1000, the amount of data of sub-organization A7 in the third quarter of 2017 is 500, the amount of data of sub-organization A7 in the fourth quarter of 2017 is 500, the amount of data of sub-organization A8 in the third quarter of 2016 is 1000, and the amount of data of sub-organization A8 in the fourth quarter of 2016 is 1000. For data sources that support multi-condition OR parallel search queries, the above six combination dimensions can be merged without incorrect overlapping combination dimensions. For example, the data volume of sub-organization A7 in the first quarter of 2016 is 1000 and the data volume of sub-organization A7 in the second quarter of 2016 is 1000, and a sub-combination B1 is obtained. The data volume of sub-organization A7 in the third quarter of 2017 is 500 and the data volume of sub-organization A7 in the fourth quarter of 2017 is 500, and a sub-combination B2 is obtained. The data volume of sub-organization A8 in the third quarter of 2016 is 1000 and the data volume of sub-organization A8 in the fourth quarter of 2016 is 1000, and a sub-combination B3 is obtained. The total data volume of sub-combinations B1, B2 and B3 does not exceed 5000. Sub-combinations B1, B2 and B3 are merged into a total combination, and the total combination constitutes a target dimension. The search subtask corresponding to the target dimension is executed using one thread through the OR function. For data sources that do not support multi-condition OR parallel retrieval queries, the sub-combinations B1, B2, and B3 are each used to form a target dimension, and the retrieval subtask corresponding to each target dimension is executed through a separate thread.

[0096] In this embodiment, on the basis of not exceeding the threshold condition, multiple main dimension members or multiple combined dimensions are merged to form a target dimension, and a single thread can be used to complete the retrieval task of multiple main dimension members or multiple combined dimensions, avoiding thread waste, speeding up data retrieval, and saving retrieval time. For data sources that support multi-condition OR parallel retrieval queries, a thread can also be used to simultaneously execute multiple sub-combination retrieval subtasks on the basis of not meeting the threshold condition, further improving data retrieval efficiency.

[0097] In one embodiment, determining a main dimension and dimensions to be processed among multiple retrieval dimensions corresponding to a retrieval request includes: determining multiple retrieval dimensions corresponding to the retrieval request, wherein each retrieval dimension includes at least one dimension member; taking the retrieval dimension including the largest number of dimension members as the main dimension, and taking the other retrieval dimensions among the multiple retrieval dimensions except the main dimension as dimensions to be processed.

[0098] Specifically, the computer device determines multiple search dimensions corresponding to the search request according to the search request, each search dimension including at least one dimension member. The computer device uses the search dimension including the largest number of dimension members as the main dimension, and uses other search dimensions except the main dimension as dimensions to be processed.

[0099] In one of the embodiments, if the search request is to obtain the financial data of N organizations within the company in 2016-2020, each year including four quarters, if N>5, the organization is used as the main dimension, and the year and quarter are used as the dimensions to be processed.

[0100] In this embodiment, the computer device uses the retrieval dimension with the largest number of dimension members as the main dimension, so that the number of threads participating in the retrieval can be utilized as much as possible when estimating the data volume of the main dimension members, thereby reducing thread waste and improving data retrieval efficiency.

[0101] In one embodiment, the method further includes: sorting each retrieval dimension from large to small according to the number of dimension members it contains; selecting one of the retrieval dimensions from the dimensions to be processed as a sub-dimension, including: selecting an uncombined retrieval dimension from the dimensions to be processed as a sub-dimension according to the sorting order of the dimensions to be processed.

[0102] Specifically, the computer device sorts the search dimensions included in the search request from large to small according to the number of dimension members included, to form a dimension list. The computer device selects one uncombined search dimension from the dimensions to be processed one by one as a secondary dimension according to the sorting order of the dimensions to be processed.

[0103] In one embodiment, if the search request is to obtain data that satisfies the conditions of dimension 1, dimension 2, and dimension 3, where dimension 1 contains N dimension members, dimension 2 contains M dimension members, dimension 3 contains Q dimension members, and Q>M>N, then the dimensions are sorted in the order of dimension 3, dimension 2, and dimension 1. During the search, dimension 3 is used as the main dimension, and dimension 1 and dimension 2 are used as dimensions to be processed. If a search dimension needs to be selected as a secondary dimension, dimension 2 is selected first, and dimension 1 is selected second.

[0104] In this embodiment, each retrieval dimension is sorted from large to small according to the number of dimension members it contains, and a dimension list is determined. The computer device can select the required dimension for data retrieval based on this list, and first select the to-be-processed dimension with many dimension members as the secondary dimension. The main dimension members or combined dimensions with a large amount of data can be quickly divided into dimensions to be decomposed that do not meet the threshold conditions, thereby accelerating the data retrieval rate.

[0105] In one embodiment, the number of main dimension members included in the main dimension is not less than the total number of retrieval threads.

[0106] Specifically, for the retrieval of massive data, the number of main dimension members included in the main dimension selected by the computer device is not less than the total number of retrieval threads.

[0107] For example, the total number of threads participating in the retrieval is J, then the relationship between the number O of main dimension members contained in the main dimension of the retrieval request selected by the computer device and J should be O≥J.

[0108] In this embodiment, a dimension whose dimension members are greater than the number of search threads is selected as the main dimension, which can ensure that the number of threads participating in the search can be fully utilized when estimating the data volume of the main dimension members, thereby avoiding thread waste and accelerating data retrieval efficiency.

[0109] In another embodiment, the number of main dimension members included in the main dimension may also be less than the total number of search threads. This is based on actual circumstances and is not limited in this embodiment of the present application.

[0110] In a specific embodiment, Figure 3The figure shows a flowchart summarizing the idea of ​​using the data retrieval method of the present application for retrieval, which can be applied to the client during data retrieval. The first step is to sort all dimensions by the number of members; the second step is to select non-sparse dimensions, wherein non-sparse dimensions are dimensions with the most dimension members among undecomposed dimensions; the third step is to obtain the data distribution of each member of the dimension; the fourth step is divided into two cases. For dimension members with less data, query instructions can be generated individually or in combination, and then the data query task is executed through the corresponding thread to complete the data retrieval; for dimension members whose data volume exceeds the grouping threshold, secondary dimensional division is uniformly performed, and the execution is returned to the second step to continue until the required data query is completed. Figure 4 (a) and Figure 4 (b) respectively show the distribution of dimension members when multi-dimensional data is stored in the database according to a single-dimensional data structure and a double-dimensional data structure.

[0111] In a specific embodiment, Figure 5 The figure shows a detailed flow chart of the data retrieval method during use. The method can be applied to the client during data retrieval. The starting process of "multi-dimensional data query" is the backbone of a multi-dimensional data retrieval task. The starting process of "data query" is the process of executing data query subtasks; the starting process of "dimensional decomposition" is the process of executing data dimension decomposition. In the process of "multi-dimensional data query", step one, determine the main dimension and the dimension to be processed in the retrieval request; step two, determine the amount of data of the main dimension members; step three has two situations. The first situation: traverse the dimension members that do not need to be decomposed in the main dimension members, and generate query subtasks for the dimension members that do not need to be decomposed. In this case, it is necessary to execute the sub-process in the starting process of "data query". The sub-process steps include: executing the data query and calculation tasks of the current dimension members, and obtaining the data query results of the current dimension members. The second situation of step three: there are dimension members to be decomposed in the main dimension members, and the corresponding decomposition subtasks are generated. In this case, it is necessary to execute the sub-process in the starting process of "dimensional decomposition". The process, the sub-process steps include: obtaining the list of dimensions to be processed in step one, selecting a secondary dimension to decompose the dimension to be decomposed, and obtaining a new combined dimension; step four, using the new combined dimension obtained in the "dimensional decomposition" process as the new combined dimension in step four of the "multi-dimensional data query" process; step five, judging whether there is a dimension to be decomposed in the combined dimension, if so, continuing to execute the "dimensional decomposition" sub-process; if not, entering step six, generating query sub-tasks according to the dimension members that do not need to be decomposed in the combined dimension; finally, after waiting for all sub-tasks to be completed, the data query results in all "data query" sub-processes are combined, which is the final result of this multi-dimensional data query.

[0112] In the above embodiment, the sub-process of "obtaining a list of dimensions to be decomposed" and the sub-process of "grouping query calculation tasks" can be executed in parallel, that is, for dimension members that do not need to be decomposed, the corresponding thread is used to complete the query subtask corresponding to the dimension member. At the same time, for dimension members that need to be decomposed, the corresponding thread can be used to simultaneously execute the dimension decomposition task, and a corresponding query subtask is generated for one of the multiple combined dimensions obtained by decomposition, and the thread is used to execute the data query work of the subtask. The remaining multiple combined dimensions obtained by decomposition can be included in the task list of queued queries, waiting for idle threads to continue to execute the corresponding data query work or dimension decomposition work on them. In this way, the multi-thread or multi-node parallel execution of data query and dimension decomposition tasks, as well as the dimension decomposition of dimension members with a large amount of data, can control the amount of data processed by each thread within a controllable range, thereby balancing the workload of the client multi-threads, and can greatly accelerate the data retrieval efficiency.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a data retrieval device for implementing the data retrieval method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data retrieval device embodiments provided below can refer to the limitations on the data retrieval method above, and will not be repeated here.

[0115] In one embodiment, Figure 6 As shown, a data retrieval device is provided, including: an acquisition module 601, a determination module 602, a selection module 603 and a return module 604, wherein:

[0116] An acquisition module 601 is used to acquire a search request and determine a main dimension and a to-be-processed dimension among a plurality of search dimensions corresponding to the search request;

[0117] A determination module 602 is used to determine the amount of data corresponding to each main dimension member in the main dimension;

[0118] A selection module 603 is used to form the to-be-decomposed dimension from the main dimension members whose data volume meets the threshold condition, select one of the retrieval dimensions from the dimensions to be processed as the secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension;

[0119] Return module 604, for returning to the step of selecting one of the retrieval dimensions as the secondary dimension from the dimensions to be processed if the newly formed combined dimension has a dimension to be decomposed whose data volume meets the threshold condition, and continuing to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition;

[0120] The determination module 602 is further configured to generate and execute corresponding retrieval subtasks based on each combined dimension finally decomposed, and determine the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask.

[0121] In one embodiment, the determination module 602 is also used to, if there are main dimension members whose data volume does not meet the threshold condition, use the main dimension members whose data volume does not meet the threshold condition as the target dimension, and directly generate and execute the corresponding retrieval subtask based on the target dimension; if there are combined dimensions whose data volume does not meet the threshold condition in the combined dimensions, use the combined dimensions whose data volume does not meet the threshold condition as the target dimension, and directly generate and execute the corresponding retrieval subtask based on the target dimension; determine the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask, including: determining the retrieval result corresponding to the retrieval request based on the execution result of the retrieval subtask corresponding to each target dimension and the execution result of the retrieval subtask corresponding to each combined dimension obtained by the final decomposition.

[0122] In one embodiment, the determination module 602 is also used to merge the main dimension members corresponding to the total amount of data that does not meet the threshold condition if there are main dimension members whose data volume does not meet the threshold condition, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, to obtain a target dimension; if there are combined dimensions whose data volume does not meet the threshold condition in the combined dimension, then use the combined dimension whose data volume does not meet the threshold condition as the target dimension, including: if there are combined dimensions whose data volume does not meet the threshold condition in the combined dimension, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, then merge the combined dimensions corresponding to the total amount of data that does not meet the threshold condition to obtain the target dimension.

[0123] In one embodiment, the determination module 602 is further used to determine multiple retrieval dimensions corresponding to the retrieval request, wherein each retrieval dimension includes at least one dimension member; the retrieval dimension including the largest number of dimension members is used as the main dimension, and the other retrieval dimensions among the multiple retrieval dimensions except the main dimension are used as dimensions to be processed.

[0124] In one embodiment, the selection module 603 is also used to sort each retrieval dimension from large to small according to the number of dimension members it contains; select one of the retrieval dimensions from the dimensions to be processed as a sub-dimension, including: according to the sorting order of the dimensions to be processed, select an uncombined retrieval dimension from the dimensions to be processed as a sub-dimension.

[0125] In one embodiment, the number of main dimension members included in the main dimension is not less than the total number of retrieval threads.

[0126] The above-mentioned data retrieval device determines the main dimension and the dimension to be processed in the multiple retrieval dimensions corresponding to the retrieval request. The main dimension can determine the main scope of the data retrieved this time, and the dimension to be processed can assist the secondary dimension division of the main dimension in the retrieval process. Then determine the amount of data corresponding to each main dimension member in the main dimension, so that it can be determined whether the main dimension member needs to be decomposed according to the amount of data corresponding to each main dimension member. The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, select one of the retrieval dimensions from the dimension to be processed as the secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the dimension to be decomposed. In this way, the secondary dimension is used to decompose the dimension to be decomposed composed of the main dimension members whose data volume meets the threshold condition, that is, the secondary dimension is used to decompose the dimension to be decomposed with a large data volume, which can reduce the amount of data contained in the dimension to be decomposed, so that the thread executing the corresponding retrieval subtask will not have the risk of memory overflow, jamming or downtime. If there are still dimensions to be decomposed whose data volume meets the threshold condition in the newly formed combined dimension, a new secondary dimension is selected to decompose the newly formed combined dimension again, and the judgment iteration is continuously performed until the data volume corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition, so as to ensure that the data volume of each combined dimension obtained will not exceed the performance range of the thread executing the corresponding retrieval subtask. Finally, the corresponding retrieval subtask is generated and executed based on each combined dimension obtained by the final decomposition, and the retrieval result corresponding to the retrieval request is determined based on the execution result of each retrieval subtask. In this way, for the retrieval of multi-dimensional data, the main dimension data in the retrieval request is progressively decomposed by using one or more secondary dimensions, so that the data volume of all combined dimensions obtained in the end is within a completely controllable range, avoiding the problem of long single-node processing time or memory overflow caused by uneven data distribution when each thread executes the retrieval task, and also avoiding the problem of database read and write blocking or downtime caused by excessive I / O and computing processing pressure of the database during a single query, thereby accelerating the efficiency of data retrieval.

[0127] Each module in the above data retrieval device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0128] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store member data of each dimension. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data retrieval method is implemented.

[0129] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining a retrieval request, determining a main dimension and a to-be-processed dimension among a plurality of retrieval dimensions corresponding to the retrieval request; determining the amount of data corresponding to each main dimension member in the main dimension; constituting a to-be-decomposed dimension with main dimension members whose data amount meets a threshold condition, selecting one of the retrieval dimensions from the to-be-processed dimension as a secondary dimension, and constituting at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension; if there is a to-be-decomposed dimension whose data amount meets the threshold condition in the newly constituted combined dimension, returning to the step of selecting one of the retrieval dimensions from the to-be-processed dimension as the secondary dimension and continuing to execute until the amount of data corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition; generating and executing corresponding retrieval subtasks based on each combined dimension obtained by the final decomposition, and determining a retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask.

[0131] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if there are main dimension members whose data volume does not meet the threshold condition, the main dimension members whose data volume does not meet the threshold condition are used as target dimensions, and corresponding retrieval subtasks are generated and executed directly based on the target dimensions; if there are combined dimensions whose data volume does not meet the threshold condition, the combined dimensions whose data volume does not meet the threshold condition are used as target dimensions, and corresponding retrieval subtasks are generated and executed directly based on the target dimensions; based on the execution results of each retrieval subtask, the retrieval result corresponding to the retrieval request is determined, including: based on the execution results of the retrieval subtasks corresponding to each target dimension and the execution results of the retrieval subtasks corresponding to each combined dimension obtained by the final decomposition, the retrieval result corresponding to the retrieval request is determined.

[0132] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, the main dimension members whose data volume does not meet the threshold condition are used as the target dimension, including: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, the main dimension members corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension; if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, the combined dimensions whose data volume does not meet the threshold condition are used as the target dimension, including: if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, the combined dimensions corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension.

[0133] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining a main dimension and dimensions to be processed among multiple retrieval dimensions corresponding to the retrieval request, including: determining multiple retrieval dimensions corresponding to the retrieval request, wherein each retrieval dimension includes at least one dimension member; taking the retrieval dimension including the largest number of dimension members as the main dimension, and taking other retrieval dimensions among the multiple retrieval dimensions except the main dimension as dimensions to be processed.

[0134] In one embodiment, when the processor executes the computer program, the following steps are also implemented: sorting each retrieval dimension from large to small according to the number of dimension members it contains; selecting one of the retrieval dimensions from the dimensions to be processed as a sub-dimension, including: selecting an uncombined retrieval dimension from the dimensions to be processed as a sub-dimension according to the sorting order of the dimensions to be processed.

[0135] In one embodiment, when the processor executes the computer program, the following steps are further implemented: the number of main dimension members included in the main dimension is not less than the total number of search threads.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a retrieval request, determining a main dimension and a dimension to be processed among multiple retrieval dimensions corresponding to the retrieval request; determining the amount of data corresponding to each main dimension member in the main dimension; constituting a dimension to be decomposed with the main dimension members whose data amount meets a threshold condition, selecting one of the retrieval dimensions from the dimension to be processed as a secondary dimension, and constituting at least one new combined dimension based on each secondary dimension member of the secondary dimension and the dimension to be decomposed; if there is a dimension to be decomposed whose data amount meets the threshold condition in the newly constituted combined dimension, returning to the step of selecting one of the retrieval dimensions from the dimension to be processed as the secondary dimension and continuing to execute until the amount of data corresponding to each combined dimension obtained by the final decomposition does not meet the threshold condition; generating and executing corresponding retrieval subtasks based on each combined dimension obtained by the final decomposition, and determining the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask.

[0137] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if there are main dimension members whose data volume does not meet the threshold condition, the main dimension members whose data volume does not meet the threshold condition are used as target dimensions, and corresponding retrieval subtasks are generated and executed directly based on the target dimensions; if there are combined dimensions whose data volume does not meet the threshold condition, the combined dimensions whose data volume does not meet the threshold condition are used as target dimensions, and corresponding retrieval subtasks are generated and executed directly based on the target dimensions; based on the execution results of each retrieval subtask, the retrieval result corresponding to the retrieval request is determined, including: based on the execution results of the retrieval subtasks corresponding to each target dimension and the execution results of the retrieval subtasks corresponding to each combined dimension obtained by the final decomposition, the retrieval result corresponding to the retrieval request is determined.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, the main dimension members whose data volume does not meet the threshold condition are used as the target dimension, including: if there are main dimension members whose data volume does not meet the threshold condition among the main dimension members, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, the main dimension members corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension; if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, the combined dimensions whose data volume does not meet the threshold condition are used as the target dimension, including: if there are combined dimensions whose data volume does not meet the threshold condition among the combined dimensions, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, the combined dimensions corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining a main dimension and dimensions to be processed among multiple retrieval dimensions corresponding to the retrieval request, including: determining multiple retrieval dimensions corresponding to the retrieval request, wherein each retrieval dimension includes at least one dimension member; taking the retrieval dimension including the largest number of dimension members as the main dimension, and taking other retrieval dimensions among the multiple retrieval dimensions except the main dimension as dimensions to be processed.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sorting each retrieval dimension from large to small according to the number of dimension members it contains; selecting one of the retrieval dimensions from the dimensions to be processed as a sub-dimension, including: selecting an uncombined retrieval dimension from the dimensions to be processed as a sub-dimension according to the sorting order of the dimensions to be processed.

[0141] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: the number of main dimension members included in the main dimension is not less than the total number of retrieval threads.

[0142] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data retrieval method, characterized in that: The method comprises: Get the retrieval request; Determine multiple search dimensions corresponding to the search request, wherein each search dimension includes at least one dimension member; use the search dimension including the largest number of dimension members as the main dimension, and use the other search dimensions among the multiple search dimensions except the main dimension as dimensions to be processed; Determine the amount of data corresponding to each main dimension member in the main dimension; The main dimension members whose data volume meets the threshold condition constitute the dimension to be decomposed, one of the retrieval dimensions is selected from the dimensions to be processed as the secondary dimension, and at least one new combined dimension is constituted based on each secondary dimension member of the secondary dimension and the dimension to be decomposed; If there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition; Based on each combined dimension obtained by the final decomposition, a corresponding search subtask is generated and executed, and based on the execution result of each search subtask, a search result corresponding to the search request is determined.

2. The method according to claim 1, characterized in that The method further comprises: If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, the main dimension member whose data volume does not meet the threshold condition is used as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension; If there is a combined dimension whose data volume does not meet the threshold condition among the combined dimensions, the combined dimension whose data volume does not meet the threshold condition is taken as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension; The determining of the search result corresponding to the search request based on the execution result of each search subtask includes: Based on the execution results of the retrieval subtasks corresponding to the target dimensions and the execution results of the retrieval subtasks corresponding to the combined dimensions obtained by the final decomposition, a retrieval result corresponding to the retrieval request is determined.

3. The method according to claim 2, characterized in that If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, taking the main dimension member whose data volume does not meet the threshold condition as the target dimension includes: If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, and the total amount of data corresponding to multiple main dimension members whose data volume does not meet the threshold condition does not meet the threshold condition, the main dimension members corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension; If there is a combined dimension whose data volume does not meet the threshold condition among the combined dimensions, taking the combined dimension whose data volume does not meet the threshold condition as the target dimension includes: If there are combined dimensions whose data volume does not meet the threshold condition, and the total amount of data corresponding to multiple combined dimensions whose data volume does not meet the threshold condition does not meet the threshold condition, then the combined dimensions corresponding to the total amount of data that does not meet the threshold condition are merged to obtain the target dimension.

4. The method according to claim 1, characterized in that The method further comprises: Sort each retrieval dimension from largest to smallest according to the number of dimension members it contains; The selecting one of the search dimensions from the dimensions to be processed as the secondary dimension includes: According to the sorting order of the dimensions to be processed, a search dimension that has not been combined is selected from the dimensions to be processed as a secondary dimension.

5. The method according to any one of claims 1 to 4, characterized in that The number of main dimension members included in the main dimension is not less than the total number of retrieval threads.

6. A data retrieval device, characterized in that: The device comprises: An acquisition module, used for acquiring a retrieval request; a determination module, configured to determine a plurality of search dimensions corresponding to the search request, wherein each search dimension includes at least one dimension member; taking the search dimension including the largest number of dimension members as a main dimension, and taking the other search dimensions among the plurality of search dimensions except the main dimension as dimensions to be processed; The determination module is further used to determine the amount of data corresponding to each main dimension member in the main dimension; A selection module, configured to form a to-be-decomposed dimension from the main dimension members whose data volume meets a threshold condition, select one of the retrieval dimensions from the to-be-processed dimensions as a secondary dimension, and form at least one new combined dimension based on each secondary dimension member of the secondary dimension and the to-be-decomposed dimension; A return module, configured to return to the step of selecting one of the retrieval dimensions from the dimensions to be processed as the secondary dimension if there is a dimension to be decomposed whose data volume meets the threshold condition in the newly constructed combined dimension, and continue to execute until the data volume corresponding to each combined dimension finally decomposed does not meet the threshold condition; The determination module is further used to generate and execute corresponding retrieval subtasks based on each combined dimension finally decomposed, and determine the retrieval result corresponding to the retrieval request based on the execution result of each retrieval subtask.

7. The device according to claim 6, characterized in that The determining module is further used for: If there is a main dimension member whose data volume does not meet the threshold condition among the main dimension members, the main dimension member whose data volume does not meet the threshold condition is used as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension; If there is a combined dimension whose data volume does not meet the threshold condition among the combined dimensions, the combined dimension whose data volume does not meet the threshold condition is taken as the target dimension, and a corresponding retrieval subtask is directly generated and executed based on the target dimension; The determining of the search result corresponding to the search request based on the execution result of each search subtask includes: Based on the execution results of the retrieval subtasks corresponding to the target dimensions and the execution results of the retrieval subtasks corresponding to the combined dimensions obtained by the final decomposition, a retrieval result corresponding to the retrieval request is determined.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images