A method for optimal asynchronous data search based on distributed processing

Abstract

The application relates to the technical field of information retrieval, in particular to an optimal asynchronous data search method based on distributed processing, which comprises the following operation steps: determining a retrieval formula, then randomly selecting a certain amount of data from a database to form a retrieval library; calculating the proportion of various data types in the retrieval library; determining various data types corresponding to the retrieval formula, and sorting the various data types in ascending order according to the proportion of the various data types in the retrieval library; importing the content corresponding to the data type with the smallest proportion in the retrieval formula into the retrieval library to retrieve a certain amount of data; comparing each piece of data with the content, if the data is the same as the content, the data is the required retrieval result, if not, deleting the retrieved data from the retrieval library, and the data type no longer participates in the sorting, and the above operation steps are circularly operated until the required result is retrieved or all data types are traversed; the application can quickly and accurately retrieve the required retrieval result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, in particular to an optimal asynchronous data search method based on distributed processing. BACKGROUND

[0002] In the Internet era, people's life every moment can not do without data, computer mobile phone is the most intimate partner, can find what they want without leaving home. But we are curious about how we can quickly find the information we care about through these network terminals. Obviously, the information on the network is not as easy to find as the things in our home. First, we don't know if the information we want exists. Second, if such information exists, where is the information we want, in a data body or in multiple data bodies. Finally, the uncertainty of the scale of the search data.

[0003] Now the network platform almost all provides the function of user search engine, the user can through "directory classification", "keyword" and "mixed search" engine way and platform interaction, the result of search can sometimes find the information that meets the requirements, sometimes find a lot of information that does not meet the requirements, even can not find the information that meets the requirements. The reason why the information that meets the requirements can not be found is that sometimes there is no information that the user wants in the network world, but sometimes there is information that the user needs in the network world, but because the engine method used in the platform database to find such information takes too long, exceeds the "limit" of the communication time limit in the network, and the network terminates the user's request. At present, the engine method used in the platform database is mostly "full traversal" and "random" method. These methods have relatively low maintenance cost in the early stage of the database, but the amount of key words for search in the database is unknown, which may lead to a large amount of data in each search, so it is urgent to solve. SUMMARY

[0004] In order to avoid and overcome the technical problems existing in the prior art, the present application provides an optimal asynchronous data search method based on distributed processing. The present application can quickly and accurately retrieve the required search results.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] An optimal asynchronous data search method based on distributed processing comprises the following operation steps:

[0007] S1, first determine the search formula to be searched, then randomly select a certain amount of data from the database to form a search library, and the data types in the search formula are all contained in the search library;

[0008] S2, calculate the proportion of various data types in the search library;

[0009] S3, determine the various data types corresponding to the search formula, and sort the various data types in the search library according to the proportion in ascending order;

[0010] S4, import the content corresponding to the data type with the smallest proportion in the search formula into the search library for searching, and search for a predetermined number of data;

[0011] S5, compare each piece of data with the content in step S4 one by one, if one piece of data is the same as the content, then the data is the required search result, if there is no data that is the same as the content, then go to step S6;

[0012] S6, delete the searched data from the search library, and the data type no longer participates in the sorting;

[0013] S7, according to the operation steps of steps S2-S6, the operation is circular, until the required search result is searched or all data types are traversed.

[0014] As a further scheme of the application: the specific steps of step S1 are as follows:

[0015] S11, determine the content to be searched, which is the corresponding search formula;

[0016] S12, determine the various data types in the search formula;

[0017] S13, select a certain amount of data from the database according to the data types corresponding to the search formula to form a search library, and each piece of data in the search library contains each data type corresponding to the search formula.

[0018] As a further scheme of the application: the specific steps of step S2 are as follows:

[0019] S21, calculate the data amount of each data type in the search library;

[0020] S22, calculate the total data amount of the data in the search library, which is the original search library;

[0021] S23, import the results of steps S21 and S22 into the proportion calculation formula to calculate the proportion of each data type, and the proportion calculation formula is as follows:

[0022]

[0023] Wherein, n 0,i represents the data amount of the i-th data type in the original search library; N0 represents the total data amount of the data in the original search library; θ 0,i represents the proportion of the i-th data type in the original search library.

[0024] As a further scheme of the present application: the specific steps of step S3 are as follows:

[0025] S31, find the proportion corresponding to the data type in the original search library according to the data type in the search formula;

[0026] S32, arrange the various data types in the search formula in order from small to large according to the size of the proportion.

[0027] As a further scheme of the present application: the specific steps of step S4 are as follows:

[0028] S41, find the content corresponding to the data type with the smallest proportion in the search formula;

[0029] S42, import the content into the original search library for searching, and search out d 0,i pieces of data, where d 0,i represents the data amount corresponding to the ith data type searched in the first search.

[0030] As a further scheme of the present application: the specific steps of step S5 are as follows:

[0031] S51, compare the content of the searched d 0,i pieces of data with the content found in step S41 one by one;

[0032] S52, if the content of one of the searched d 0,i pieces of data is the same as the content found in step S41, then the searched data is the required search result; if none of the contents is the same, then the next operation is needed.

[0033] As a further scheme of the present application: the specific steps of step S6 are as follows:

[0034] S61, delete the d 0,i pieces of data searched in step S42 from the original search library, and the data type corresponding to the data is no longer involved in the subsequent proportion sorting;

[0035] S62, record the original search library after deleting the data as a first search library, and record the data amount in the first search library as N1.

[0036] As a further scheme of the present application: the specific steps of step S7 are as follows:

[0037] S71, calculate the data amount N1 in the first search library, and the calculation formula is as follows:

[0038] N1=N0-d 0,i

[0039] Where N1 is the amount of data in the database during a single search;

[0040] S72. Use a search database to replace the original database and perform the operations of steps S21-S71 until the required search results are found or all data types are traversed.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] 1. This invention can reduce the amount of relevant but useless information, maximize the effective search range and minimize the invalid search range, reduce the amount of filtering for users, and improve the efficiency of information retrieval.

[0043] 2. In theory, multiple scenarios may occur, which may seem complex and cumbersome. However, in actual operation, there is only one scenario, and each scenario has only one possible outcome. This is because the user has already provided some specific information during the operation. The method will perform targeted deletion, subsequent calculations, and sorting based on this specific information, and finally list the information for the user to search and filter. This method is actually quick and efficient, greatly improving efficiency. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the operation steps of the present invention.

[0045] Figure 2 This is a schematic diagram of the engine efficiency distribution in this invention.

[0046] Figure 3 This is a schematic diagram illustrating the changes in engine efficiency in this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Algorithm ideas such as Figure 1 As shown:

[0049] Where Δ represents the change, θ represents the proportion of a data type, and n represents the quantity of a certain data type in the database. Let τ denote the square root order, (logΔi) / Δi denote the logarithmic order, τ denote the matrix, the original retrieval library, and θ. i This represents the proportion of the i-th data type in the database, ο(k1, k2, ..., k n () indicates the search data type. This indicates the optimal asynchronous data engine efficiency.

[0050] The Δ-scale optimal asynchronous data engine algorithm records the data types and proportions θ of data in a distributed database, such as... Figure 2 As shown, during the search engine process, a series of ascending-order lists (θ1, θ2, ..., θ3) can be formed based on the data type list. n The search proceeds sequentially based on the data type corresponding to the ascending list (k1, k2, ..., k...). n Based on data "empirical values" observed, when θ i When ≤(logΔi) / Δi, the efficiency of this engine algorithm can reach the "square root". At this time, the efficiency of the engine algorithm can reach the ideal "logarithmic order". When the data scale is very large, θ i It is highly likely that the value is much smaller than logΔi) / Δi. In this case, the size of the search data decreases rapidly, and only when the target data is close to appearing will the data size decrease slowly. This algorithm can adapt to distributed processing under current cloud technology.

[0051] First time: Search for data type:

[0052] ο(k1, k2, ..., k n ); Find the matrix τ0: k1: θ1; k2: θ2; ...; k n θ n ;θ i (1≤i≤n) represents the proportion of the i-th data type in the matrix τ, i.e., the probability. θ1, θ2, ..., θ n There exists an ascending order in τ, with the minimum value denoted as θ. min The optimal asynchronous data engine concept is to choose θ. i The corresponding k i The engine output scale is Δ1, where Δ1 is Δθ1, Δθ2, ..., Δθ n The minimum value in is Δθ min .

[0053] Second time: Search for data type:

[0054] ο(k1, k2, ..., k n )-ο(k i ):ο(k1, k2, ..., k n-1 Find the matrix τ1 (its data size is Δ1): k1: θ1; k2: θ2; ...; k n-1 θ n-1 θ i(1≤i≤n-1) represents the proportion of the i-th data type in the matrix τ1, that is, the probability. θ1, θ2, …..., θ n-1 There is an "ascending order" in τ1, and the minimum value is denoted as θ min . The optimal asynchronous data engine idea is to select θ i . The corresponding k i , the engine output size is Δ2, Δ2 is the minimum value of Δθ1, Δθ2, …, Δθ n-1 , that is, Δθ min .

[0055] The j-th time: find the data type:

[0056] o(k1, k2, …, k n )-o(k1, k2, …, k j ): o(k1, k2, …, k n-j ); Find the matrix τ1 (whose data size is Δ1): k1: θ1; k2: θ2; …; k n-j : θ n-j . θ i (1≤i≤n-j) represents the proportion of the i-th data type in the matrix τ j , that is, the probability. θ1, θ2, …..., θ n-j There is an "ascending order" in τ j , and the minimum value is denoted as θ min . The optimal asynchronous data engine idea is to select θ i . The corresponding k i , the engine output size is Δ2, Δ2 is the minimum value of Δθ1, Δθ2, …, Δθ n-1 , that is, Δθ min . The efficiency of the optimal asynchronous data engine:

[0057] Test case:

[0058] The size of the data Δ (500000), the search object is based on the key o(a, b, c, d), a: 0.04; b: 0.002; c: 0.4; d: 0.075.

[0059] The first time: find the object according to the key:

[0060] 0.04, 0.002, 0.4, 0.075 in τ0 there is an "ascending order": 0.002, 0.04, 0.075, 0.4, θ1 takes 0.002, Δ1 = Δ*0.002 = 1000.

[0061] The second time: find the object according to the key:

[0062] o(a, b, c, d) - o(b): o(a, c, d), a: 0.2, c: 0.6, d: 0.3. There is an "ascending order" of 0.2, 0.3, 0.6 in τ1, so θ1 takes 0.20, Δ2 = Δ1 * 0.2 = 200.

[0063] Third time: find objects according to keywords:

[0064] o(a, b, c, d) - o(b, a): o(c, d), c: 0.8, d: 0.4. There is an "ascending order" of 0.4, 0.8 in τ2, so θ1 takes 0.4, Δ3 = Δ2 * 0.4 = 80.

[0065] Fourth time: find objects according to keywords:

[0066] o(a, b, c, d) - o(b, a, d): o(c).

[0067] Find the base τ3 (its data size is Δ3 = 80).

[0068] Optimal asynchronous data engine efficiency: 500000 + 1000 + 200 + 80.

[0069] Performance analysis:

[0070] The optimal asynchronous data engine algorithm can make the data volume decrease "faster", and in the ideal state, it can capture the "square root order" or "log order". The results are shown in Table 1 and Figure 3 .

[0071] Table 1 Optimal asynchronous data engine example analysis table

[0072]

[0073] Example:

[0074] A high school has fifty classes, forty science classes, and ten arts classes. A science class has a total of 40 students, and some of the students' data is shown in Table 2.

[0075] Table 2 Student basic information table

[0076]

[0077] The user wants to find Liu Yiyi's height and weight data, calculate the BMI index, and determine Liu Yiyi's physical health condition. The following steps are the method idea:

[0078] The network platform will call out the database of the class to form a search library, and the search library has a total of 5 data types, namely name, gender, student ID, height and weight.

[0079] For the name, there are two cases, one is the same name, and the other is not the same name, that is, each non-repeated name is a data sub-type, and the repeated name is only one data sub-type.

[0080] For the student ID, there is no repetition, so each student ID is a data sub-type.

[0081] For gender, gender only includes male and female, so gender male is a data sub-type, and gender female is a data sub-type, a total of two data sub-types.

[0082] For weight, weight is repeated, that is, each different weight is a data sub-type, and the same is only one data sub-type.

[0083] For height, height is repeated, that is, each different height is a data sub-type, and the same is only one data sub-type.

[0084] (1) Student ID keyword: Each student has a unique student ID, so the probability of finding the target by keyword is 1 / 40;

[0085] (2) Same name keyword: Through the table, it is found that there are 4 students with the same name, so the probability of finding the target by the same name keyword is 4 / 40;

[0086] (3) Three-character name keyword: Through the table, it is found that there are 5 students with three-character names, so the probability of finding the target by the three-character name keyword is 5 / 40;

[0087] (4) Two-character name keyword: Through the table, it is found that there are 35 students with two-character names, so the probability of finding the target by the two-character name keyword is 35 / 40;

[0088] (5) Male keyword: Through the table, it is found that there are 26 students who are male, so the probability of finding the target by the male keyword is 26 / 40;

[0089] (6) Female keyword: Through the table, it is found that there are 14 students who are female, so the probability of finding the target by the female keyword is 14 / 40.

[0090] The method arranges the keyword proportion "ascending order" list from small to large as follows: 1 / 40, 4 / 40, 5 / 40, 14 / 40, 26 / 40, 35 / 40.

[0091] The probability of the student number keyword is the smallest, that is, the search range is the smallest, the user randomly checks a student number and checks its related information (name, gender, height, weight), if it is 9021011962, the search is successful, if not, delete all its related information, and then search the next keyword.

[0092] After deleting the student number keyword, the keyword proportion is automatically recalculated, and the "ascending order" list is arranged from small to large as follows: 3 / 39 or 4 / 39, 4 / 39 or 5 / 39, 13 / 39 or 14 / 39, 25 / 39 or 26 / 39, 34 / 39 or 35 / 39.

[0093] The probability of the duplicate name keyword is the smallest, that is, the search range is the smallest, and after deletion, it becomes 3 / 39 (the number 3 is the student number keyword deletion affecting the duplicate name keyword, and the number 39 is the remaining student number after deletion) or 4 / 39 (the numbers 4 and 39 are only deleted the student number keyword and do not affect the duplicate name keyword). First look at 3 / 39, the network platform lists the data of three duplicate students, through the above table data, it is found that it is not the target, delete the information of three students and search the next keyword; then look at the case of 4 / 39, the network platform lists the data of four duplicate students, through the above table data, it is found that it is not the target, delete the information of four students and search the next keyword.

[0094] After deleting the student number keyword and the duplicate name keyword, the keyword proportion is automatically recalculated, and the "ascending order" list is arranged from small to large as follows: in the first case, the deleted duplicate name keyword information does not contain the deleted student number keyword information, which is 4 / 35 or 5 / 35, 13 / 35 or 14 / 35, 25 / 35 or 26 / 35, 34 / 35 or 35 / 35; in the second case, the deleted duplicate name keyword information contains the deleted student number keyword information, which is 4 / 36 or 5 / 36, 13 / 36 or 14 / 36, 25 / 36 or 26 / 36, 34 / 36 or 35 / 36; in both cases, the probability of the three-character duplicate name keyword is the smallest, that is, the search range is the smallest, the network platform lists the related information in each case, and finally finds the information of Liu Yiyi, the user finds the required information, and the method ends.

[0095] In the above process, under the theoretical consideration, there are many cases, which looks complex and cumbersome, but in actual operation, there is only one case, and the subsequent possibility of each case is only one, because the user has provided some specific information when operating, the method will delete and calculate and arrange according to the specific information, and finally list the information for the user to find and screen. This method is actually fast and efficient.

[0096] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for optimal asynchronous data search based on distributed processing, characterized in that, The method comprises the following steps: S1, first determine the search formula to be searched, then randomly select a certain amount of data from the database to form a search library, and the data types in the search formula are all contained in the search library; S2, calculate the proportion of various data types in the search library; S21, calculate the data amount corresponding to various data types in the search library; S22, calculate the total data amount of the data in the search library, and the search library at this time is the original search library; S23, import the results of steps S21 and S22 into the proportion calculation formula to calculate the proportion of various data types, and the proportion calculation formula is as follows: wherein, n 0,i represents the data amount of the data type corresponding to the data type in the original search library; i N 0 represents the total data amount of the data in the original search library; θ 0,i represents the proportion of the data type in the original search library; i represents the proportion of the data type in the original search library;​ S3, determine the various data types corresponding to the search formula, and sort them in ascending order according to the proportion of the various data types in the search library; S4, import the content corresponding to the data type with the smallest proportion in the search formula into the search library for searching, and search out a predetermined number of data; S5, compare each piece of data with the content in step S4 one by one, if one piece of data is the same as the content, then the data is the required search result, if there is no data the same as the content, then enter step S6; S6, delete the searched data from the search library, and the data type will not participate in the sorting any more; S7, perform the operation process of steps S2-S6 in a loop until the required search result is searched out or all data types are traversed. The specific steps of step S1 are as follows:

2. The method for optimal asynchronous data search based on distributed processing according to claim 1, wherein, S11, determine the content to be searched, which is the corresponding search formula; S12, determine various data types in the search formula; S13, select a certain amount of data from the database according to the data types corresponding to the search formula to form a search library, and each piece of data in the search library contains each data type corresponding to the search formula. The specific steps of step S3 are as follows:

3. The method of claim 2, wherein, S31, find the proportion of the data type corresponding to the data type in the search formula in the original search library; S32, arrange the various data types in the search formula in order from small to large according to the size of the proportion. The specific steps of step S4 are as follows:

4. The method for optimal asynchronous data search based on distributed processing according to claim 3, wherein, S41, find the content corresponding to the data type with the smallest proportion in the search formula; The specific steps of step S5 are as follows: S42, the content is imported into the original search database for searching, and the d 0,i data, wherein d 0,i indicates the data amount corresponding to the data type searched in the first search. i data type.

5. The method for optimal asynchronous data search based on distributed processing according to claim 4, wherein, The specific steps of step S6 are as follows: S51, compare the contents of the retrieved d 0,i the contents of the data with the contents found in step S41. S52、If the retrieved d 0,i If one of the contents of the data is the same as the content found in step S41, the retrieved data is the required search result; if none of the contents is the same, the next step is required.

6. The method for optimal asynchronous data search based on distributed processing according to claim 5, wherein, The specific steps of step S7 are as follows: S61. The data retrieved in step S42 d 0,i The data item is deleted from the original retrieval database, and the data type corresponding to this data will no longer participate in the subsequent sorting process. S62, the original search library after deleting data is recorded as a primary search library, and the data amount in the primary search library is recorded as N 1.

7. The method for optimal asynchronous data search based on distributed processing according to claim 6, wherein, S72, use the search library to replace the original database, perform the operation of steps S21-S71, and stop until the required search result is searched out or all data types are traversed. S71, calculate the amount of data in the first search library N 1, the calculation formula is as follows: wherein, N 1 is the amount of data in the primary search library; ​

Images