Retrieval method for large-scale data
The method addresses the challenge of fast and accurate retrieval of large-scale data by dividing data into databases, using a distributed architecture, and applying dynamic programming to enhance query efficiency and accuracy.
Patent Information
- Application Number
- CN202411176183.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-17
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods fail to provide fast and accurate associative queries for large-scale data due to the vast amount of data, dynamic changes, high data dimensionality, and the inability to distinguish valid query data, especially in distributed databases and tables.
A retrieval method that divides large-scale data into multiple databases and tables, establishes a distributed architecture, uses dynamic programming algorithms to process and combine data, and employs distributed query algorithms to ensure fast and accurate retrieval.
Enables efficient and precise retrieval of large-scale data by optimizing the query process through distributed query and dynamic programming, ensuring fast response times and accurate data association.
Smart Images
Figure CN119149536A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of big data retrieval, and in particular relates to a retrieval method for large-scale data. Background Art
[0002] Large-scale data refers to data volumes of tens of millions or more, which are usually stored in multiple tables and databases. In such a large amount of data, if you want to query all other fields associated with a certain field and exclude the wrong fields, it is impossible to complete the query task using only a single common data query algorithm. Common data query algorithms include: sequential search, binary search, interpolation search, Fibonacci search, block search, hash search, and tree table search.
[0003] The binary search has a time complexity of O(log2 n). Due to the data structure, the query speed is fast but the data editing speed is extremely slow, which is not scalable and is usually used for data ranging from thousands to millions. Hash search uses a large number of hash tables to achieve fast search functions, and the format of the hash table is key:value, which also means that one key only corresponds to one piece of data, and the time complexity is O(1). In theory, it can handle data of any magnitude, but in fact, due to cost considerations, it is impossible to purchase such a large disk space just to store hash tables. B-tree, a data structure suitable for large-scale queries, tries to access fewer layers of the disk each time it searches. It is suitable for large-scale data sets, especially data on storage devices. It can handle data records ranging from millions to billions, but there is a problem that editing data may reconstruct the data. Full table scan, due to data structure problems, is only suitable for medium and small data queries. In order to achieve fast associated queries on large amounts of data, fast response, and improve user service experience, how to shorten the response time for associated queries on massive data has become a widely concerned issue in the academic community in recent years.
[0004] At present, the main difficulties faced by large-scale data in fast and accurate associative query are as follows: the amount of data is huge, and it is stored in separate databases and tables. Faced with the huge amount of data, numerous databases and data tables, it will inevitably take a lot of time to perform associative query, and it cannot meet the requirements of fast response; the data has strong dynamic changes, and the data in the database will be updated and added according to the user's behavior. If the database cannot be updated quickly, there will be omissions and missing when querying related data; the data dimension is too high, also known as too many data types. How to calculate the correlation between the queried data and the metadata, and how to set the threshold to determine whether the two are related is also a problem; the traditional query algorithm queries based on keywords and only supports one-layer query. It is impossible to perform associative query based on the queried information, and it is also impossible to distinguish the validity of the query data. It is even more impossible to achieve accurate association.
[0005] For fast and accurate retrieval of large-scale data, the existing technology proposes to combine B-trees, hash indexes, etc., and improve the retrieval efficiency and the accuracy of retrieved data by optimizing the index structure. The existing technology indicates that data compression can significantly improve the response time of the database system. By using lightweight compression technology, the purpose of improving retrieval efficiency can be achieved without significantly increasing the running time of CPU-intensive queries; the existing technology proposes that the use of distributed architecture can make full use of hardware resources, distribute large-scale data on multiple computers, and use distributed shared memory architecture to make it a logical whole, so as to speed up data processing by parallel processing of multiple computers, improve data retrieval efficiency, and improve retrieval accuracy; the existing technology proposes that the purpose of improving query accuracy and efficiency can be achieved by optimizing the query process, combining random search, single start, multi start, simulated annealing and other technologies.
[0006] For association retrieval of large-scale data, the existing technology proposes to use a parallel computing framework to discover and mine spatiotemporal association relationships to improve the integrity of data associations; the existing technology proposes to use hash-based approximate similarity search to process high-dimensional data (multi-feature data) to ensure the integrity of data associations.
[0007] The existing technical solutions are that the data association algorithm is a data association algorithm, and the data fast query algorithm is a data fast query algorithm. The two are clearly distinguished. The data association algorithm does not mention the query efficiency, and the data fast retrieval algorithm does not mention how to perform the association query, which cannot meet the current market demand for large-scale, fast and accurate association query. Summary of the invention
[0008] In view of the above-mentioned deficiencies in the prior art, the present invention provides a retrieval method for large-scale data, which solves the problem that it is impossible to perform fast and accurate association query on large-scale data.
[0009] In order to achieve the above objectives, the technical solution adopted by the present invention is: a retrieval method for large-scale data, comprising the following steps:
[0010] S1. Based on different information sources of large-scale data, perform database and table operations on large-scale data, divide large-scale data into multiple databases and data tables for storage, build indexes in databases and data tables, and establish a distributed architecture for databases and data tables;
[0011] S2, obtaining the original features input by the user, using the distributed architecture, retrieving the first associated data associated with the original features in each database, screening the first associated data to obtain the original feature set; processing the original feature set using a dynamic programming algorithm, and using the first associated data in the processed original feature set as a query condition to query each database in a distributed manner to obtain the second associated data, and obtaining the associated data set from the second associated data;
[0012] S3. Determine whether to continue searching. If yes, use the dynamic programming algorithm to process the associated data set, and use the data in the processed associated data set as the query condition to query each database in a distributed manner to obtain the final original data. Use the dynamic programming algorithm to process the final original data to obtain the final data, and add the final data to the associated data set. If no, output the associated data set to complete the retrieval of large-scale data.
[0013] The beneficial effects of the present invention are as follows: the present invention breaks large-scale data into multiple databases and data tables, and constructs a distributed architecture to ensure that each database and data table is a whole; in the data processing flow, the present invention combines a distributed query algorithm with a dynamic programming algorithm, uses the dynamic programming algorithm to achieve accurate retrieval of large-scale data, and the dynamic programming algorithm optimizes the processing of multiple sub-queries, and then uses the distributed query algorithm to achieve fast retrieval of large-scale data, thereby achieving fast and accurate retrieval of large-scale data.
[0014] Further, the S2 comprises the following steps:
[0015] S201, obtaining original features input by the user, using the distributed architecture, using indexes in parallel in various databases, retrieving first associated data associated with the original features, using data screening to process missing and erroneous first associated data, and saving the screened first associated data to obtain an original feature set;
[0016] S202. Split the first associated data in the original feature set by using a dynamic programming algorithm, and recombine the first associated data with strong association, use the processed first associated data as query conditions, perform distributed query from each database to obtain query data, process the query data by using weighted data and reduced weight data methods, remove unreasonable query data and save them as second associated data, and obtain an associated data set from the second associated data.
[0017] The beneficial effects of the above further scheme are as follows: the present invention utilizes distributed query based on a distributed architecture to improve query efficiency, and combines the dynamic programming algorithm to process the data obtained from the query, thereby improving the accuracy of the query; by combining distributed query with the dynamic programming algorithm, while utilizing distributed query to ensure fast and efficient feature query, the dynamic programming algorithm is also utilized to take into account the accuracy of the data.
[0018] Furthermore, the S3 comprises the following steps:
[0019] S301, determine whether the user controls to continue the search, if so, proceed to step S302, otherwise, output the associated data set to complete the search of large-scale data;
[0020] S302, using the dynamic programming algorithm to process the associated data set, splitting the data in the associated data set, and recombining the data with strong correlation, using the processed data in the associated data set as a query condition, performing a distributed query from each database to obtain the final original data;
[0021] S303, using a dynamic programming algorithm to process the final original data, using weighted data and reduced weighted data methods to process the final original data, and eliminating unreasonable final original data to obtain final data, and adding the final data to the associated data set retrieved last time.
[0022] The beneficial effect of the above further scheme is: the present invention adopts multiple rounds of distributed query and dynamic programming algorithm processing, divides the input feature-related data set with many data into small data sets, and uses them as retrieval conditions, uses distributed query to query each database, and performs weight calculation again to screen out accurate input feature-related data sets, so that the data retrieval process is faster and the retrieved data obtained is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 The figure is a flow chart of the method of the present invention.
[0024] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION
[0025] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.
[0026] Example
[0027] like Figure 1 As shown, the present invention provides a retrieval method for large-scale data, and its implementation method is as follows:
[0028] S1. Based on the different information sources of large-scale data, perform database and table operations on large-scale data, divide large-scale data into multiple databases and data tables for storage, build indexes in the databases and data tables, and establish a distributed architecture for the databases and data tables.
[0029] In this embodiment, according to the different information sources of large-scale data, the large-scale data is divided into databases and tables. After the large-scale data is broken down into multiple databases and data tables for storage, a distributed architecture is established to ensure that each database and data table remains a whole.
[0030] S2, obtaining the original features input by the user, using the distributed architecture, retrieving the first associated data associated with the original features in each database, screening the first associated data to obtain the original feature set; processing the original feature set using a dynamic programming algorithm, and using the processed first associated data in the original feature set as a query condition to query each database in a distributed manner to obtain the second associated data, and obtaining the associated data set from the second associated data, wherein the implementation method is as follows:
[0031] S201, obtaining original features input by the user, using the distributed architecture, using indexes in parallel in various databases, retrieving first associated data associated with the original features, using data screening to process missing and erroneous first associated data, and saving the screened first associated data to obtain an original feature set;
[0032] S202. Split the first associated data in the original feature set by using a dynamic programming algorithm, and recombine the first associated data with strong association, use the processed first associated data as query conditions, perform distributed query from each database to obtain query data, process the query data by using weighted data and reduced weight data methods, remove unreasonable query data and save them as second associated data, and obtain an associated data set from the second associated data.
[0033] In this embodiment, first, through distributed query, other feature data associated with the original feature data input by the user are found from various databases, and multiple data with the same feature but different values are judged to obtain complete and reliable feature data that are highly correlated with the original feature data input by the user. The original feature data and the highly correlated feature data are integrated to obtain an original feature set.
[0034] Specifically, it can be understood as follows: when a user's name is known, relevant information is retrieved from various databases through distributed queries. When there are data with the same characteristics but different data, weights will be added first based on the degree of association with the user. Data with a high degree of association and uniqueness (such as ID card number) will have a high weight, while data with a relatively low degree of association and non-uniqueness (such as residential address) will have a low weight. Factors such as the number of occurrences will be added for weighting, and finally the user's accurate information can be obtained.
[0035] In this embodiment, the original feature set is obtained by segmenting the data of the original feature set through a dynamic programming algorithm, and the data with strong correlation is recombined and used as a search condition to search each database again to obtain more complete and accurate user information and obtain a related data set.
[0036] In this embodiment, distributed query is used to perform query tasks on feature data on multiple tables and multiple libraries in the corresponding database distributed server, thereby improving query efficiency. A distributed server is a server cluster that does not share data, performs its own tasks, and finally calls each other to complete the task.
[0037] In this embodiment, a dynamic programming algorithm is used to divide a large problem into small problems to be solved, so as to obtain the optimal solution. The dynamic programming algorithm is essentially an exhaustive method, but it can cut off overlapping small problems to reduce the overall complexity, so the algorithm is highly efficient. In the dynamic programming algorithm, when splitting sub-problems, an array is created to store the sub-problems and problem results. When the same sub-problem appears, the query result is directly obtained from the array to achieve the purpose of improving efficiency.
[0038] S3, determine whether to continue the search, if yes, use the dynamic programming algorithm to process the associated data set, and use the data in the processed associated data set as the query condition to query each database in a distributed manner to obtain the final original data, and use the dynamic programming algorithm to process the final original data to obtain the final data, and add the final data to the associated data set; if no, output the associated data set to complete the retrieval of large-scale data, and the implementation method is as follows:
[0039] S301, determine whether the user controls to continue the search, if so, proceed to step S302, otherwise, output the associated data set to complete the search of large-scale data;
[0040] S302, using the dynamic programming algorithm to process the associated data set, splitting the data in the associated data set, and recombining the data with strong correlation, using the processed data in the associated data set as a query condition, performing a distributed query from each database to obtain the final original data;
[0041] S303, using a dynamic programming algorithm to process the final original data, using weighted data and reduced weighted data methods to process the final original data, and eliminating unreasonable final original data to obtain final data, and adding the final data to the associated data set retrieved last time.
[0042] In this embodiment, the user controls whether to continue the search. If the user chooses to continue, the data set finally obtained in step S2 is split and combined again through the dynamic programming algorithm, and a distributed query task is issued, and data processing is performed to find the most accurate feature data and add it to the feature set screened last time. If the user does not need to continue screening, the user can choose to output. After two rounds of retrieval, all the information of the user stored in the database can be determined.
[0043] like Figure 2 As shown, in this embodiment, the original input features are first used to directly query through the distributed architecture, and accurate associated data sets are extracted through weight calculation and other methods. Then, the associated data set with many data is divided into small data sets with only two or three items using the dynamic programming algorithm, which are used as retrieval conditions, and the distributed architecture is used to query each database, and the weight calculation is performed to filter out the accurate associated data set; the user controls whether to continue the search, and if so, the dynamic programming algorithm is used again to divide the associated data set with many data into small data sets with only two or three items, which are used as retrieval conditions, and the distributed architecture is used to query each database, and the weight calculation is performed to filter out the accurate associated data set; if not, the associated data set is output as the final output.
Claims
1. A retrieval method for large-scale data, It is characterized in that The following steps are involved: S1. Based on different information sources of large-scale data, perform database and table operations on large-scale data, divide large-scale data into multiple databases and data tables for storage, build indexes in databases and data tables, and establish a distributed architecture for databases and data tables; S2, obtaining the original features input by the user, using the distributed architecture, retrieving the first associated data associated with the original features in each database, screening the first associated data to obtain the original feature set; processing the original feature set using a dynamic programming algorithm, and using the first associated data in the processed original feature set as a query condition to query each database in a distributed manner to obtain the second associated data, and obtaining the associated data set from the second associated data; S3. Determine whether to continue searching. If yes, use the dynamic programming algorithm to process the associated data set, and use the data in the processed associated data set as the query condition to query each database in a distributed manner to obtain the final original data. Use the dynamic programming algorithm to process the final original data to obtain the final data, and add the final data to the associated data set. If no, output the associated data set to complete the retrieval of large-scale data.
2. The method for searching large-scale data according to claim 1, It is characterized in that The S2 comprises the following steps: S201, obtaining original features input by the user, using the distributed architecture, using indexes in parallel in various databases, retrieving first associated data associated with the original features, using data screening to process missing and erroneous first associated data, and saving the screened first associated data to obtain an original feature set; S202. Split the first associated data in the original feature set by using a dynamic programming algorithm, and recombine the first associated data with strong association, use the processed first associated data as query conditions, perform distributed query from each database to obtain query data, process the query data by using weighted data and reduced weight data methods, remove unreasonable query data and save them as second associated data, and obtain an associated data set from the second associated data.
3. The method for searching large-scale data according to claim 1, It is characterized in that The S3 comprises the following steps: S301, determine whether the user controls to continue the search, if so, proceed to step S302, otherwise, output the associated data set to complete the search of large-scale data; S302, using the dynamic programming algorithm to process the associated data set, splitting the data in the associated data set, and recombining the data with strong correlation, using the processed data in the associated data set as a query condition, performing a distributed query from each database to obtain the final original data; S303, using a dynamic programming algorithm to process the final original data, using weighted data and reduced weighted data methods to process the final original data, and eliminating unreasonable final original data to obtain final data, and adding the final data to the associated data set retrieved last time.