Massive astronomical data query processing method

Abstract

The invention relates to the technical field of database optimization, provides a massive astronomical data query processing method, and aims to solve the efficiency bottleneck of astronomical data in storage, management and query processes. By constructing a distributed spatial database, efficient storage and management of large-scale astronomical data are realized; an index structure of astronomical data is optimized by utilizing an LB-ANN index, so that query load balance and data retrieval efficiency are improved; besides, efficient fusion of structured query and unstructured query is realized through SQL-ANN mixed query in combination with a JIT compiling technology, the query execution sequence is optimized, and the query efficiency is remarkably improved; the overall method breaks through the bottlenecks of traditional astronomical data query in calculation complexity, storage efficiency and index optimization, and an accurate and efficient solution is provided for efficient intelligent processing of large-scale astronomical data.

Description

Technical Field

[0001] The invention relates to the technical field of database optimization, and in particular to a method for querying and processing massive astronomical data. Background Art

[0002] With the rapid development of modern astronomy, the improvement of observation technology and the diversification of data collection methods have brought astronomy into the era of big data. At present, major observatories, space telescopes, radio telescopes and other scientific equipment around the world generate massive amounts of astronomical data every day, including spectral data, image data, electromagnetic spectrum data, star catalog data, time series data, etc. However, traditional astronomical data query and processing methods have many shortcomings in practical applications, which are mainly reflected in the following aspects: First, when processing large-scale astronomical data, traditional relational databases are limited by storage structure and index mechanism, and the query speed is restricted by the growth of data scale; at the same time, the multimodal characteristics of astronomical data make it difficult to unify the storage model, and the complexity of data management increases; secondly, astronomical data usually has high-dimensional characteristics, for example, spectral data and time series data often contain hundreds to thousands of dimensions; traditional index structures such as B-Tree, R-Tree, KD-Tree face the "dimensionality disaster" problem when processing high-dimensional data, resulting in a significant decrease in indexing efficiency; finally, traditional astronomical database systems usually adopt a centralized architecture with low computing resource utilization, which is difficult to meet the high-concurrency query requirements of large-scale data. Summary of the invention

[0003] The present invention proposes an efficient query processing method for massive astronomical data. Aiming at the multi-modal, multi-time series, large-scale and highly dynamic characteristics of astronomical data, the present invention uses time series database, column storage database and distributed database to build an efficient distributed spatial database, and optimizes the index structure and query load balancing through LB-ANN index (load balancing-approximate nearest neighbor search index); in the index construction process, an index subset partitioning strategy based on K-Means clustering is proposed, and an index connectivity loss function is introduced, and vector allocation is dynamically adjusted through gradient optimization relaxation parameters to balance query efficiency and recall rate; in addition, the present invention constructs SQL-ANN hybrid query to achieve efficient fusion of structured query and unstructured query, optimizes query execution through JIT compilation, avoids redundant calculation, and improves distributed query performance; the overall method breaks through the bottlenecks of traditional astronomical data query in computational complexity, storage efficiency and index optimization, and provides an efficient and accurate solution for intelligent processing of large-scale astronomical data.

[0004] The present invention provides a method for querying and processing massive astronomical data, which specifically comprises the following steps:

[0005] Step S1: Collecting astronomical data sets, which include spectral data, image data, electromagnetic spectrum data, star catalog data, sound field data, time series data and virtual data;

[0006] Step S2: construct a distributed spatial database, analyze the data type, data scale, query mode, data update frequency, data access method and data sharing requirements of the astronomical data set, and store them in the distributed spatial database. The distributed spatial database includes a time series database, a column storage database and a distributed database;

[0007] Step S3: constructing an LB-ANN index through an overload-aware adaptive strategy and distributed ANNS index construction technology, and optimizing the index structure, data allocation, and query load balancing of the distributed spatial database through the LB-ANN index;

[0008] Step S4: Through SQL-ANN hybrid query, the query execution of the distributed spatial database is optimized, redundant calculations are reduced, query efficiency is improved, and data retrieval is accelerated.

[0009] Furthermore, step S3, the process of constructing the LB-ANN index specifically includes the following contents:

[0010] Step S31: obtaining a vector data set of a distributed spatial database, setting a maximum storage capacity and a maximum overlap factor of a subset in the vector data set, and calculating a minimum number of index partitions;

[0011] Step S32: generating an initial cluster center set according to the minimum number of index partitions, and constructing an index subset according to the initial cluster center set;

[0012] Step S33: optimizing the basic structure of the index subset to obtain the overload-aware index subset, allocating the vector data set to the overload-aware index subset, and performing quantization encoding and storing in the distributed spatial database;

[0013] Step S34: After the vector data set is allocated, ANNS subgraphs are constructed in distributed parallel, and a global index is formed by subgraph aggregation to obtain an LB-ANN index to support efficient retrieval.

[0014] Further, step S32 specifically includes the following contents:

[0015] Step S321: randomly extracting a sampling subset from the vector data set;

[0016] Step S322: using the K-Means clustering algorithm based on the sampling subset to calculate the initial cluster centers corresponding to the minimum number of index partitions, and obtaining an initial cluster center set;

[0017] Step S323: constructing the basic structure of the index subset according to the initial cluster center set, that is, determining the division method and initial center point of the index subset.

[0018] Furthermore, in step S33, an overload-aware adaptive strategy is used to optimize the basic structure of the index subset to construct an overload-aware index subset.

[0019] Furthermore, the process of allocating the vector dataset to the overload-aware index subset specifically includes the following:

[0020] Step Q1: Calculate the Euclidean distance from each vector in the vector data set to the initial cluster center set, and preliminarily determine the index subset to which the vector belongs as the initial candidate set;

[0021] Step Q2: Based on the initial candidate set, define relaxation parameters, introduce the index connectivity loss function, dynamically update the relaxation parameters through the adaptive optimization method, generate gradient optimized relaxation parameters, dynamically control the allocation of vectors to index subsets through gradient optimized relaxation parameters, and preliminarily evaluate the load status of index subsets; the process of dynamically controlling vector allocation through gradient optimized relaxation parameters optimizes the redundant partitioning degree of vectors in index subsets, ensures the connectivity of the index structure, and balances query performance and storage redundancy;

[0022] If the relaxation parameter is small, the vector data set strictly belongs to the nearest index subset, which reduces redundancy and improves index query efficiency;

[0023] The large vector data set with relaxed parameters can be attributed to multiple index subsets, which enhances the connectivity between subsets and improves the recall rate. It is suitable for situations with complex data distribution.

[0024] Step Q3: Set the maximum load limit, dynamically monitor the load status of the index subset, reach the maximum load limit, optimize the vector attribution through angle constraints, and allocate the vector data set to the overload-aware index subset to keep the index structure balanced and improve query efficiency.

[0025] Furthermore, step S4 specifically includes the following contents:

[0026] Step S41: obtaining an astronomical query statement, parsing and semantically analyzing the astronomical query statement, and generating a structured query semantic result;

[0027] Query parsing includes identifying the structured and unstructured data involved in astronomical query statements, and semantic analysis identifies the types of astronomical query statements, including Top-K nearest neighbor query, distance range query, distance Join query, and KNN-Join query;

[0028] Step S42: constructing a SQL-ANN balance formula through complexity analysis, query optimization cost minimization and data distribution modeling, and combining the structured query semantic results to perform SQL-ANN hybrid query and generate the optimal execution order; ensuring that SQL filtering and LB-ANN indexing are in the optimal execution position;

[0029] Step S43: according to the optimal execution order, use the LB-ANN index to limit the search range, and then perform JOIN to improve the calculation efficiency, avoid full table JOIN, and reduce the calculation complexity;

[0030] Step S44: Determine whether it is necessary to JIT compile the astronomical query statement based on the structured query semantic results, SQL-ANN balance formula and optimal execution order in steps S41 to S43, so as to avoid the additional overhead of function calls, improve execution efficiency, and optimize the query execution of the distributed spatial database.

[0031] By adopting the above scheme, the beneficial effects achieved by the present invention are as follows:

[0032] The present invention provides an efficient query processing method for massive astronomical data. By constructing an efficient distributed spatial database combining a time series database, a column storage database and a distributed database, efficient storage and management of large-scale astronomical data is achieved. The architecture can effectively cope with the multi-modal, multi-time series, large-scale and highly dynamic characteristics of astronomical data, significantly improve data access efficiency and query response speed, and provide a solid technical foundation for the intelligent processing of massive astronomical data.

[0033] The present invention introduces the load balancing-approximate nearest neighbor search (LB-ANN indexing) technology, innovatively optimizes the index structure of astronomical data, and significantly improves query load balancing and data retrieval efficiency; through the index subset partitioning strategy of K-Means clustering and the gradient optimization relaxation parameter based on the index connectivity loss function, the distribution mode of vector data is dynamically adjusted, thereby solving the problems of low query efficiency and insufficient recall rate in traditional methods; the optimized LB-ANN indexing method greatly improves the index optimization capability of distributed spatial databases, ensures efficient and accurate query processing, effectively avoids redundant calculations, and significantly improves the response speed of data retrieval;

[0034] In addition, the present invention realizes the efficient integration of structured query and unstructured query through SQL-ANN hybrid query; combined with JIT compilation technology, the query execution order is optimized, unnecessary full table scans and redundant calculations are avoided, and the query efficiency is significantly improved; this method not only effectively solves the computational complexity problem in traditional astronomical data queries, but also enhances the flexibility and intelligence of the system through reasonable query optimization; ultimately, the query processing capability of the distributed spatial database in a large-scale astronomical data environment is greatly improved, ensuring efficient and accurate data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a method for querying and processing massive astronomical data proposed by the present invention;

[0036] Figure 2 This is a schematic flow chart of step S4 proposed in Embodiment 7. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Embodiment 1, according to Figure 1 The present invention provides a method for querying and processing massive astronomical data, which specifically comprises the following steps:

[0039] Step S1: Collecting astronomical data sets, which include spectral data, image data, electromagnetic spectrum data, star catalog data, sound field data, time series data and virtual data;

[0040] Step S2: construct a distributed spatial database, analyze the data type, data scale, query mode, data update frequency, data access method and data sharing requirements of the astronomical data set, and store them in the distributed spatial database. The distributed spatial database includes a time series database, a column storage database and a distributed database;

[0041] Step S3: constructing an LB-ANN index through an overload-aware adaptive strategy and distributed ANNS index construction technology, and optimizing the index structure, data allocation, and query load balancing of the distributed spatial database through the LB-ANN index;

[0042] Step S4: Introduce SQL filtering, and optimize the query execution of the distributed spatial database through SQL-ANN hybrid query according to the LB-ANN index, reduce redundant calculations, improve query efficiency, and speed up data retrieval.

[0043] Embodiment 2: This embodiment is based on embodiment 1. In this embodiment, step S3, the process of constructing the LB-ANN index, specifically includes the following contents:

[0044] Step S31: obtaining a vector data set of a distributed spatial database, setting a maximum storage capacity and a maximum overlap factor of a subset in the vector data set, and calculating a minimum number of index partitions;

[0045] Step S32: generating an initial cluster center set according to the minimum number of index partitions, and constructing an index subset according to the initial cluster center set;

[0046] Step S33: optimizing the basic structure of the index subset to obtain the overload-aware index subset, allocating the vector data set to the overload-aware index subset, and performing quantization encoding and storing in the distributed spatial database;

[0047] Step S34: After the vector data set is allocated, ANNS subgraphs are constructed in distributed parallel, and a global index is formed by subgraph aggregation to obtain an LB-ANN index to support efficient retrieval.

[0048] Embodiment 3: This embodiment is based on embodiment 2. In this embodiment, step S32 specifically includes the following contents:

[0049] Step S321: randomly extracting a sampling subset from the vector data set;

[0050] Step S322: using the K-Means clustering algorithm based on the sampling subset to calculate the initial cluster centers corresponding to the minimum number of index partitions, and obtaining an initial cluster center set;

[0051] Step S323: constructing the basic structure of the index subset according to the initial cluster center set, that is, determining the division method and initial center point of the index subset.

[0052] Embodiment 4: This embodiment is based on embodiment 3. In this embodiment, in step S33, an overload-aware adaptive strategy is adopted to optimize the basic structure of the index subset to construct an overload-aware index subset.

[0053] Embodiment 5: This embodiment is based on embodiment 4. In this embodiment, the process of allocating a vector data set to an overload-aware index subset specifically includes the following contents:

[0054] Step Q1: Calculate the Euclidean distance from each vector in the vector data set to the initial cluster center set, and preliminarily determine the index subset to which the vector belongs as the initial candidate set;

[0055] Step Q2: Based on the initial candidate set, define relaxation parameters, introduce the index connectivity loss function, dynamically update the relaxation parameters through the adaptive optimization method, generate gradient optimized relaxation parameters, dynamically control the allocation of vectors to index subsets through gradient optimized relaxation parameters, and preliminarily evaluate the load status of index subsets; the process of dynamically controlling vector allocation through gradient optimized relaxation parameters optimizes the redundant partitioning degree of vectors in index subsets, ensures the connectivity of the index structure, and balances query performance and storage redundancy;

[0056] If the relaxation parameter is small, the vector data set strictly belongs to the nearest index subset, which reduces redundancy and improves index query efficiency;

[0057] The large vector data set with relaxed parameters can be attributed to multiple index subsets, which enhances the connectivity between subsets and improves the recall rate. It is suitable for situations with complex data distribution.

[0058] ;

[0059] in, represents the index connectivity loss function, represents a vector index, represents the total number of vectors in the vector dataset, represents the index subset index, represents a vector, Representation vector the set of index subsets to which they are assigned, Represents an index subset The cluster center of Representation vector The number of index subsets to which it belongs, Represents the calculation vector Go to Index Center The weighted average distance of represents the balance coefficient; represents the global average size of the index subset, Represents the current vector The degree of redundancy partitioning;

[0060] ;

[0061] in, represents the iteration index, represents the relaxation parameter, represents the relaxation parameter updated after iteration; represents the relaxation parameter of the current iteration, represents the learning rate, Represents the index connectivity loss function Partial derivatives with respect to the relaxation parameters;

[0062] ;

[0063] in, Representation vector Belongs to the index subset ; represents the gradient optimization relaxation parameter, Represents the average distance of all currently assigned vectors; Indicates the load balancing factor;

[0064] Step Q3: Set the maximum load limit, dynamically monitor the load status of the index subset, reach the maximum load limit, optimize the vector attribution through angle constraints, and allocate the vector data set to the overload-aware index subset to keep the index structure balanced and improve query efficiency.

[0065] Embodiment 6: This embodiment is based on embodiment 4. In this embodiment, the process of allocating a vector data set to an overload-aware index subset specifically includes the following contents:

[0066] Step T1: Calculate the Euclidean distance from each vector in the vector data set to the initial cluster center set, and preliminarily determine the index subset to which the vector belongs as the initial candidate set;

[0067] Step T2: Based on the initial candidate set, define a relaxation parameter, dynamically control the degree of redundant partitioning of vectors assigned to index subsets through the relaxation parameter, and preliminarily evaluate the load status of the index subset;

[0068] Step T3: Set the maximum load limit, dynamically monitor the load status of the index subset, reach the maximum load limit, optimize the vector attribution through angle constraints, ensure that the vector is reasonably allocated to the index subset, keep the index structure balanced, and improve query efficiency.

[0069] Embodiment seven, according to Figure 2 This embodiment is based on the fifth embodiment. In this embodiment, step S4 specifically includes the following contents:

[0070] Step S41: query analysis: obtaining an astronomical query statement, parsing and semantically analyzing the astronomical query statement, and generating a structured query semantic result;

[0071] Query parsing includes identifying the structured and unstructured data involved in astronomical query statements, and semantic analysis identifies the types of astronomical query statements, including Top-K nearest neighbor query, distance range query, distance Join query, and KNN-Join query;

[0072] Step S42: Generate execution order: construct SQL-ANN balance formula through complexity analysis, query optimization cost minimization and data distribution modeling, combine structured query semantic results, perform SQL-ANN hybrid query, and generate optimal execution order; ensure that SQL filtering and LB-ANN index are in the optimal execution position;

[0073] ;

[0074] in, Indicates the optimal SQL filtering ratio, Represents the total number of data points in a distributed spatial database, represents the number of target neighbors indexed by LB-ANN, Indicates the probability of still containing the target data after SQL filtering. Indicates the index scan calculation factor, Indicates the SQL filtering full table scan cost, represents the LB-ANN index calculation cost, represents the Top-K computation cost of the LB-ANN index, represents the LB-ANN index complexity coefficient, Represents a correction item, represents logarithmic growth, represents power growth;

[0075] Step S43: Execute JOIN: According to the optimal execution order, use the LB-ANN index to limit the search range, and then execute JOIN to improve the calculation efficiency, avoid full table JOIN, and reduce the calculation complexity;

[0076] Step S44: JIT compilation: According to the structured query semantic results, SQL-ANN balance formula and optimal execution order in steps S41-S43, it is determined whether the astronomical query statement needs to be JIT compiled to avoid the additional overhead of function calls, improve execution efficiency, and thus optimize the query execution of the distributed spatial database.

[0077] Embodiment 8: This embodiment is based on embodiment 5. In this embodiment, step S4 specifically includes the following contents:

[0078] Step R1: receiving an astronomical query statement, parsing and semantically analyzing the astronomical query statement, and generating a structured query semantic result; query parsing includes identifying structured data and unstructured data involved in the astronomical query statement, and semantic analysis identifies the type of astronomical query statement, including Top-K nearest neighbor query, distance range query, distance Join query, and KNN-Join query;

[0079] Step R2: Based on the structured query semantic results, map them into execution strategies suitable for the current query type and generate the optimal execution order;

[0080] Step R3: According to the optimal execution order, use the LB-ANN index to limit the search range, and then perform JOIN to improve the calculation efficiency, avoid full table JOIN, and reduce the calculation complexity;

[0081] Step R4: Determine whether it is necessary to perform JIT compilation on the astronomical query statement based on the structured query semantic results and the optimal execution order, so as to avoid the additional overhead of function calls, improve execution efficiency, and thus optimize the query execution of the distributed spatial database.

[0082] The present invention and its embodiments are described above, which is not restrictive. What is shown in the accompanying drawings is only one of the embodiments of the present invention, and the actual structure is not limited to this. In short, if ordinary technicians in this field are inspired by it and do not deviate from the purpose of the invention, they can creatively design structural methods and embodiments similar to the technical solution, which should all fall within the scope of protection of the present invention.

Claims

1. A method for querying and processing massive astronomical data, characterized in that: The method specifically comprises the following steps: Step S1: Collect astronomical data sets; Step S2: construct a distributed spatial database, analyze the characteristics of the astronomical data set, and store it in the distributed spatial database; Step S3: constructing an LB-ANN index through an overload-aware adaptive strategy and distributed ANNS index construction technology, and optimizing the index structure, data allocation, and query load balancing of the distributed spatial database through the LB-ANN index; Step S4: Optimize the query execution of the distributed spatial database through SQL-ANN hybrid query.

2. A method for querying and processing massive astronomical data according to claim 1, characterized in that: The process of building the LB-ANN index includes the following: Step S31: obtaining a vector data set of a distributed spatial database, and calculating the minimum number of index partitions; Step S32: Generate an initial cluster center set according to the minimum number of index partitions and construct an index subset; Step S33: optimizing the index subset, constructing the overload-aware index subset, and assigning the vector data set to the overload-aware index subset; Step S34: After the vector data set is allocated, ANNS subgraphs are constructed in distributed parallel, and then the subgraphs are aggregated to form a global index to construct a LB-ANN index.

3. A method for querying and processing massive astronomical data according to claim 2, characterized in that: In step S33, an overload-aware adaptive strategy is used to optimize the basic structure of the index subset to construct an overload-aware index subset.

4. A method for querying and processing massive astronomical data according to claim 2, characterized in that: The process of assigning a vector dataset to an overload-aware index subset includes the following: Step Q1: Calculate the Euclidean distance from each vector in the vector data set to the initial cluster center set, and preliminarily determine the index subset to which the vector belongs as the initial candidate set; Step Q2: Based on the initial candidate set, dynamically control the vector allocation and preliminarily evaluate the load status of the index subset; Step Q3: Dynamically monitor the load status of the index subset, optimize the vector attribution through angle constraints, and assign the vector data set to the overload-aware index subset.

5. A method for querying and processing massive astronomical data according to claim 4, characterized in that: The dynamic control of vector allocation specifically includes: defining relaxation parameters based on the initial candidate set, introducing the index connectivity loss function, dynamically updating the relaxation parameters, generating gradient optimized relaxation parameters, and dynamically controlling the allocation of vectors to index subsets through gradient optimized relaxation parameters.

6. A method for querying and processing massive astronomical data according to claim 1, characterized in that: The step S4 specifically includes the following contents: Step S41: obtaining an astronomical query statement, parsing and semantically analyzing the astronomical query statement, and generating a structured query semantic result; Step S42: constructing a SQL-ANN balance formula through complexity analysis, query optimization cost minimization and data distribution modeling, and performing SQL-ANN hybrid query in combination with structured query semantic results to generate an optimal execution order; Step S43: Execute JOIN according to the optimal execution order; Step S44: According to steps S41 to S43, it is determined whether it is necessary to perform JIT compilation on the astronomical query statement to optimize the query execution of the distributed spatial database.

Images