A distributed intelligent retrieval method and system for spatiotemporal data

Abstract

The present invention discloses a distributed intelligent retrieval method and system for spatiotemporal data, the method comprising: obtaining a spatiotemporal source data file, preprocessing and parsing the spatiotemporal source data file to obtain a spatiotemporal parsed data set; generating a load-balanced partitioning scheme for the spatiotemporal parsed data set based on reservoir sampling and R‑Star tree algorithm, and allocating each partition to an executor; obtaining the user's query content, locating relevant keywords and context information using a retrieval model, and generating a query instruction through a pre-trained large language model; executing the query instruction, and globally screening candidate partitions that may contain query results according to the partition metadata, and performing local retrieval in the candidate partitions, thereby obtaining retrieval results. The partition metadata is used to perform global screening first and then local retrieval, avoiding loading all data into memory, reducing the requirements for device hardware performance and improving retrieval efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent analysis of spatiotemporal data, and in particular relates to a distributed intelligent retrieval method and system for spatiotemporal data. Background Art

[0002] With the surge in GPS embedded devices and location-based services, more and more spatiotemporal data are generated in various fields, including urban planning, intelligent transportation, etc. Users have an increasing demand for efficient storage, retrieval and analysis of collected spatiotemporal data, and stand-alone spatiotemporal data retrieval can no longer meet user needs. Distributed spatiotemporal data retrieval technology has emerged in the context of the big data era, aiming to efficiently process massive spatiotemporal information and extract value from it. By accurately capturing the timestamp and geographic location information of the data, the distributed spatiotemporal data retrieval system can support refined management and decision-making at multiple levels.

[0003] At the same time, with the development of artificial intelligence technology, especially the progress in the field of natural language processing (NLP), large language models are used to bridge human natural language queries with complex SQL or spatiotemporal query languages. Users can use natural language to ask queries involving spatiotemporal dimensions, and the system can automatically convert them into executable SQL or specific query language instructions, which greatly reduces the query threshold and improves data accessibility and analysis efficiency.

[0004] In the process of implementing the present invention, the inventors found that the existing distributed spatiotemporal data retrieval technology has at least the following problems:

[0005] First, the loads between the executors are not balanced. Spatiotemporal data usually has significant spatial and temporal uneven distribution characteristics, resulting in the amount of data in some areas or time periods far exceeding other parts. This brings challenges to the data partitioning and load balancing of the distributed system, easily forming data hotspots and reducing the overall processing efficiency. Secondly, there is a lack of extensive support for user-defined operators, especially for operators executed on GPUs. Calculations on GPUs will greatly accelerate the spatiotemporal computing process. Finally, the existing technology does not support interactive retrieval functions in natural language, which increases the requirements for users' retrieval expertise. At the same time, although large language models have made significant progress in natural language-SQL conversion, there are still problems of understanding bias and inaccurate conversion when understanding and parsing the spatiotemporal query SQL functions that are proprietary to spatiotemporal retrieval systems, affecting user experience and the reliability of query results. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the embodiments of the present application is to provide a distributed intelligent retrieval method and system for spatiotemporal data, which can efficiently achieve spatiotemporal data load balancing, user-defined spatiotemporal calculation operators, spatiotemporal data retrieval and natural language retrieval.

[0007] According to a first aspect of an embodiment of the present application, a distributed intelligent retrieval method for spatiotemporal data is provided, comprising:

[0008] Acquire a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file to obtain a spatiotemporal parsed data set;

[0009] Based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme for spatiotemporal resolution datasets is generated, and each partition is assigned to the executor;

[0010] Obtain the user's query content, use the retrieval model to locate relevant keywords and context information, and generate query instructions through the pre-trained large language model;

[0011] The query instruction is executed, and according to the partition metadata, candidate partitions that may contain the query results are globally screened, and local searches are performed in the candidate partitions to obtain search results.

[0012] Further, a spatiotemporal source data file is obtained, and the spatiotemporal source data file is preprocessed and parsed to obtain a spatiotemporal parsed data set, including:

[0013] Obtaining a spatiotemporal source data file provided by a user, and adaptively selecting a loading parser according to the suffix of the spatiotemporal source data file and the loading parameters specified by the user;

[0014] The spatiotemporal source data file is parsed by using the loading parser, wherein, during the loading process of the spatiotemporal source data file, the spatiotemporal source data file is checked for data format consistency and integrity, and abnormal data therein is repaired;

[0015] The parsed spatiotemporal objects are standardized to obtain a spatiotemporal parsed dataset.

[0016] Furthermore, based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme for spatiotemporal resolution datasets is generated, and each partition is assigned to the executors;

[0017] Reservoir sampling is performed on each partition of the spatiotemporal resolution dataset, and a two-dimensional grid is constructed according to the range of the sampled samples in the time dimension and the space dimension. The number of points is constructed according to the number and category of the sampled spatiotemporal objects in each grid and the spatiotemporal objects it contains, and a statistical matrix is ​​obtained;

[0018] Based on the R-Star tree algorithm, a first objective function for generating a partitioning scheme is constructed according to the statistical matrix, and a load-balanced partitioning scheme is generated based on the first objective function;

[0019] Based on the partition boundaries in the partition scheme, the spatiotemporal objects in the spatiotemporal resolution dataset are allocated to corresponding partitions;

[0020] Each partition is assigned to an executor, wherein the allocation process takes the minimum variance change of the sum of the temporal and spatial ranges of the partitions carried by each executor as the second objective function, and ensures that the difference in the number of partitions on each executor is less than a predetermined threshold to achieve load balancing deployment among executors.

[0021] Further, based on the R-Star tree algorithm, a first objective function for generating a partition scheme is constructed according to the statistical matrix, and a load-balanced partition scheme is generated based on the first objective function, including:

[0022] Divide the leaf nodes of the R-Star tree into several initial partitions, each of which contains adjacent nodes distributed in time and space;

[0023] Acquire the number of spatiotemporal objects and computational complexity in each partition according to the statistical matrix, and construct a first objective function for partition scheme generation, wherein the computational complexity is determined according to the type of spatiotemporal object and the number of spatiotemporal object construction points contained in the spatiotemporal object;

[0024] The partition boundaries and node allocation are adjusted to minimize the first objective function, thereby obtaining a load-balanced partitioning scheme.

[0025] Furthermore, the user's query content is obtained, the retrieval model is used to locate relevant keywords and context information, and the query instructions are generated through the pre-trained large language model, including:

[0026] Obtain the user's query content, use the bag-of-words model to construct a corresponding query vector, and locate relevant keywords and context information by calculating the similarity between the query vector and each entry vector in the vector database;

[0027] The query content is subjected to reference resolution and expansion optimization using a pre-trained large language model, and a query instruction is generated based on the context information and the optimized query content.

[0028] Further, the query instruction is executed, and according to the partition metadata, candidate partitions that may contain the query results are globally screened, and local searches are performed in the candidate partitions to obtain search results, including:

[0029] Based on the STR tree containing the metadata of each partition, filter the partitions that are unlikely to contain the query results, thereby screening out candidate partitions;

[0030] For the candidate partitions, the query results are retrieved through the R-tree or spatial coding index.

[0031] According to a second aspect of an embodiment of the present application, a distributed intelligent retrieval system for spatiotemporal data is provided, comprising:

[0032] A parsing module is used to obtain a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file, and obtain a spatiotemporal parsed data set;

[0033] The partition allocation module is used to generate a load-balanced partitioning scheme for spatiotemporal resolution data sets based on reservoir sampling and R-Star tree algorithm, and allocate each partition to the executor;

[0034] The query instruction generation module is used to obtain the user's query content, locate relevant keywords and context information using the retrieval model, and generate query instructions through the pre-trained large language model;

[0035] The retrieval module is used to execute the query instruction, globally screen the candidate partitions that may contain the query results according to the partition metadata, and perform local retrieval in the candidate partitions to obtain the retrieval results.

[0036] According to a third aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0037] According to a fourth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0041] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0042] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0043] It can be seen from the above embodiments that the present application adaptively parses, corrects and unifies data modeling of spatiotemporal source data files; fully utilizes the computing resources of each Spark executor through a load-balanced partitioning method; has a large number of spatiotemporal computing operators built in and provides user-defined operator processes and interfaces, greatly improving user convenience; supports users to perform spatiotemporal data retrieval and report generation through natural language through pre-trained large language models and retrieval enhancement generation technology; when performing retrieval, uses partition metadata to perform global screening followed by local retrieval operations to avoid loading all data into memory, reducing the requirements for device hardware performance and improving retrieval efficiency.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 The present invention is a flowchart of a distributed intelligent retrieval method for spatiotemporal data according to an exemplary embodiment.

[0047] Figure 2 It is a block diagram of a distributed intelligent retrieval system for spatiotemporal data according to an exemplary embodiment.

[0048] Figure 3 is a schematic diagram of a custom operator according to an exemplary embodiment.

[0049] Figure 4 The diagram is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0051] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] Figure 1 is a flow chart of a distributed intelligent retrieval method for spatiotemporal data according to an exemplary embodiment. Figure 1 As shown, the method is implemented based on the distributed platform Spark and may include the following steps:

[0054] S1: Obtain a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file, and obtain a spatiotemporal parsed data set;

[0055] S2: Based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme is generated for the spatiotemporal resolution dataset, and each partition is assigned to the executor;

[0056] S3: Obtain the user's query content, use the retrieval model to locate relevant keywords and context information, and generate query instructions through the pre-trained large language model;

[0057] S4: Execute the query instruction, globally filter the candidate partitions that may contain the query results according to the partition metadata, and perform local search in the candidate partitions to obtain the search results.

[0058] From the above implementation, it can be seen that the present application adaptively parses, corrects and unifies data modeling of spatiotemporal source data files; fully utilizes the computing resources of each Spark executor through a load-balanced partitioning method; has a large number of built-in spatiotemporal computing operators and provides user-defined operator processes and interfaces, greatly improving user convenience; supports users to perform spatiotemporal data retrieval and report generation through natural language through pre-trained large language models and retrieval enhancement generation technology; when performing retrieval, uses partition metadata to perform global screening followed by local retrieval operations to avoid loading all data into memory, reducing the requirements for device hardware performance and improving retrieval efficiency.

[0059] In a specific implementation of S1, a spatiotemporal source data file is obtained, and the spatiotemporal source data file is preprocessed and parsed to obtain a spatiotemporal parsed data set;

[0060] S11: obtaining a spatiotemporal source data file provided by a user, and adaptively selecting a loading parser according to the suffix of the spatiotemporal source data file and the loading parameters specified by the user;

[0061] In the distributed intelligent retrieval method for spatiotemporal data of the present application, the spatiotemporal source data file can be of the following types: GeoJSON file, GeoJSON is a lightweight geospatial data exchange format based on the JSON standard. It supports geometric types such as points, lines, surfaces, polygons, and can embed attribute data, which is very suitable for storing map feature services, trajectory data, or socioeconomic statistics with geographic location information; Shapefile is a widely used vector data format developed by Esri, which is actually a collection of multiple files such as .shp (storing geometric information), .shx (index information) and .dbf (attribute table). It supports a variety of geometric types, such as points, lines, polygons, etc., and is commonly used in map making, urban planning, environmental analysis and other fields; TIFF (Tagged Image File Format), TIFF is a flexible image storage format, widely used in geographic information systems to store satellite images, aerial photographs, digital elevation models (DEM) and other raster data.

[0062] When processing the spatiotemporal source files provided by the user, the file format is first automatically identified by analyzing the file extension (such as .geojson, .shp, .tiff, etc.), and the corresponding file loading parser is matched accordingly. The file loading parser encapsulates and optimizes the existing loading and parsing code functions. The factory mode is adopted. The method can select the most appropriate parsing strategy according to the file extension, ensuring the adaptability of the parsing process. Subsequently, the parameters of the loading parser are set according to the additional loading parameters provided by the user, such as encoding format, delimiter, date format, etc. During parsing, the data will be streamed and the parsing operation will be started while the file is read, so as to be suitable for the processing scenarios of large-scale data files, reduce memory usage and speed up the data preparation process.

[0063] S12: parsing the spatiotemporal source data file using the loading parser, wherein, during the loading process of the spatiotemporal source data file, the spatiotemporal source data file is checked for data format consistency and integrity, and abnormal data therein is repaired;

[0064] When the loading parser attempts to load file data, the system automatically performs a series of checks, including but not limited to data format consistency checks (such as whether each column of data conforms to the expected data type), timestamp format verification, geographic coordinate system consistency, etc., to ensure that the data format is correct. Check whether there are missing rows or columns in the file, and whether the number of records is consistent with expectations to ensure data integrity. For detected abnormal data records, the system will immediately issue an alarm, record the abnormal information in detail, including data location, abnormal type, etc., and try to use default values ​​to fill in to repair the abnormal data.

[0065] S13: standardizing the parsed spatiotemporal objects to obtain a spatiotemporal parsed data set;

[0066] Convert the parsed and verified data output by the load parser into a unified user-defined or method-default data model to standardize the time and space attributes of spatiotemporal objects. For example, identify and convert timestamp information into a unified timestamp format, and convert geographic location information into standard longitude and latitude coordinates. Store the data in Apache Hadoop for subsequent distributed storage and computing.

[0067] In the specific implementation of S2, based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme of the spatiotemporal resolution data set is generated, and each partition is assigned to an executor;

[0068] In order to optimize the management of distributed storage data sets and ensure efficient data processing and query capabilities, this application uses a load balancing partitioner to reorganize the data set. The detailed steps are as follows:

[0069] S21: Perform reservoir sampling on each original partition of the spatiotemporal analysis data set, construct a two-dimensional grid according to the range of the sampled samples in the time dimension and the space dimension, construct the number of points according to the number and category of the sampled spatiotemporal objects in each grid and the spatiotemporal objects it contains, and obtain a statistical matrix;

[0070] First, the Reservoir Sampling algorithm is applied to randomly extract representative sample data sets at a fixed ratio from each original partition of the spatiotemporal parsing dataset parsed when the file is loaded into the parser. ,in, is the extracted spatiotemporal data sample, i=1, 2, …, n, is the sampling size. Sampling aims to efficiently estimate the temporal and spatial distribution characteristics of the complete data set without traversing all the data. Subsequently, the following statistics are performed on the sample:

[0071] 1. Create a 2D grid for the sample dataset ,in Indicates Line Grid space for columns, Represents the time interval within the grid.

[0072] 2. Calculate the number of spatiotemporal objects (single data object in the spatiotemporal dataset) in each grid of the sample dataset , Category frequency and the sum of the number of space-time object construction points required to form the object , forming a statistical matrix .

[0073] S22: Based on the R-Star tree algorithm, construct a first objective function for partition scheme generation according to the statistical matrix, and generate a load-balanced partition scheme based on the first objective function;

[0074] Using the R-Star tree algorithm, based on sampling information and grid statistics , design a dynamically balanced Apache Spark partitioning strategy, and the child nodes of the constructed R-Star tree will be used as partitions. The R-Star tree is an adaptive spatial index structure that is particularly suitable for processing spatial data with multi-dimensional attributes. It keeps the number of data objects in each node relatively balanced by continuously splitting nodes, and minimizes the overlap of coverage areas between nodes, which provides a good foundation for subsequent partitioning.

[0075] Assume the total number of partitions of the data set is , each partition (in ) is assigned a spatiotemporal eigenvector , respectively represent the number of spatiotemporal objects and computational complexity of the partition (obtained by weighting the object type and the number of spatiotemporal object construction points).

[0076] For this space-time object, when performing space-time calculations such as intersection and inclusion, the computational complexity can be expressed as:

[0077]

[0078] in, It is a weight coefficient determined according to the spatial type of the spatiotemporal object. The spatial types of the spatiotemporal object include point, line and surface. The default weight coefficients are 0.1, 0.3 and 0.5. Constructs the number of points for the spatiotemporal objects contained in the object, reflecting the computational resources required to process the object.

[0079] The specific process of generating the partition scheme driven by the R-Star tree algorithm is as follows

[0080] 1. Preliminarily divide the leaf nodes of the R-Star tree into several initial partitions, each of which contains adjacent nodes distributed in time and space;

[0081] Specifically, the number of partitions is equal to the number of leaf nodes initialized by the R-Star tree generated by the parameters of the R-Star tree, and the spatial and temporal ranges of each partition are equally divided. The goal of this stage is to quickly and roughly divide the partitions to prepare for subsequent fine-tuning.

[0082] 2. Obtain the number of spatiotemporal objects and the computational complexity in each partition according to the statistical matrix, and construct a first objective function for partition scheme generation, wherein the computational complexity is determined according to the type of spatiotemporal object and the number of spatiotemporal object construction points contained in the spatiotemporal object;

[0083] Based on the conventional splitting logic of the R-Star tree, the consideration of computational complexity is added. Specifically, when calculating the candidate splitting hyperplane of the R-Star tree, in addition to considering the number of spatiotemporal objects in each sub-region, the average computational complexity of each sub-region must also be calculated (according to the formula mentioned above). In this way, the splitting process not only pursues the balance of data volume, but also the balance of computational burden.

[0084] In order to maximize the balance of partitions, the objective function F can be defined as minimizing the sum of squared deviations of the number of spatiotemporal objects and computational complexity of each child node, that is:

[0085]

[0086] in, , is the adjustment factor, and are the average number of spatiotemporal objects and computational complexity of all partitions, respectively.

[0087] 3. Adjusting the partition boundaries and node allocation to minimize the first objective function, thereby obtaining a load-balanced partitioning scheme;

[0088] Specifically, a heuristic optimization algorithm is used to gradually adjust the partition boundaries and node allocations to minimize the objective function Each adjustment may involve the migration of nodes between partitions, or the merging / splitting of certain nodes. The node splitting and merging method of R-Tree will ensure that the partitions still maintain continuity in time and space.

[0089] The partitioning scheme is considered to be optimal by continuous adjustment until the decrease of the objective function is lower than the predetermined threshold or the maximum number of iterations is reached.

[0090] S23: allocating the spatiotemporal objects in the spatiotemporal resolution data set to corresponding partitions based on the partition boundaries in the partition scheme;

[0091] Based on the spatiotemporal partition boundaries generated by the R-Star tree, each spatiotemporal object in the spatiotemporal parsing dataset is Assign to the partition that intersects its time and space range, that is, find the partition that meets the conditions , so that .

[0092] S24: Allocate each partition to an executor, wherein the allocation process takes the minimum variance change of the sum of the temporal and spatial ranges of the partitions carried by each executor as the second objective function, and ensures that the difference in the number of partitions on each executor is less than a predetermined threshold, so as to achieve load balancing deployment among executors;

[0093] When deploying partitions to Apache Spark executors, an optimization algorithm is used to find the variance that makes the sum of the temporal and spatial ranges of the partitions on each executor The minimal solution, while keeping the number of partitions on the executors close, is:

[0094]

[0095] in, is the allowed difference in the number of partitions to ensure load balancing between executors.

[0096] S25: The allocated partition data and corresponding partition metadata (including partition boundaries, object quantity statistics, etc.) are persistently stored in the Apache Hadoop Distributed File System (HDFS) in the form of files, providing an efficient data access basis for subsequent data processing and query.

[0097] In a specific implementation of S3, the user's query content is obtained, the relevant keywords and context information are located using the retrieval model, and the query instruction is generated through the pre-trained large language model, specifically including:

[0098] S31: Obtain the user's query content, construct a corresponding query vector using the bag-of-words model, and obtain context information in the large model input Prompt by calculating the similarity between the query vector and each item vector in the vector database;

[0099] In this embodiment, the user's query content may be "calculate the spatiotemporal objects in the data set that intersect with the given query range", and each entry vector in the vector database may be "all supported operators, operator descriptions, and operator examples in the system". By calculating the similarity between the query content and each entry vector, the ST_Intersect operator entry will be retrieved, and the operator description and operator example of ST_Intersect will be used as context information in the Prompt input by the large model.

[0100] The vector space model enhanced by term frequency-inverse document frequency (TF-IDF) is used to retrieve keywords and contextual information related to the query content in the vector database. This method not only considers the importance of term frequency in the query, but also weighs the prevalence of the term in the entire knowledge base to better measure the uniqueness and information value of the keyword. Specifically, the bag-of-words model is used to construct the query vector using the TF-IDF (term frequency-inverse document frequency) value of each word in the query content. Each entry in the knowledge base is represented as a vector consisting of the weights of its vocabulary items, which are calculated using TF-IDF. Subsequently, by calculating the similarity between the query vector and the vectors of each entry in the vector database (usually using cosine similarity), the function descriptions, data attribute descriptions, and contextual environments that are most relevant to the query are quickly located. This preliminary screening step ensures that subsequent processing focuses on the subset of information that is most likely to meet user needs.

[0101] This process can be summarized as:

[0102]

[0103] in, It is a function that measures the similarity between vectors.

[0104] For the vector database, the present application first constructs a comprehensive knowledge base system, which includes two levels: one is the system function knowledge base, which contains detailed information of spatiotemporal computing operators, such as function name, function description, parameter description and call example; the other is the data description knowledge base, which records in detail the data mode, statistical matrix and partition metadata of the data set in distributed storage. Then use the pre-trained text-to-embedding model to convert the text in the knowledge base into vectors and store them in the vector database. The vector database can be a PostgreSQL database with the pgvector plug-in installed. Specifically, in order to achieve effective understanding of natural language queries, the system uses the pre-trained text-to-embedding model BERT to convert the text information in the knowledge base into a high-dimensional vector representation. These vectors can capture semantic similarity and facilitate subsequent retrieval operations. These vectors are stored in a PostgreSQL database equipped with a pgvector plug-in. pgvector is a PostgreSQL extension optimized for vector search, which uses an efficient approximate nearest neighbor algorithm HierarchicalNavigable Small World (HNSW) graph to achieve fast and relatively accurate vector retrieval.

[0105] More specifically, the spatiotemporal computing operators are designed to cover a wide range of application scenarios and analysis requirements, including not only basic operations such as data filtering (filtering data based on timestamps or spatial coordinates), aggregation (aggregating statistical data by time window or geographic area), and type conversion (such as converting time series data into spatiotemporal points), but also more complex query algorithms, such as:

[0106] 1. Range query: quickly locate spatiotemporal objects that meet specific conditions within a specific time period or geographic area.

[0107] 2. K-nearest neighbor query: For a given time and space point, find the k nearest objects around it, which is often used in recommendation systems and navigation services.

[0108] 3. Join query: realize joint query based on spatiotemporal association between different data tables, suitable for multi-source data fusion analysis.

[0109] In addition, the system also supports advanced spatiotemporal analysis methods, such as:

[0110] 1. Cluster analysis: Grouping data based on spatiotemporal characteristics to identify potential spatial patterns or abnormal behaviors, which is particularly important for analyzing urban migrant populations.

[0111] 2. Overlay analysis: Overlay different geographic feature layers to discover the spatial relationship between them. It is widely used in land use planning and disaster risk assessment.

[0112] 3. Vector clipping: Clipping vector data according to specific geographic boundaries to extract areas of interest for in-depth analysis, which is common in the fields of ecology and urban planning.

[0113] 4. OD analysis: OD analysis focuses on the movement patterns of individuals or goods from the starting point to the destination, and is often used in traffic planning, passenger flow analysis, and logistics optimization.

[0114] 5. Regional summary analysis: Regional summary analysis is a statistical analysis of data within each specific area (such as administrative divisions, river basins), such as calculating the average temperature, total population, etc.

[0115] 6. Topology check: Topology check ensures the logical consistency of geospatial data, such as checking whether the features are correctly connected, whether there are hanging points or overlapping areas. This is crucial to maintaining data quality and preventing error propagation.

[0116] 7. Kernel density analysis: Kernel density analysis is a method of estimating continuous spatial distribution density based on point data, and is often used for population density or wildlife habitat analysis.

[0117] 8. Buffer Analysis: Buffer analysis creates an area of ​​fixed distance around a specific geographic feature (such as a road, building, or river) to study the impact or accessibility around that feature.

[0118] In addition, the spatiotemporal calculation operator may also be a user-defined operator, which is not limited in this application.

[0119] S32: using a pre-trained large language model to perform reference resolution and expansion optimization on the query content, and generating a query instruction based on the context information and the optimized query content;

[0120] On the basis of retrieving relevant knowledge from the knowledge base, we further use the SQL statement generation capability of pre-trained large language models (such as ChatGPT and Tongyi Qianwen) to generate accurate Spark SQL query instructions, mainly including:

[0121] (1) Co-reference resolution: The large language model resolves possible ambiguity in queries (such as “it”, “the second point”, etc.) by understanding the context.

[0122] (2) Extension and optimization: Based on the understanding ability of the large language model, the original query is semantically enriched and structurally improved, such as adding missing subject-verb-object structures or keywords, so as to more accurately match the content in the knowledge base.

[0123] (3) Generate query instructions: Combining the context information obtained from vector retrieval and the optimized statement of the question, the large language model generates targeted Spark SQL query instructions.

[0124] In general, based on the pre-trained large language model, seamless conversion from natural language to program instructions is achieved by understanding the context, inferring user intent, and generating grammatically correct and logically coherent code snippets.

[0125] S4: executing the query instruction, globally screening the candidate partitions that may contain the query results according to the partition metadata, and performing a local search in the candidate partitions to obtain the search results;

[0126] The pre-trained large language model generates a spatiotemporal data retrieval analysis report based on the obtained retrieval results and the user's input instructions. The system provides a prompt word template for the pre-trained large language model, for example, requiring the report to include introduction, methods, results, discussion and conclusion. The system uses the pre-trained large language model to generate the content of each part of the template based on user prompts, retrieval requirements and retrieval result data, thereby generating a complete spatiotemporal data retrieval analysis report.

[0127] In order to speed up the retrieval of spatiotemporal data, this application adopts a hybrid indexing strategy based on global STR-Tree and local R-Tree or spatial coding index. For the global scope of all partitions, STR-Tree is used to summarize the statistical information of each partition metadata to help quickly filter out a large number of partitions that do not meet the conditions. In the local scope of a single partition, R-Tree or spatial coding index is built for the data in each partition to speed up the data retrieval within a single partition.

[0128] S41: based on the STR tree containing metadata of each partition, filter the partitions that are unlikely to contain the query results, thereby screening out candidate partitions;

[0129] Query optimization rules based on partition metadata. New optimization rules are added to Catalyst, the query processing framework of Apache Spark, which uses partition metadata to filter out partitions that are unlikely to contain query results in advance. For example, when performing spatial intersection queries, only the partition data whose partition boundaries intersect with the query range is calculated. This step significantly reduces the number of partitions that need to be further checked.

[0130] S42: For the candidate partition, retrieve the query result through the R-tree or spatial coding index;

[0131] Specifically, for the selected candidate partitions, local indexes are used to perform more refined data retrieval to find the exact data records that meet the query conditions. In the data reading and output stages, parallel I / O operations are performed based on the partition metadata to improve data processing speed. Throughout the process, the retrieval strategy is dynamically adjusted according to the query characteristics and data distribution, such as selecting the most appropriate index type and determining the size of the data block for parallel reading and writing, to achieve optimal execution efficiency.

[0132] This application implements the following data file splitting and parallel reading and writing rules: For file formats that natively support parallel reading and writing, such as Csv and Parquet, existing parallel reading and writing methods are used. For file formats that do not natively support parallel reading and writing, such as GeoJSON, a data preprocessing strategy is adopted to reasonably split the data according to the partition metadata, combined with customized JSON parsing logic to ensure the data integrity of the split points and achieve parallel reading. At the same time, parallel writing is supported, and the processed data is output to multiple files, and finally aggregated into a single output file, which greatly improves the efficiency of data processing and export.

[0133] Corresponding to the aforementioned embodiment of the distributed intelligent retrieval method for spatiotemporal data, the present application also provides an embodiment of a distributed intelligent retrieval system for spatiotemporal data.

[0134] Figure 2is a block diagram of a distributed intelligent retrieval system for spatiotemporal data according to an exemplary embodiment. Figure 2 As shown, the system is built on the distributed platform Spark and can include:

[0135] The parsing module 21 is used to obtain a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file, and obtain a spatiotemporal parsed data set;

[0136] A partition allocation module 22 is used to generate a load-balanced partitioning scheme for the spatiotemporal resolution data set based on reservoir sampling and R-Star tree algorithm, and allocate each partition to an executor;

[0137] A query instruction generation module 23 is used to obtain the user's query content, locate relevant keywords and context information using the retrieval model, and generate a query instruction through a pre-trained large language model;

[0138] The search module 24 is used to execute the query instruction, globally screen the candidate partitions that may contain the query results according to the partition metadata, and perform local search in the candidate partitions to obtain the search results.

[0139] It should be noted that the system supports custom operators, including:

[0140] Development and integration of user-defined operators: In order to meet the diverse analysis needs of users, the system provides detailed development guides and API documentation, supporting users to write custom operators in high-level programming languages ​​such as Python and Java. This flexibility allows users to design and implement more complex or customized analysis algorithms based on specific business logic. The system uses an efficient Kryo serializer to ensure efficient and consistent transmission of data during distributed computing, reducing the performance overhead of serialization and deserialization and improving overall computing efficiency.

[0141] Registration and calling mechanism of custom operators: Figure 3As shown in the figure, in order to simplify the use process of user-defined operators, the system has designed a convenient registration and calling mechanism, including spatiotemporal operators executed on the CPU and GPU. Specifically, users design and implement operators on different runs through the system function API, computing library and resource manager. Operators on GPU are mainly implemented based on Rapids library, and operators on CPU are mainly implemented based on JTS library. The memory management of all operators is implemented through Java Virtual Machine (JVM) resource manager. Finally, the written custom operator is registered as a user-defined function (UDF) of Spark SQL. After successful registration, the function can be directly called by its function name in the Spark SQL query statement. Spark SQL is checked and decomposed into execution plan in Spark driver, compiled into Java bytecode and sent to Spark executor. For operators running on CPU, they are executed directly through JVM. For operators running on GPU, data is exchanged between GPU and JVM through Java Native Interface (JNI). The system also establishes a knowledge base to summarize and record the names, function descriptions and calling examples of all registered custom operators, so that users can quickly find and reference them through natural language retrieval function.

[0142] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0143] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0144] Correspondingly, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned distributed intelligent retrieval method for spatiotemporal data.

[0145] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned distributed intelligent retrieval method for spatiotemporal data. Figure 4As shown in FIG. 1 , a hardware structure diagram of a distributed intelligent retrieval system for spatiotemporal data provided by an embodiment of the present invention is provided for any device with data processing capability, except Figure 4 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0146] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the distributed intelligent retrieval method for spatiotemporal data as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be other external storage devices, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0147] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

Claims

1. A distributed intelligent retrieval method for spatiotemporal data, characterized in that: include: Acquire a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file to obtain a spatiotemporal parsed data set; Based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme for spatiotemporal resolution datasets is generated, and each partition is assigned to the executor; Obtain the user's query content, use the retrieval model to locate relevant keywords and context information, and generate query instructions through the pre-trained large language model; Execute the query instruction, globally filter the candidate partitions that may contain the query results according to the partition metadata, and perform local search in the candidate partitions to obtain the search results; Among them, based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme for spatiotemporal resolution data sets is generated, and each partition is assigned to the executor; Reservoir sampling is performed on each partition of the spatiotemporal resolution dataset, and a two-dimensional grid is constructed according to the range of the sampled samples in the time dimension and the space dimension. The number of points is constructed according to the number and category of the sampled spatiotemporal objects in each grid and the spatiotemporal objects it contains, and a statistical matrix is ​​obtained; Based on the R-Star tree algorithm, a first objective function for generating a partitioning scheme is constructed according to the statistical matrix, and a load-balanced partitioning scheme is generated based on the first objective function; Based on the partition boundaries in the partition scheme, the spatiotemporal objects in the spatiotemporal resolution dataset are allocated to corresponding partitions; Allocate each partition to an executor, wherein the allocation process takes the minimum variance change of the sum of the temporal and spatial ranges of the partitions carried by each executor as the second objective function, and ensures that the difference in the number of partitions on each executor is less than a predetermined threshold, so as to achieve load balancing deployment among executors; Wherein, based on the R-Star tree algorithm, a first objective function for generating a partition scheme is constructed according to the statistical matrix, and a load-balanced partition scheme is generated based on the first objective function, including: Divide the leaf nodes of the R-Star tree into several initial partitions, each of which contains adjacent nodes distributed in time and space; Acquire the number of spatiotemporal objects and computational complexity in each partition according to the statistical matrix, and construct a first objective function for partition scheme generation, wherein the computational complexity is determined according to the type of spatiotemporal object and the number of spatiotemporal object construction points contained in the spatiotemporal object; The partition boundaries and node allocation are adjusted to minimize the first objective function, thereby obtaining a load-balanced partitioning scheme.

2. The method according to claim 1, characterized in that Obtaining a spatiotemporal source data file, preprocessing and parsing the spatiotemporal source data file, and obtaining a spatiotemporal parsed data set, including: Obtaining a spatiotemporal source data file provided by a user, and adaptively selecting a loading parser according to the suffix of the spatiotemporal source data file and the loading parameters specified by the user; The spatiotemporal source data file is parsed by using the loading parser, wherein, during the loading process of the spatiotemporal source data file, the spatiotemporal source data file is checked for data format consistency and integrity, and abnormal data therein is repaired; The parsed spatiotemporal objects are standardized to obtain a spatiotemporal parsed dataset.

3. The method according to claim 1, characterized in that Obtain the user's query content, use the retrieval model to locate relevant keywords and context information, and generate query instructions through the pre-trained large language model, including: Obtain the user's query content, use the bag-of-words model to construct a corresponding query vector, and locate relevant keywords and context information by calculating the similarity between the query vector and each entry vector in the vector database; The query content is subjected to reference resolution and expansion optimization using a pre-trained large language model, and a query instruction is generated based on the context information and the optimized query content.

4. The method according to claim 1, characterized in that Execute the query instruction, globally filter the candidate partitions that may contain the query results according to the partition metadata, perform local search in the candidate partitions, and obtain the search results, including: Based on the STR tree containing the metadata of each partition, filter the partitions that are unlikely to contain the query results, thereby screening out candidate partitions; For the candidate partitions, the query results are retrieved through the R-tree or spatial coding index.

5. A distributed intelligent retrieval device for spatiotemporal data, characterized in that: include: A parsing module is used to obtain a spatiotemporal source data file, preprocess and parse the spatiotemporal source data file, and obtain a spatiotemporal parsed data set; The partition allocation module is used to generate a load-balanced partitioning scheme for spatiotemporal resolution data sets based on reservoir sampling and R-Star tree algorithm, and allocate each partition to the executor; The query instruction generation module is used to obtain the user's query content, locate relevant keywords and context information using the retrieval model, and generate query instructions through the pre-trained large language model; A retrieval module is used to execute the query instruction, globally screen the candidate partitions that may contain the query results according to the partition metadata, and perform local search in the candidate partitions to obtain the search results; Among them, based on reservoir sampling and R-Star tree algorithm, a load-balanced partitioning scheme for spatiotemporal resolution data sets is generated, and each partition is assigned to the executor; Reservoir sampling is performed on each partition of the spatiotemporal resolution dataset, and a two-dimensional grid is constructed according to the range of the sampled samples in the time dimension and the space dimension. The number of points is constructed according to the number and category of the sampled spatiotemporal objects in each grid and the spatiotemporal objects it contains, and a statistical matrix is ​​obtained; Based on the R-Star tree algorithm, a first objective function for generating a partitioning scheme is constructed according to the statistical matrix, and a load-balanced partitioning scheme is generated based on the first objective function; Based on the partition boundaries in the partition scheme, the spatiotemporal objects in the spatiotemporal resolution dataset are allocated to corresponding partitions; Allocate each partition to an executor, wherein the allocation process takes the minimum variance change of the sum of the temporal and spatial ranges of the partitions carried by each executor as the second objective function, and ensures that the difference in the number of partitions on each executor is less than a predetermined threshold, so as to achieve load balancing deployment among executors; Wherein, based on the R-Star tree algorithm, a first objective function for generating a partition scheme is constructed according to the statistical matrix, and a load-balanced partition scheme is generated based on the first objective function, including: Divide the leaf nodes of the R-Star tree into several initial partitions, each of which contains adjacent nodes distributed in time and space; Acquire the number of spatiotemporal objects and computational complexity in each partition according to the statistical matrix, and construct a first objective function for partition scheme generation, wherein the computational complexity is determined according to the type of spatiotemporal object and the number of spatiotemporal object construction points contained in the spatiotemporal object; The partition boundaries and node allocation are adjusted to minimize the first objective function, thereby obtaining a load-balanced partitioning scheme.

6. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images