A three-dimensional geoscience data lake warehouse index scheduling method

Abstract

The application discloses a three-dimensional geoscience data lake warehouse index scheduling method and relates to the fields of geoscience information and big data technologies. A three-dimensional geoscience query request is received, attribute constraints, a three-dimensional space range, a vertical interval, a time version constraint and a scale requirement are analyzed, and a request state is uniformly referenced and aligned to construct; a request type is identified based on the request state, and a scheduling template is generated; a block-level three-dimensional enclosing index, a geoscience semantic adjacent index and an attribute statistical index are called to calculate a comprehensive score of a candidate block, and a candidate block set is obtained through screening; a candidate path cost is calculated, a main path is selected, a data pulling granularity is determined, target data blocks are pulled, and historical statistics and index information are updated, so that the candidate block inflation and the I / O page reading amount are reduced, and the three-dimensional geoscience retrieval efficiency and response stability are improved.

Description

Technical Field

[0001] This invention belongs to the field of geoscience information and big data technology, and specifically relates to a three-dimensional spatial data lake warehouse indexing and scheduling method. Background Technology

[0002] With the development of natural resource surveys, geological modeling, remote sensing interpretation, subsurface space analysis, and digital twin earth applications, the scale and heterogeneity of 3D geoscientific data continue to rise. Data objects in practical applications are no longer limited to two-dimensional rasters or single-table records, but simultaneously include voxel models, borehole curves, 3D stratigraphic layers, fault topology, time-series remote sensing images, attribute tables, point clouds, and derived analysis results. This type of data typically exhibits characteristics of multiple sources, multiple scales, multiple coordinate references, and multiple storage media: hot data can be stored in object storage or columnar engines to support parallel analysis, cold data is retained in file lakes, structured attributes and task logs are stored in relational databases, while spatial boundaries, chunk summaries, and statistical information are scattered in metadata catalogs. For retrieval systems, the difficulty lies not in the existence of accessible data, but in how to identify whether a query is geometrically localized, attribute-filtered, cross-scale stitching, or spatiotemporally coupled, and select different indexing paths accordingly. If a single table directory or single-dimensional spatial index is still used, the query optimizer can only make local decisions in an incomplete state, which can easily lead to problems such as overexpansion of candidate blocks, repeated scanning across levels, and indiscriminate back-to-source of cold and hot data. This can result in increased I / O page read volume, excessively deep metadata traversal, and the fetching of spatially adjacent but semantically unrelated data in three-dimensional range queries.

[0003] In the integrated lake-warehouse architecture, the focus of data management has gradually shifted from a unified storage format to the collaborative use of heterogeneous indexes under a unified retrieval portal. This is particularly relevant in 3D geoscience scenarios, where spatial objects exhibit significant hierarchical nesting, geological body cutting, vertically non-uniform sampling, and cross-scale representation. For example, the same target area may simultaneously require meter-level digital elevation model slices, ten-meter-level remote sensing images, borehole columnar sections, and regional-level stratigraphic framework models. In such cases, if the index only describes file-level locations or two-dimensional bounding boxes, it becomes difficult to express vertical coverage relationships, volumetric density differences, and geological semantic adjacency relationships. If scheduling relies solely on static rules to retrieve data, it cannot dynamically determine whether to perform attribute filtering followed by spatial pruning or 3D spatial candidate reduction followed by table verification based on the request type. Furthermore, the vertical sampling of 3D geoscience data is typically non-uniform; fault cutting and stratigraphic interruptions exist between boreholes, stratigraphic bodies, and profile lines, and two-dimensional planar adjacency does not equate to profile continuity. Data blocks within the same planar range may also correspond to different interpretations due to version, stratigraphy, or scale. Therefore, the technical challenges of 3D geoscience data lake warehouses have evolved from simple data organization to scheduling issues related to query intent recognition, composite index selection, and the linkage of differentiated retrieval strategies.

[0004] To address specific issues in the processing, existing technologies have explored various approaches. Based on their technical approaches, they can be mainly divided into two categories: one focuses on the unified organization and cross-domain management of lake warehouse data, and the other focuses on feature representation and physical prior modeling in geoscientific scenarios.

[0005] The first category concerns the unified organization and cross-domain management of lakehouse data. For example, Chinese patent CN202511568408.3 discloses a customer operation method and system for federated data lakehouses based on community search. It solves the problem of customer group identification and operation under the condition that cross-domain data does not leave its domain through multimodal heterogeneous customer graph modeling, federated collaborative graph embedding and alignment, and a community search mechanism. The focus of this type of technology is on graph representation learning and information fusion under privacy compliance. Its application is limited to inter-domain relationship mining and community-level search, and it does not address spatial partitioning, voxel candidate shrinking, or real-time query path switching for 3D geoscientific objects. Another example is Chinese patent CN202310390219.6, which discloses a multi-source remote sensing spatiotemporal big data processing method and device based on an integrated lakehouse. It generates a metadata catalog by parsing raw remote sensing spatiotemporal data, constructs data asset file snapshots, and forms a unified table format to support unified lookup by the computing engine. The core of this type of technology lies in data discovery driven by a unified tabular format and metadata catalog. Its default query entry mainly revolves around parsing the tabular catalog, which is suitable for minute-level or batch data processing and unified searching. However, it is still insufficient for scheduling depth of interactive range retrieval, attribute filtering and joint shrinking of spatial candidates in 3D geoscience scenes.

[0006] The second category involves feature representation and physical prior modeling within geoscientific scenarios. For example, Chinese patent CN202610807884.4 discloses a remote sensing landslide segmentation method and system based on geoscientific physical priors and cross-modal decoupling enhancement. This method improves landslide identification accuracy through slope field construction, heterogeneous co-construction of input tensors, and physical prior constraints. This type of technology processes raster features resulting from the fusion of imagery and digital elevation models, with its application concentrated in the disaster identification model inference stage. While this method illustrates the importance of physical constraints in geoscientific scenarios, it does not address the indexing organization, query intent identification, access cost prediction, and differentiated data retrieval of 3D geoscientific data blocks within the lake-canopy system. Therefore, it cannot directly solve the index scheduling problem in the 3D lake-canopy retrieval chain.

[0007] The aforementioned existing technologies have at least the following shortcomings in practical applications.

[0008] The unified catalog is searchable but not schedulable. While the first type of method described above can improve data discovery capabilities, its retrieval entry point is essentially still a catalog-driven static search mode. This mode defaults to locating the table first and then accessing the data, lacking joint modeling for 3D range constraints, vertical layer constraints, attribute selectivity, and cross-scale stitching requirements. This means that the query optimization process does not establish a correlation between request semantics, candidate block reduction, and actual pull cost, and can only make path decisions on coarse-grained metadata, resulting in the candidate result set being over-expanded before entering the underlying lakehouse scan.

[0009] The single-track operation of index structures leads to spatiotemporal semantic mismatch. Ordinary relational database indexes are better at filtering single-table attributes, and single-dimensional spatial indexes are better at planar bounding box filtering. However, 3D geoscientific requests often simultaneously include spatial location, vertical intervals, scale levels, geological types, and time versions. If these conditions are processed sequentially, spatially adjacent data blocks with inconsistent stratigraphy, data blocks separated by faults on the profile, or data blocks with different versions may prematurely enter the candidate set, or data blocks that meet the attribute requirements but have insufficient 3D coverage may repeatedly need to be checked against the table. None of the existing technologies mentioned above have solved the index mismatch problem caused by the asynchrony between spatial adjacency relationships and geoscientific semantic relationships in 3D lake warehouses.

[0010] Static data retrieval strategies lead to increased I / O page reads and an imbalance between hot and cold data. Most existing solutions default to a uniform approach to data retrieval after a query hit, lacking differentiated scheduling based on request type. For example, overview queries only require low-resolution summary blocks, profile analysis queries rely more on vertically contiguous blocks, and detailed queries require local high-resolution voxels and their adjacency attributes. If static data retrieval is still used, the system will retrieve too much detailed data in ordinary queries and frequently perform secondary accesses in detailed queries due to insufficient prefetching. The first type of method mentioned above addresses the issue of uniform organization rather than dynamic scheduling; it cannot predict which index path and retrieval granularity are more suitable for the current request before access, thus making it difficult to suppress the amplification of I / O page reads in cross-scale retrieval.

[0011] Therefore, there is a need in this field for a 3D geoscience data lakehouse index scheduling method that can perform request type identification, joint index candidate shrinking, path cost evaluation, and differentiated data retrieval for 3D geoscience requests under a unified lakehouse entry point. Summary of the Invention

[0012] The purpose of this invention is to provide a three-dimensional geoscience data lake warehouse index scheduling method to solve the problem that the unified directory in the prior art can only support static search and cannot perform index path linkage selection for three-dimensional geoscience requests, solve the problem of the separation between relational attribute index, single-dimensional spatial index and three-dimensional geoscience semantic relationship, and solve the problem that data retrieval is executed according to static rules and is difficult to adapt to different request modes.

[0013] To achieve the above objectives, this invention provides a three-dimensional geoscientific data lake warehouse index scheduling method, executed by a computer device. The computer device includes at least a processor, memory, a network interface, and a bus interface connected to object storage, a metadata catalog, a relational attribute database, and an index service. The method revolves around the main processing lines of request type identification, composite index candidate shrinking, scheduling path allocation, differentiated data retrieval, and feedback updates, and includes the following steps.

[0014] Step S1, Request Access and Unified State Construction: Receive a 3D geoscience query request, parse it to obtain attribute constraints, 3D spatial range, vertical interval, time version constraints, and scale requirements; decompose the query request into attribute atomic items, 3D spatial atomic items, vertical atomic items, time atomic items, and scale atomic items; convert the 3D spatial atomic items into 3D minimum bounding range, the vertical atomic items into upper and lower depth bounds, the time atomic items into logical timestamps, and the scale atomic items into target resolution levels; and perform unified reference alignment on the 3D spatial range and time version constraints to construct a request state containing query semantic features, spatial range features, vertical features, scale features, hot and cold distribution features, version density features, connection complexity features, and historical access smoothing statistics.

[0015] Step S2, Request Type Identification and Template Selection: Based on the request status, the request type is identified, and a corresponding scheduling template is generated according to the identification result; wherein, the request type includes at least overview request, range analysis request, profile continuous request, and local detailed investigation request; when the uncertainty of request type identification meets the preset conditions, a parallel candidate path template corresponding to the hybrid request is generated so that the boundary request is compared with multiple paths before the formal detailed back-to-source; when the uncertainty of request type identification does not meet the preset conditions, a single path scheduling template is generated according to the main type.

[0016] Step S3, Joint Index Candidate Shrinking: The block-level 3D bounding index, geoscientific semantic adjacency index, and attribute statistics index associated with a unified block identifier are invoked to calculate a comprehensive candidate score for the candidate data blocks. The comprehensive candidate score is jointly determined by the spatial overlap relationship between the requested range and the block bounding information, the matching relationship between attribute constraints and the block-level statistical summary, the geoscientific semantic adjacency relationship, and the block back-to-source cost. Based on the comprehensive candidate score, threshold filtering, sorting, candidate truncation, and adjacency sparsification are performed to obtain a set of candidate blocks.

[0017] Step S4, scheduling path allocation and differentiated data retrieval: calculate the path cost for attribute-priority path, spatial-priority path, vertical chain-priority path and summary preview path respectively; the path cost is determined by at least the estimated response time, estimated I / O page read volume, candidate block redundancy rate and path uncertainty; select the main path according to the path cost; under the parallel candidate path template corresponding to the hybrid request, perform summary prefetching on at least two candidate paths and then make a second decision; and determine the data retrieval granularity according to the selected path, candidate block set, request type, candidate density and hot / cold distribution.

[0018] Step S5, Result Return and Feedback Update: Pull the target data block according to the determined data granularity and return the retrieval results, record the real response indicators and actual access results, update the historical access smoothing statistics and index statistics, and roll back to the previous stable version when the performance deteriorates to meet the rollback conditions after the online update of the model.

[0019] Furthermore, the request state generation process in step S1 includes: generating a unified-dimensional request state based on the query length, three-dimensional spatial bounding volume, vertical interval length, time range length, target resolution level, number of various constraints, hot and cold data ratio, version density, connection complexity, and smoothed statistics of historical hit files, historical page fetches, and historical response latency. By mapping constraints from different sources, scales, and semantic types to a unified input dimension, subsequent request type identification and path scheduling can be performed within the same numerical space.

[0020] Furthermore, step S1 also includes in-situ missing value encoding. When any one of the spatial range feature, vertical feature, or temporal feature is missing, the corresponding state component is set to a preset null value encoding, preferably a zero value encoding; the constraint count term corresponding to the missing item is set to zero; and a complexity compensation amount corresponding to the missing item is superimposed on the connection complexity feature to reflect the uncertainty of subsequent path selection. The input dimension of the request state is kept constant without increasing the additional input dimension to avoid inconsistencies in the input structure between the training and online inference phases.

[0021] Further, the request type identification in step S2 includes: inputting the request status into the request type identification model, and outputting the type probabilities of the corresponding overview request, range analysis request, profile continuous request, and local detailed investigation request; selecting the request type corresponding to the maximum type probability as the main type; when the maximum type probability is lower than a preset threshold, marking the current request as a hybrid request and triggering parallel evaluation of at least two candidate paths; when the maximum type probability is not lower than the preset threshold, generating a single-path scheduling template based on the main type.

[0022] Furthermore, the geoscientific semantic adjacency index records at least one of the following: stratigraphic number, fault association number, subject asset category, version family marker, and adjacency block relationship; and the candidate adjacency relationships in the geoscientific semantic adjacency index are established according to the following rules: candidate blocks satisfy spatial contact relationship or spatial distance does not exceed a preset tolerance; candidate blocks satisfy version consistency constraint or belong to the same valid version family; candidate blocks satisfy at least one semantic constraint of stratigraphic number consistency and fault number adjacency; when the request is a profile continuity request, blocks that are vertically adjacent within the continuous band are marked as vertical priority adjacency edges. By incorporating spatial adjacency, version relationship, and geoscientific semantic relationship into the adjacency establishment rules, invalid blocks that enter the candidate set solely due to projection proximity can be suppressed.

[0023] Furthermore, the comprehensive candidate score is generated jointly by at least spatial overlap, attribute matching degree, semantic continuity, and back-to-origin cost. Specifically, spatial overlap is determined by the ratio of the intersection volume of the query enclosing range and the data block enclosing range to the total volume of the merged range; attribute matching degree is determined by the compatibility between the attribute predicate and the block-level statistical summary; semantic continuity is determined by at least two of the following: stratigraphic consistency score, fault association consistency score, and adjacency continuity score; and back-to-origin cost is jointly determined by the hot-layer hit rate of the data block, the block size, and the remote transmission level. Thus, candidate blocks do not enter the result set solely based on spatial proximity or a single attribute match, but are narrowed down through multi-dimensional constraints of space, attributes, semantics, and cost.

[0024] Furthermore, the step of performing candidate shrinkage based on the comprehensive candidate score includes: sorting candidate data blocks in descending order according to the comprehensive candidate score; retaining candidate data blocks whose comprehensive candidate score is not lower than a preset retention threshold; retaining the first preset number of candidate data blocks from the candidate data blocks after threshold filtering; establishing adjacency edges only for candidate block pairs that meet the requirements of spatial contact, consistent version, and identical stratigraphic numbers or adjacent fault numbers, and limiting the number of adjacent blocks retained for each candidate data block to complete adjacency sparsity. Through the dual constraints of threshold filtering followed by candidate truncation, the complexity of the adjacency graph can be controlled while retaining effective candidate blocks.

[0025] Furthermore, the path cost in step S4 includes the estimated response time, estimated I / O page read volume, candidate block redundancy rate, and path uncertainty for each candidate path; and the path cost for each candidate path is generated based on the estimated response time, estimated I / O page read volume, candidate block redundancy rate, and path uncertainty; the candidate path with the lowest path cost is selected as the primary path; when the current request is a hybrid request, at least two candidate paths with the lowest path cost are selected to perform the first batch of digest prefetching, and a secondary decision is performed based on the hit density and path cost of the first batch of digest prefetching. This allows the uncertainty in the request identification stage to be further mitigated in the lightweight prefetching stage, rather than directly entering the high-cost detailed source retrieval.

[0026] Furthermore, the step of fetching target data blocks at a determined data fetching granularity includes: when the request type is an overview request, fetching at least one of a summary block, a statistics block, and a low-resolution block; when the request type is a range analysis request, first fetching the header summary of the candidate data blocks, and then performing detailed source retrieval on the high-scoring candidate data blocks; when the request type is a profile continuous request, serially fetching the data blocks corresponding to the continuous hit segments in the order of vertical adjacency; when the request type is a local detailed search request, performing high-resolution fetching on the target data block and its first-order adjacent blocks, and simultaneously fetching attribute details.

[0027] Furthermore, when the main path prediction times out, the index service is unavailable, or the candidate hit density is lower than a preset threshold, the process degrades to a local incremental back-to-source mode with summary blocks first. When the attribute statistical summary missing rate exceeds a preset threshold, attribute screening is skipped and a combined spatial and semantic constraint contraction is adopted. This abnormal degradation mechanism is embedded in the path execution stage to output usable results even when the index service is abnormal, the summary is incomplete, or the candidate density is abnormal.

[0028] Furthermore, the request type recognition model employs an offline training, online inference, and controlled incremental update deployment approach. Training samples are derived from historical 3D geoscientific query logs, real scheduling logs, historical task records from the underground space survey platform, and simulated samples. For samples lacking explicit labels, weak labels can be generated based on the actual access trajectory after the query. After an online update is triggered, the current stable version is replaced only when the performance meets preset conditions during rolling verification; if the average response time does not improve or deteriorates after the update, meeting the rollback conditions, it is rolled back to the previous stable version. This controlled update mechanism suppresses the disruption of scheduling stability caused by online learning.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described three-dimensional geoscientific data lake warehouse indexing and scheduling method.

[0030] Compared with the prior art, the present invention has the following beneficial effects.

[0031] Firstly, this invention addresses the problem of candidate block bloat caused by directory-driven static search in existing technologies. It constructs a unified request state encompassing 3D spatial range, vertical intervals, time versions, scale requirements, hot / cold distribution, version density, and historical access statistics. Furthermore, it introduces spatial overlap, attribute matching, semantic continuity, and back-to-origin costs during the candidate block scoring stage, ensuring that candidate blocks are sorted, filtered, and sparsified before actual retrieval. Therefore, the candidate set is no longer solely determined by coarse-grained directory positioning but undergoes multi-constraint synchronous contraction before entering the underlying detailed scan. In a 3D comprehensive query scenario for mining areas, the average number of candidate blocks is reduced from 420 to 168.

[0032] Secondly, this invention addresses the problems of amplified I / O page reads and indiscriminate source retrieval of hot and cold data in cross-scale retrieval caused by fixed retrieval paths and static retrieval methods in existing technologies. It first identifies overview-type, range analysis-type, profile-continuous type, and local detailed search-type requests at the request initiation stage. Then, it selects the primary path from attribute-priority path, spatial-priority path, vertical chain-priority path, and summary preview path based on path cost. Furthermore, it adjusts the retrieval granularity of summary blocks, low-resolution blocks, or high-resolution blocks according to the request type and candidate density. Thus, the scheduling path remains consistent with the request semantics, reducing unnecessary detailed retrieval and repeated access. In multi-source cross-scale retrieval tasks, the average I / O page reads decreased from 12,500 pages to 7,400 pages, and the average response time decreased from 2.84 seconds to 1.67 seconds.

[0033] Thirdly, this invention addresses the problem in existing technologies where inconsistencies between spatial adjacency and geoscientific semantics lead to frequent repeated table lookups and significant latency fluctuations in continuous profile queries and local detailed queries. By setting a geoscientific semantic adjacency index, it incorporates stratigraphic consistency, fault correlation consistency, adjacency continuity, and version consistency into the candidate block sorting. Furthermore, in hybrid requests, a secondary decision-making mechanism following dual-path digest prefetching is employed, enabling the system to suppress erroneous main path selection in boundary scenarios where profile continuity and local detailed querying coexist. In the continuous subsurface profile query scenario, the effective block ratio increased from 58.0% to 81.0%, and the 95th percentile response time decreased from 3.90 seconds to 2.26 seconds. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments are briefly described below. The drawings are only used to illustrate technical content strongly related to the method flow of the present invention.

[0035] Figure 1 This is a flowchart of the overall method in an embodiment of the present invention; Figure 2 This is a flowchart of the access request and state construction unit in an embodiment of the present invention; Figure 3 This is a flowchart of the request type identification and joint index candidate shrinking unit in an embodiment of the present invention; Figure 4 This is a flowchart of the scheduling path allocation and differentiated data retrieval unit in an embodiment of the present invention; Figure 5 This is a flowchart of the result merging and online feedback update unit in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0037] Example 1: Overall Flow of a Three-Dimensional Geoscience Data Lakehouse Indexing and Scheduling Method

[0038] like Figures 1 to 5 As shown in the figure, the three-dimensional geoscientific data lake warehouse index scheduling method provided in this embodiment includes five collaborative stages: request access and unified state construction, request type identification and template selection, composite index candidate shrinking, scheduling path allocation and differentiated retrieval, and result return and feedback update. This method is executed by the processor of the scheduling master node, which communicates with the index service node, object storage cluster, metadata catalog, and relational attribute database through a network interface.

[0039] In this embodiment, the query monitoring period can be 1 second, which is used for refreshing the scheduling status; the main path determination for a single request is completed instantly within the current period. The execution entity can be one scheduling master node, multiple index service nodes, and one object storage cluster. The processor reads the index dictionary, scheduling parameters, and request identification model from memory, and completes the following steps within each query period.

[0040] Step S1: Requesting Access and Building a Unified State

[0041] S101, Query Parsing and Atomic Item Splitting

[0042] This sub-step is executed by the processor of the scheduling master node. The processor receives the query request submitted by the user, and the input includes at least the SQL query, the three-dimensional spatial range, the vertical interval, the time version constraint, and the return accuracy requirement. The processor decomposes the attribute predicates, projection fields, join conditions, and spatial conditions in the SQL into attribute atomic items, three-dimensional spatial atomic items, vertical atomic items, time atomic items, and scale atomic items.

[0043] Among them, attribute atomic items retain field names, comparison operators, and target values; 3D spatial atomic items are uniformly converted to 3D minimum bounding range; vertical atomic items are converted to upper and lower depth bounds; time atomic items are converted to version numbers or time periods; and scale atomic items are converted to target resolution levels.

[0044] In this embodiment, the resolution levels are defined using a discrete mapping table: high resolution is defined as a target spatial resolution of no more than 1 meter; medium resolution is defined as a target spatial resolution greater than 1 meter but no more than 10 meters; and low resolution is defined as a target spatial resolution greater than 10 meters. For voxel or profile data, the same-aperture mapping is performed based on the voxel side length or sampling interval. The mapping threshold can be obtained offline through existing data product resolution grading rules.

[0045] S102, Coordinate and Time Alignment and In-situ Missing Code

[0046] This sub-step is executed by the processor of the scheduling master node and is used to map multi-source data requests to a unified processing reference. Since different geoscientific data sources may use different coordinate systems and elevation datums, the processor calls the coordinate transformation component to uniformly map all spatial boundaries to the target 3D coordinate reference system; the time version is uniformly mapped to a logical timestamp.

[0047] Continuous numerical fields are linearly normalized using fixed offline statistics, with the normalization rule remaining consistent during both training and online phases. Category fields are encoded using dictionary encoding. When vertical boundaries are missing, the processor fills in preset null values. To maintain the stability of the state vector dimension, this embodiment uses in-situ missing value encoding instead of additionally concatenating new missing bit dimensions. That is, missing spatial, vertical, or temporal components are directly set to preset null value encoding, preferably zero-value encoding, and the relevant constraint count terms are simultaneously set to zero. When the spatial or vertical conditions themselves are missing, the processor also adds a complexity compensation amount to the connection complexity feature to reflect the increased uncertainty of subsequent paths for the request.

[0048] In one optional implementation, the preset null value encoding can be zero, and the zero value is determined according to the security placeholder rule after state normalization; the complexity compensation amount can be discretely set according to whether there is a missing key constraint, for example, the compensation amount is 1 when there is a missing constraint, and this value can be obtained by offline calibration based on historical path misjudgment samples.

[0049] S103. Construct a unified request state vector

[0050] This sub-step is executed by the processor of the scheduling master node and is used to integrate the parsing results, statistical results, and historical access status into a unified dimension input. Let the request status vector of the k-th query period be: , in, This represents the normalized value of the query statement length. This represents the normalized value of the bounding box volume in three-dimensional space. This represents the normalized value of the vertical interval length; This represents the normalized value for the length of the time range; This represents the target resolution level normalized value; Indicates the number of attribute predicates; Indicates the number of spatial constraints; Indicates the number of vertical constraints; Indicates the number of time constraints; Indicates the number of scale constraints; Indicates the percentage of heat data in the target area; This indicates the percentage of cold data; Indicates the multi-version density of the target region; This indicates the estimated connection complexity; An exponentially smoothed value representing the number of historical hit files; The exponentially smoothed value representing the historical average number of pages fetched; This represents the exponentially smoothed value of the historical average response delay.

[0051] The proportions of hot and cold data are calculated from the number of hot and cold layer hits within the statistical time window of the coarse-grained block set involved in the current request. Let the coarse-grained block set involved in the current request be... The proportion of hot data is: ,

[0052] The percentage of cold data is: , in, This indicates the number of blocks recorded by the hot storage layer within the statistical time window; Indicates the number of blocks recorded by the cold storage layer; This represents a smoothing term used to avoid a denominator of zero; in this embodiment, it can be taken as... This value is set according to the requirements of numerical stability.

[0053] Multi-version density is obtained by counting the number of valid versions of an overlay block, and is defined as: , in, Indicates the number of blocks in the coarse-grained block set; This represents the i-th data block in the set; Representation block The number of valid versions in the version directory; This represents the upper bound quantile value of the version number obtained from offline statistics. In this embodiment, the 95th percentile of the historical version number is preferred. The meaning is the same as in the previous text.

[0054] Join complexity is used to characterize the structural complexity of the current query in subsequent path selection, and is defined as: , in, Indicates the number of explicit table joins; Indicates the number of fields in the attribute predicate; The number of filter groups that combine space and attributes; This represents the complexity compensation term triggered by missing spatial or vertical conditions. It takes the value 0 when there are no missing conditions and 1 when there are missing conditions. This represents the normalized upper bound of the complexity obtained from offline statistics. In this embodiment, it can be the 99th percentile of the sum of the corresponding complexities in the requests of the most recent week. This indicates an upper limit constraint operation, making No more than 1; Used for smooth protection.

[0055] In this embodiment, when a query does not contain a certain type of constraint, the corresponding count term is set to 0; when the bounding box volume or vertical interval length is missing, the corresponding normalized value is set to a preset null value code, preferably a zero value code, and processed according to the above-mentioned in-situ missing code; if a normalized denominator is close to 0, a smoothing term is used for protection. The exponential smoothing coefficient can be 0.2, which can be determined through offline statistics of historical request volatility. After this processing, request constraints from different sources and with different structures are mapped to a unified processing scale, and the subsequent request identification model can perform stable inference under the same input dimension.

[0056] Step S2: Request Type Identification and Template Selection

[0057] S201, Execution Request Type Identification Model Reasoning

[0058] This sub-step is executed by the processor of the scheduling master node. This embodiment employs a scheduling-oriented request recognition network. Its input is a 17-dimensional state vector, the hidden layers are two fully connected layers with dimensions of 32 and 16 respectively, the activation function can be a non-linear rectified function, and the output is a 4-dimensional type probability, corresponding to overview-type requests, range analysis-type requests, profile continuity-type requests, and local detail-oriented requests. The output probability vector is denoted as: , in, This represents the probability vector of request types in the k-th query period; This represents a 4-dimensional real space; the four components of the vector correspond to the prediction probabilities of the four types of requests. Since step S102 uses in-situ missing encoding, the online input dimension of this network remains 17 dimensions.

[0059] In one alternative implementation, the training samples for the request identification model are derived from historical 3D geoscientific query logs, real scheduling logs, historical task records of the underground space survey platform, and simulated construction samples. For scenarios lacking explicit labels, weak labels can be generated based on the actual access trajectory after the query: if the final hit mainly hits low-resolution summary blocks and the user does not drill down further, it is marked as overview type; if there are continuous vertical block accesses, it is marked as profile continuity type; if the attribute filtering is strong and there are many detailed table returns, it is marked as local detailed search type.

[0060] S202. Identify the main request type and determine if it is a mixed request.

[0061] This sub-step is executed by the processor of the scheduling master node. The processor selects the request type corresponding to the highest type probability from the probability vector as the primary type, and extracts the second highest probability as an uncertainty reference. When the highest probability is lower than a preset threshold, the current request is marked as a mixed request, and parallel evaluation of at least two candidate paths is triggered; when the highest probability is not lower than the threshold, the corresponding single-path template is entered.

[0062] In this embodiment, the preset threshold can be 0.55, which can be obtained offline through historical request verification sets. The calibration target can be the minimum weighted sum of the average response time increment and the additional I / O page increment caused by misclassification. This mechanism is suitable for requests located near the boundaries of range analysis, profile continuity, and local detailing, and can reduce the probability of hard classification directly selecting the wrong main path.

[0063] S203, Generate scheduling template

[0064] This sub-step is executed by the processor of the scheduling master node. Overview requests prioritize the use of summary indexes and low-resolution blocks; range analysis requests prioritize the reduction of 3D spatial candidate blocks before attribute verification; profile continuity requests prioritize the use of vertical continuity chain indexes; local detail requests prioritize the use of attribute-first paths and perform spatial refinement.

[0065] In one alternative implementation, the template not only corresponds to different index entry orders, but also to different candidate retention limits, summary prefetching ratios, and detailed source retrieval granularities. After being mapped by this template, the request identification results can be directly applied to subsequent joint index calls and path cost evaluations.

[0066] Step S3: Shrink candidate composite index

[0067] S301. Construct and read a three-level index dictionary.

[0068] This sub-step is built offline by the index service node and read online by the scheduling master node. The first layer is a block-level 3D bounding index, recording the minimum 3D bounding box, depth range, resolution level, and version number of each data block; the second layer is a geoscientific semantic adjacency index, recording the stratigraphic number, fault association number, subject asset category, and adjacent block list of the data block; the third layer is an attribute statistics index, recording the minimum, maximum, quantile summary, and null value ratio of the main attribute fields in each block. The three-layer index is linked by a unified block identifier (block ID).

[0069] The semantic adjacency index is constructed under the following constraints: Candidate adjacency relationships are only allowed when two blocks are spatially adjacent or within a preset tolerance, have the same version number or belong to the same valid version family, and satisfy either the same stratigraphic number or adjacent fault number. For profiling tasks, adjacent blocks within a continuous band are also marked as vertically preferred adjacency edges. The preset tolerance can preferably be a certain proportion of the average thickness of adjacent blocks, which can be obtained offline based on historical profiling tasks.

[0070] S302. Calculate the comprehensive score of candidate blocks.

[0071] This sub-step is executed by the processor of the scheduling master node. For any data block The comprehensive candidate score is defined as: , in, This represents the data block in the k-th query period. The overall candidate score; , , , These represent spatial overlap weight, attribute matching weight, semantic continuity weight, and back-to-origin cost penalty weight, respectively. All four are non-negative and their weights sum to 1. Indicates the three-dimensional overlap between the requested spatial extent and the block bounding box; The score representing the consistency between the attribute predicate and the block-level statistical summary; The continuity score of the semantic chain between the block and the requested target; This represents the normalized value of the block back-to-source cost.

[0072] The three-dimensional overlap is defined as follows: , in, This represents the 3D query bounding box of the k-th request; Represents data block 3D bounding box; Represents a three-dimensional volume operator; Indicates spatial intersection; Represents the union of spaces; This represents the volume smoothing term, which can be taken as follows in this embodiment. When the requested bounding box does not intersect with the block bounding box, simply set... .

[0073] The attribute matching score is defined as: , in, Indicates the number of attribute predicates in the current request; Represents the u-th attribute predicate and data block The score is calculated based on the compatibility of the block-level statistical summaries; a score of 1 is given when the predicate target value falls within the support interval of the block's statistical summary, 0.5 is given for partial overlap, and 0 is given for incompatibility; when... When, define In this embodiment, stratigraphic numbers, fault numbers, subject asset categories, and version numbers are not included in the attribute predicate count, but are instead classified under semantic continuity constraints or version constraints, in order to maintain consistency in the statistical caliber of attribute items and semantic items.

[0074] Semantic continuity score is defined as: , in, Represents data block Consistency score with the target stratigraphic number; Represents data block Consistency score associated with the target fault; Represents data block Adjacency continuity score with the selected seed block; , , This represents the weights of the three semantic components, which are non-negative and their sum is 1.

[0075] All three scores are normalized to the range of 0 to 1: When the block stratigraphic number is completely consistent with the target stratigraphic unit, the stratigraphic consistency score is 1; when it is located in the direct upper or lower adjacent stratigraphic units of the target stratigraphic unit, it is 0.5; otherwise, it is 0. If the request includes a fault target and the block's fault association number is consistent, the fault consistency score is 1; when it matches the adjacent fault zone of the target fault, it is 0.5; otherwise, it is 0; if the request does not include a fault target, this item is 0. If the block and the seed block share a surface and the vertical depth gap does not exceed the preset tolerance, the adjacency continuity score is 1; when they only share edges or vertices and the depth gap does not exceed the tolerance, it is 0.5; otherwise, it is 0.

[0076] The cost of homing to the origin is determined by the hot layer hit rate, block size, and remote transport level. Specifically, the hot layer hit rate can be obtained by the proportion of the number of times the block was hot-hit within the most recent statistical time window to the total number of accesses to the block; the block size is normalized relative to the historical high-quantile block size; the remote transport level is estimated based on the gateway, object storage partition, and remote node levels traversed by the access path. If the local cache hits, the remote transport level can be directly treated as zero; if it crosses one remote inode or object storage partition, it is recorded as a single hop; if it crosses two or more, it is accumulated and normalized according to the upper bound of the maximum number of hops allowed by the system.

[0077] In this embodiment, the weight of the comprehensive score can be taken as follows: , , , Semantic component weights can be chosen. , , The aforementioned weights can be obtained through offline grid search calibration of historical query samples.

[0078] S303, Candidate Block Truncation and Adjacency Sparsity

[0079] This sub-step is executed by the processor of the scheduling master node. The system first sorts the candidates in descending order of their comprehensive scores and performs an initial screening based on a minimum retention threshold, retaining only candidate blocks with scores not lower than the threshold. Subsequently, a preset number of candidate data blocks are retained from the initial screening results to form a unified retention rule that combines score threshold screening with candidate truncation.

[0080] In this embodiment, the candidate score retention threshold can be 0.32, which can be obtained offline by calibrating the overlapping area of ​​the historical valid block score distribution and invalid block score distribution; the upper limit of candidate block retention can be 180, which is determined according to the system's acceptable detailed back-to-source concurrency.

[0081] To prevent the adjacency graph from degenerating into a complete graph, only block pairs that simultaneously satisfy spatial contact, version consistency, and identical stratigraphic numbers or adjacent fault numbers are considered as candidate adjacency edges. Each block retains a maximum of 6 adjacent blocks; this upper limit can be determined based on a trade-off between index graph complexity and scheduling overhead. After this processing, the candidate block set retains three-dimensional continuity while controlling the size of the scheduling graph.

[0082] Step S4: Scheduling Path Allocation and Differentiated Data Retrieval

[0083] S401. Evaluate the cost of each candidate path.

[0084] This sub-step is executed by the processor of the scheduling master node. The path cost comprehensively considers the estimated response time, estimated I / O page read volume, candidate block redundancy rate, and path uncertainty. The total cost of the m-th candidate path in the k-th query cycle is defined as: , in, Represents the total cost of the path; This represents the normalized value of the estimated response time; This represents the normalized value of the estimated I / O page reads; This represents the normalized value of the candidate block redundancy rate; This represents the cost of path uncertainty; , , , The four values ​​represent cost weights, which are non-negative and their sum is 1.

[0085] In this embodiment, the estimated response time is obtained by looking up historical similar paths and adding piecewise linear correction; the estimated I / O page read volume is also obtained by adding the historical average volume and the predicted number of scanned blocks and the predicted detailed fetch ratio; the candidate block redundancy rate is calculated by inversely from the ratio of the final number of valid blocks to the actual number of fetched or scanned blocks; the path uncertainty is determined by the degree of dispersion of request identification probability, the candidate summary missing rate and the remote source return ratio.

[0086] In one alternative implementation, path uncertainty can be determined based on a comprehensive risk value calculated from the maximum type probability, the complement of the difference between the maximum and second-highest probabilities, and the statistical summary missing rate and the proportion of remote origin requests in the candidate set. This comprehensive risk value can be obtained by averaging the summary missing rate and the remote origin request ratio. Thus, when a request is located at a type boundary, has many missing candidate summaries, and requires cross-layer remote origin requests, path uncertainty increases, thereby preventing high-risk paths from being incorrectly selected as the primary path.

[0087] In this embodiment, the path cost weight can be taken as follows: , , , The weights for the uncertain components can be chosen as follows: , , When a path has not yet been actually scanned and the number of valid blocks cannot be directly obtained, the processor uses the historical statistical mean to estimate the number of blocks; when the total number of blocks is zero, the candidate block redundancy rate is directly treated as unavailable.

[0088] S402, Select the primary path or dual paths for parallel evaluation

[0089] This sub-step is executed by the processor of the scheduling master node. Candidate paths include at least attribute-priority paths, spatial-priority paths, vertical chain-priority paths, and digest preview paths. The processor selects the path with the lowest total cost as the primary path. If the request is marked as a hybrid request in step S2, the first two lowest-cost paths are simultaneously selected to prefetch a small number of digest blocks in parallel, followed by a secondary decision based on the initial hit density.

[0090] In this embodiment, the initial digest prefetching ratio can be the top 10% of each candidate path, not exceeding 32 blocks. This ratio can be obtained offline based on the digest prefetching cost and path decision benefit. If the initial hit density difference between two paths exceeds 0.08, the path with the higher hit density is retained as the primary path; otherwise, the path with the lower total cost is selected for further execution. The difference threshold can be determined based on the offline verification results of the boundary request subset.

[0091] S403, Perform differentiated data retrieval based on request type

[0092] This sub-step is executed collaboratively by the scheduling master node and the object storage cluster. Overview requests only fetch summary blocks, statistics blocks, and low-resolution visualization blocks; range analysis requests first fetch the header summary of candidate blocks, and then perform local detail retrieval on high-scoring blocks; profile continuity requests fetch sequentially in vertically adjacent order to reduce invalid block switching caused by fault crossings; local detailed search requests perform high-resolution fetching on the target block and its first-order adjacent blocks, and simultaneously read the attribute detail table.

[0093] Thematic asset categories are used to define the thematic domain of the source object, and must include at least one of the following: stratigraphy, borehole, fault interpretation, image slice, point cloud, or attribute table. Summary block structures under different themes maintain a consistent header format for reuse within the same scheduler.

[0094] S404, Execute abnormal degradation mechanism

[0095] This sub-step is executed by the processor of the scheduling master node. If the main path estimation times out, the index service is unavailable, or the candidate block hit density is lower than a preset threshold, the system degenerates into a local incremental back-to-source mode with summary blocks first; if the spatial index sub-service fails, it degenerates into an attribute-first path and reduces the return window; if the attribute statistical summary missing rate exceeds a preset threshold, it skips attribute screening and directly uses a combination of spatial and semantic constraints to shrink the data.

[0096] In this embodiment, the candidate density threshold can be set to 0.18, and the attribute summary missing rate threshold can be set to 0.35, both of which can be obtained offline through historical query sets. The estimated timeout can be determined based on the service level agreement's (SLA) latency limit; for example, interactive requests can use a 2.5-second limit, and batch requests can use a 10-second limit, which is determined according to the actual SLA configuration.

[0097] After this degradation process, the system can still output usable results when the index service is abnormal or the summary is incomplete, thereby improving continuous operation capability.

[0098] Step S5: Result Return and Feedback Update

[0099] S501, Result Deduplication and Version Consistency Check

[0100] This sub-step is executed by the processor of the scheduling master node. The processor merges the return results of the primary and secondary paths, performs deduplication, version consistency checks, and resolution backfilling. If multiple resolution results exist for the same spatial unit, the data block that best matches the requested scale level is retained, and the remaining blocks are registered as candidate caches.

[0101] Version consistency refers to prioritizing the retention of data blocks with the same logical timestamp or belonging to the same valid version family within the same result set; if cross-version conflicts exist, the version closest to the requested time constraint is selected. This process ensures the consistency of the final output results in terms of temporal semantics.

[0102] S502. Record the true response indicators and update the historical smoothing statistics.

[0103] This sub-step is executed by the processor of the scheduling master node. The processor records the actual response time, actual I / O pages, effective block percentage, and subsequent user interactions for this query, and updates the historical smoothing state. , , This serves as the state input for the next cycle.

[0104] In this embodiment, the exponential smoothing coefficient can be set to 0.2, which is calibrated offline based on the intensity of recent request fluctuations. After this update, the system can feed back the current execution result to subsequent scheduling decisions, forming a closed loop.

[0105] S503, Perform online lightweight updates and controlled rollbacks.

[0106] This sub-step is executed by the processor of the scheduling master node. For the request identification model, incremental updates are performed on the reserved verification set only when the rolling verification accuracy of the most recent 1000 requests drops by more than 3 percentage points. The reserved verification set can be composed of 80% of the most recent 1000 requests extracted in chronological order as rolling training candidates and 20% as rolling verification samples. The triggering frequency is preferably once every 500 new requests. If the average response time does not improve or deteriorates by more than 5% after the incremental update, the system rolls back to the previous stable version.

[0107] The composite index ontology adopts a hybrid mode of offline batch update and online incremental registration: new data blocks are written to the block-level index in real time, while semantic adjacency and statistical summaries are updated in batches every hour. The 3 percentage point and 5% thresholds can be determined based on logs from the initial system deployment and are used to control rollback oscillations while maintaining classification stability.

[0108] In one alternative implementation, the request recognition model is trained using a weighted cross-entropy loss, defined as: , in, Indicates classification loss; Indicates the number of samples in the batch; Indicates the sample index; Indicates the request type category index; This represents the class weight of the c-th class sample; This represents the true label of the i-th sample in class c; Indicates the model's predicted probability; Represents the natural logarithm operation; This represents the logarithmic smoothing term, which in this embodiment can be taken as... This is used to prevent logarithmic divergence when the predicted probability is close to zero. The loss function mitigates the imbalance in the number of different request types through class weights, preventing the scheduling model from overly favoring high-frequency, common queries. The learning rate can be 0.001, the batch size can be 256, the number of training epochs can be 40, and the number of early stopping / patient epochs can be 5. These parameters can be calibrated offline based on the historical log size and convergence performance.

[0109] Example 2: Index Scheduling in a Three-Dimensional Multi-Source Integrated Retrieval Scenario in a Mining Area

[0110] A mining area data center has constructed a 3D geoscientific data lake repository, storing objects including 10-meter resolution regional digital elevation models, 0.5-meter resolution orthophotos, borehole columnar data, stratigraphic entity models, fault interpretation results, and historical exploration attribute tables. Although the existing system has established a unified metadata catalog, when users simultaneously submit composite requests such as target stratigraphic ranges, borehole grade thresholds, specific depth intervals, and high-resolution local detailed surveys, it often first scans an excessively large set of candidate files and then repeatedly checks the tables for verification. This results in long response times, high I / O page read volumes, and frequent switching between hot and cold object storage.

[0111] This embodiment uses the method described in Embodiment 1. The equipment scale consists of 1 scheduling master node, 3 index service nodes, and 1 object storage cluster. The query monitoring cycle is 1 second, and the scheduling decision cycle is the same, also 1 second. The total number of block-level data is 120,000, of which 35% are hot storage blocks and 65% are cold storage blocks. The unified parameter dictionary includes: 17-dimensional request state dimension; 32 and 16 dimensions of the hidden layer of the request identification network; learning rate 0.001; batch size 256; 40 training rounds; upper limit of candidate block retention 180; candidate score threshold 0.32; candidate density threshold 0.18; attribute summary missing rate threshold 0.35; path cost weights of 0.35, 0.30, 0.20, and 0.15; and uncertainty component weights of 0.40, 0.30, and 0.30. All parameters can be obtained offline through historical mining area scheduling logs.

[0112] This embodiment uses a single actual processing cycle for illustration. The original input request is: to query a mining area within its eastern boundary, at a depth range of 120 to 260 meters, with a target stratum number L3, a copper grade greater than 0.8%, a high target resolution level, and the most recent time version. In step S101, the processor parses and obtains: 1 attribute predicate, 1 spatial constraint, 1 vertical constraint, 1 temporal constraint, and 1 scale constraint; among which, the target stratum number L3 is included in the semantic target constraint and is not counted in the attribute predicate count.

[0113] After completing coordinate unification and normalization in step S102, the following state vector components are formed: , , , , , , , , , , , , , , , , The hot and cold data ratio is calculated based on the hot and cold layer hit statistics of the requested coarse block set in the most recent 24 hours; the multi-version density is calculated based on the average effective version density of the overlay blocks; and the join complexity is obtained based on a single table join, a single attribute predicate, a set of spatial and attribute joint filtering, and normalization with no missing compensation term. Therefore, the model input dimension is 1 x 17.

[0114] The request identifies the network outputting a 4-dimensional probability vector. These correspond to overview type, range analysis type, continuous profile type, and local detailed investigation type, respectively. Since the maximum probability of 0.56 is higher than 0.55, the processor identifies it as a local detailed investigation type request and selects the attribute priority path and performs spatial refinement of the template.

[0115] The processor then performs a joint scoring on the candidate blocks. Let a certain candidate block... The overlap between the 3D bounding box and the requested bounding box is 0.72. The attribute statistical summary of this block meets the copper grade screening, resulting in an attribute matching score of 1.00. This block has the same stratigraphic number as the target stratigraphic unit and is adjacent to the selected seed block but has no fault hits, therefore its semantic continuity score is 0.80. The back-to-source cost of this block is 0.237. According to the comprehensive scoring rules in Example 1 and the parameters of this example, the comprehensive candidate score of this block is approximately 0.696, and the processor accordingly retains this block in the high-priority candidate set.

[0116] After candidate sorting, an initial screening was performed using a threshold of 0.32, followed by truncation according to the candidate upper limit rule, resulting in 168 candidate blocks. This is below the retention limit of 180, so a second truncation is unnecessary. The processor then calculates the total cost of the four candidate paths. For the attribute-first path, based on the statistics of the most recent similar path windows, the current predicted number of scanned blocks, the predicted detailed retrieval ratio, and the predicted remote source return ratio, the total cost of this path, after normalization, is approximately 0.33. After calculation using the same criteria, the total cost of the spatial-first path is approximately 0.49, the vertical chain-first path is approximately 0.43, and the summary preview path is approximately 0.58. Therefore, the attribute-first path is selected as the primary path.

[0117] In the differentiated fetching phase, the processor first fetches the block-level header summaries of 168 candidate blocks, of which 112 blocks pass attribute and spatial verification and proceed to the second-level fetch. Then, only the top 64 blocks are subjected to high-resolution detailed source fetching, and the corresponding attribute table fragments are fetched simultaneously. If executed using the static data fetching method, all 420 candidate blocks would undergo at least one detailed-level access. After this processing, invalid detailed accesses are significantly reduced, and the number of hot / cold layer switching operations is also decreased.

[0118] The test environment for this embodiment is a 3D geoscientific lake warehouse digital twin test platform, overlaid with replay data from real mining area scheduling logs. The test lasted 8 hours, processing a total of 12,000 mixed queries, of which 8,000 came from real historical log replays and 4,000 were perturbation queries generated according to the real distribution. Under normal operating conditions, the hot storage hit rate remained stable at around 35%; under impact conditions, 30 minutes of concentrated access to cold data and large-scale multi-user concurrent distribution were injected. The control scheme uses a unified table directory plus ordinary attribute filtering plus static data retrieval; the present invention uses request identification plus composite indexes plus differentiated retrieval. The statistical results are as follows: the average number of candidate blocks in the control scheme is 420, while in the present invention it is 168; the average I / O page read volume in the control scheme is 12,500 pages, while in the present invention it is 7,400 pages; the average response time in the control scheme is 2.84 seconds, while in the present invention it is 1.67 seconds. These results indicate that the present invention is suitable for multi-source 3D detailed exploration scenarios in mining areas.

[0119] Subsequently, based on the above calculation and processing results, the processor continues to execute the subsequent steps as described in Example 1, completing operations such as result merging, historical state updating, controlled incremental model updating, and rollback judgment. The specific execution logic is the same as in Example 1, and will not be repeated here.

[0120] Example 3: Index Path Scheduling in Continuous Subsurface Profile Analysis Scenarios

[0121] A city's underground space survey platform needs to frequently perform continuous profile queries. A typical task is to output strata, underground pipeline conflict information, and historical borehole comparisons within a depth range of 0 to 80 meters along a planned route. Existing systems based on ordinary spatial indexes often add spatially similar blocks that are not from the same stratigraphic sequence to the candidate list during continuous profile queries, and then repeatedly filter them when outputting the profile map. This results in a high 95th percentile response time, causing noticeable lag when users drag the profile lines.

[0122] This embodiment uses the same parameter dictionary and model version as Embodiment 2, but the data organization in the scenario adopts a finer vertical block strategy. The device scale is 1 scheduling master node, 2 index service nodes and 1 profile cache node, and the scheduling decision cycle is still 1 second.

[0123] In a given query cycle, the original input is a 3.2 km long profile line with a 50 m width buffer, a depth range of 0 to 80 m, a continuous stratigraphic profile as the target subject, and a medium target resolution. After parsing, the main components of the state vector are: , , , A preset null value encoding is adopted, wherein the preset null value encoding is preferably a zero value encoding. , , , , , , , , , , , , Among them, the one attribute predicate corresponds to the underground pipeline conflict type filtering, and the stratigraphic continuous profile enters the topic and semantic target constraints.

[0124] The request identifies the network output probability vector as follows: The processor identifies it as a profile continuum request and selects the vertical chain priority template. Let a certain block... The spatial overlap with the requested profile is 0.66, the attribute matching score is 0.50, and the semantic continuity score shows a high stratigraphic consistency score, a medium fault association score, and a high adjacency continuity score. According to the scoring rules in Example 1, the semantic continuity score of this block is 0.92; its back-to-source cost is 0.31, so the overall candidate score is approximately 0.579. In contrast, another block that is spatially close but stratigraphically discontinuous, although having a high spatial overlap, is excluded early because its overall score is below the retention threshold of 0.32 due to insufficient stratigraphic consistency and adjacency continuity.

[0125] Path cost evaluation results show that the total cost of the attribute-first path is approximately 0.47, the spatial-first path is approximately 0.41, the vertical chain-first path is approximately 0.29, and the summary preview path is approximately 0.45. Therefore, the processor selects the vertical chain-first path and pulls block summaries in depth-sequential order, then performs medium-resolution detailed retrieval on consecutive hit segments. When the statistical summary missing rate of a local block exceeds 0.35, the system skips attribute screening and directly relies on the spatial and semantic paths, thus avoiding blocking profile interaction due to incomplete summaries.

[0126] The test environment in this embodiment simulates a user dragging a profile line for continuous queries, lasting 4 hours with a total of 7200 requests, a maximum concurrency of 12 requests per second. Data sources consist of replays of historical task logs from a real underground space platform and real-time disturbance requests. Normal and impact conditions were set: under normal conditions, the profile line moved at a stable speed; under impact conditions, it was dragged continuously and rapidly, crossing areas with dense fault zones. The control scheme uses a single-dimensional spatial index with unified source lookup; the present invention uses profile type recognition with semantic continuous indexing and vertical chain-priority retrieval. Statistical results are as follows: the effective block ratio of the control scheme is 58.0%, while that of the present invention is 81.0%; the 95th percentile response time of the control scheme is 3.90 seconds, while that of the present invention is 2.26 seconds; the average number of repeated table lookups is 6.8 times / query for the control scheme, and 3.1 times / query for the present invention. These results demonstrate that the present invention has higher stability and lower interaction jitter in continuous profile analysis scenarios.

[0127] Subsequently, based on the above calculation and processing results, the processor continues to execute the subsequent steps as described in Example 1, completing operations such as result merging, abnormal degradation processing, historical state updating, and model version stability verification. Its specific execution logic is consistent with Example 1, and will not be repeated here.

[0128] Example 4: Electronic Equipment

[0129] This embodiment provides an electronic device, including a memory, a processor, a communication interface, and a bus.

[0130] The system includes a memory for storing computer programs, an index dictionary, scheduling parameters, historical statistics, and request identification model parameters; a processor for calling the computer programs in the memory; a communication interface for interacting with object storage, metadata catalogs, relational attribute libraries, and index service nodes; and a bus for connecting the memory, processor, and communication interface to enable data transmission.

[0131] When the processor executes the computer program, it implements the steps of a three-dimensional geoscientific data lakehouse index scheduling method as described in any one of Embodiments 1 to 3, including: receiving a three-dimensional geoscientific query request, constructing a unified request state, identifying the request type, calling the composite index to perform candidate block shrinking, evaluating the cost of each candidate path, determining the main path and data retrieval granularity, retrieving the target data block, returning the retrieval results, and updating historical statistics and model versions based on actual access results. Furthermore, when the processor executes the computer program, it can also implement a request state construction module, a request identification module, a composite index candidate shrinking module, a path allocation module, and a feedback update module, respectively, to enhance the correspondence between the method steps and the functional modules of the electronic device.

[0132] In one optional implementation, the electronic device can be deployed on a geoscience data center scheduling master node, or on an edge computing node or cloud scheduling node with index scheduling capabilities. The memory may include at least one of random access memory, read-only memory, flash memory, or disk array; the communication interface may include at least one of an Ethernet interface, a Fibre Channel interface, or other network communication interfaces.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the principles and spirit of the present invention should be included within the scope of protection of the present invention. Terms such as "first" and "second" are used only to distinguish entities or operations and do not indicate an actual order or hierarchy; terms such as "including" and "comprising" are open-ended and do not exclude other elements not explicitly listed.

Claims

1. A method for indexing and scheduling three-dimensional geoscientific data lake warehouses, characterized in that, include: The system receives 3D geoscience query requests, parses them to obtain attribute constraints, 3D spatial range, vertical interval, time version constraints and scale requirements, and performs unified reference alignment on the 3D spatial range and time version constraints to construct a request status that includes query semantic features, spatial range features, vertical features, scale features, hot and cold distribution features, version density features, connection complexity features and historical access smoothing statistics. The request type is identified based on the request status, and a corresponding scheduling template is generated based on the identification result. When the uncertainty of the request type identification meets the preset conditions, a parallel candidate path template corresponding to the hybrid request is generated. The block-level 3D bounding index, geoscientific semantic adjacency index, and attribute statistics index associated with the unified block identifier are invoked to calculate a comprehensive candidate score for the candidate data block. The comprehensive candidate score is jointly determined by the spatial overlap relationship between the requested range and the block bounding information, the matching relationship between the attribute constraints and the block-level statistical summary, the geoscientific semantic adjacency relationship, and the block back-to-source cost. Based on the comprehensive candidate score, threshold filtering, sorting, candidate truncation, and adjacency sparsification are performed to obtain a set of candidate blocks. The path cost is calculated for attribute-priority path, spatial-priority path, vertical chain-priority path and summary preview path respectively. The main path is selected based on the path cost, and at least two candidate paths are evaluated in parallel when the requested recognition result meets the preset uncertainty condition. The data granularity is determined based on the selected path, candidate block set, request type, candidate density, and hot / cold distribution. Retrieve target data blocks at a defined data granularity and return retrieval results. Record real response metrics and actual access results. Update the historical access smoothing statistics and index statistics. Roll back to the previous stable version when performance degradation meets the rollback conditions after online model update.

2. The three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 1, characterized in that, The build request status includes: The query request is broken down into attribute atomic items, 3D space atomic items, vertical atomic items, time atomic items, and scale atomic items; Convert 3D spatial atomic terms into 3D minimum bounding range, vertical atomic terms into upper and lower depth bounds, time atomic terms into logical timestamps, and scale atomic terms into target resolution levels. Based on the query length, three-dimensional bounding volume, vertical interval length, time range length, target resolution level, number of various constraints, ratio of hot and cold data, version density, connection complexity, and smoothed statistics of historical hit files, historical page fetches, and historical response latency, a unified request status is generated.

3. The three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 2, characterized in that, The build request status also includes in-situ missing encoding, which includes: When any one of the spatial range feature, vertical feature, or temporal feature is missing, the corresponding state component is set to a preset null value code, which is preferably a zero value code. Set the constraint count corresponding to the missing item to zero; The complexity compensation amount corresponding to the missing item is superimposed on the connection complexity feature to reflect the uncertainty of subsequent path selection; Keep the input dimensions of the request state constant without adding any additional input dimensions.

4. The three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 1, characterized in that, The request type identification based on request status includes: The request status is input into the request type identification model, which outputs the type probability of the corresponding overview request, range analysis request, profile continuous request, and local detailed investigation request. Select the request type corresponding to the highest type probability as the primary type; When the maximum type probability is lower than a preset threshold, the current request is marked as a mixed request, and parallel evaluation of at least two candidate paths is triggered; When the probability of the maximum type is not lower than a preset threshold, a single-path scheduling template is generated based on the main type.

5. The three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 1, characterized in that, The geoscientific semantic adjacency index records at least one of the following: stratigraphic number, fault association number, subject asset category, version family tag, and adjacency block relationship, and the candidate adjacency relationships in the geoscientific semantic adjacency index are established according to the following rules: Candidate blocks must satisfy spatial contact relationships or have spatial distances that do not exceed preset tolerances; Candidate blocks either satisfy version consistency constraints or belong to the same valid version family; Candidate blocks satisfy at least one semantic constraint, namely, consistent stratigraphic numbering and adjacent fault numbering. When the request is a continuous profile request, the vertically adjacent blocks within the continuous band are marked as vertical priority adjacent edges.

6. The three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 5, characterized in that, The comprehensive candidate score is generated by at least spatial overlap, attribute matching degree, semantic continuity, and back-to-origin cost, wherein: The spatial overlap is determined by the proportion of the intersection volume of the query enclosing range and the data block enclosing range to the total volume of the union. The attribute matching degree is determined by the compatibility between the attribute predicate and the block-level statistical summary; The semantic continuity is determined by at least two of the following: stratigraphic consistency score, fault association consistency score, and adjacency continuity score; The cost of returning to the source is determined by the hotspot hit rate of the data block, the block size, and the remote transmission level.

7. A three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 6, characterized in that, The process of performing candidate shrinkage based on the comprehensive candidate score includes: Sort the candidate data blocks in descending order according to their comprehensive candidate scores; Retain candidate data blocks whose overall candidate score is not lower than a preset retention threshold; Retain the first preset number of candidate data blocks from the candidate data blocks after threshold filtering; Adjacency edges are established only for candidate block pairs that meet the requirements of spatial contact, consistent version, and identical stratigraphic number or adjacent fault number, and the number of adjacent blocks retained for each candidate data block is limited to achieve adjacency sparsity.

8. A three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 1, characterized in that, The calculation of path cost and selection of the primary path includes: Calculate the estimated response time, estimated I / O page read volume, candidate block redundancy rate, and path uncertainty for each candidate path. The path cost of each candidate path is generated based on the estimated response time, estimated I / O page read volume, candidate block redundancy rate, and path uncertainty. Select the candidate path with the lowest path cost as the primary path; When the current request is a mixed request, select at least two candidate paths with the lowest path cost to perform the first digest prefetch, and perform a second decision based on the hit density and path cost of the first digest prefetch.

9. A three-dimensional geoscientific data lake warehouse indexing and scheduling method according to claim 1, characterized in that, The process of retrieving the target data block at a determined data retrieval granularity includes: When the request type is an overview request, at least one of the following is fetched: summary block, statistics block, and low-resolution block; when the request type is a range analysis request, the header summary of the candidate data block is fetched first, and then detailed source fetch is performed on the high-scoring candidate data block. When the request type is a profile continuous request, the data blocks corresponding to the consecutive hit segments are pulled serially in the vertical adjacent order. When the request type is a local detailed query request, a high-resolution fetch is performed on the target data block and its first-order adjacent blocks, and attribute details are fetched simultaneously. When the main path prediction times out, the index service is unavailable, or the candidate hit density is lower than a preset threshold, it degenerates into a local incremental back-to-source mode with the summary block first. When the missing rate of attribute statistical summaries exceeds a preset threshold, attributes are skipped for initial screening, and spatial and semantic constraints are combined to shrink the data.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a three-dimensional geoscientific data lake warehouse indexing and scheduling method as described in any one of claims 1-9.

Images