Multi-modal low-altitude grid data dynamic indexing method and device for improving caching mechanism

Abstract

The embodiment of the invention provides a multi-modal low-altitude grid data dynamic indexing method and device for improving a cache mechanism. The method comprises the following steps: mapping multi-modal low-altitude grid data to a unified vector space, and constructing a cross-modal semantic association hybrid index for the mapped low-altitude grid data; dividing the mapped low-altitude grid data into a plurality of dynamic sub-regions according to data distribution density and data access popularity, and managing a mixed index corresponding to each dynamic sub-region by combining a Loki log index fragmentation mechanism and tagged metadata; the mapped low-altitude grid data is cached by adopting an EHCD hierarchical multi-level cache architecture; and carrying out cross-modal low-altitude grid data retrieval by combining SIMD instruction set parallelization vector similarity calculation and GPU acceleration complex space relation operation. In this way, the real-time retrieval efficiency, the system expansibility and the energy efficiency ratio of the low-altitude grid data can be remarkably improved.

Description

Technical Field

[0001] This disclosure relates to the field of low-altitude grid data processing technology, and in particular to a method and apparatus for dynamic indexing of multimodal low-altitude grid data with an improved caching mechanism. Background Technology

[0002] In low-altitude intelligent application scenarios (such as drone logistics, urban low-altitude airspace management, and low-altitude cultural tourism services), the real-time acquisition, efficient management, and intelligent retrieval of multimodal low-altitude grid data are core technological challenges for achieving a closed-loop business model. With the rapid development of the low-altitude economy, the scale of low-altitude grid data is growing exponentially. Its multi-source heterogeneity (such as remote sensing imagery, drone aerial video, and sensor time-series data), high dynamism (such as real-time aircraft position changes and environmental status updates), and multimodal correlation (such as spatial coordinates, semantic descriptions, and physical attributes) pose severe challenges to traditional data indexing and caching mechanisms. Currently, improving the real-time retrieval efficiency, system scalability, and energy efficiency of low-altitude grid data has become an urgent technical problem to be solved. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide an improved caching mechanism for dynamic indexing of multimodal low-altitude grid data, the method comprising:

[0004] Multimodal low-altitude grid data is mapped to a unified vector space, and a hybrid index with cross-modal semantic association is constructed for the mapped low-altitude grid data;

[0005] The mapped low-altitude grid data is divided into multiple dynamic sub-regions according to data distribution density and data access frequency, and the hybrid index corresponding to each dynamic sub-region is managed by combining the Loki log index sharding mechanism and tagged metadata.

[0006] The Elastic Hierarchical Cache Design (EHCD) hierarchical multi-level caching architecture is used to cache the mapped low-altitude grid data.

[0007] We combine parallelized vector similarity computation based on the Single Instruction Multiple Data (SIMD) instruction set with GPU-accelerated computation of complex spatial relationships to perform cross-modal low-altitude grid data retrieval.

[0008] In some possible implementations of the first aspect, multimodal low-altitude grid data is mapped to a unified vector space, and a hybrid index with cross-modal semantic associations is constructed for the mapped low-altitude grid data, including:

[0009] Feature extraction of each modal low-altitude grid data is performed using the encoder corresponding to each modal low-altitude grid data;

[0010] The extracted low-altitude grid data features of each modality are aligned in a unified vector space, and a dynamic routing mechanism is used to optimize feature aggregation.

[0011] A hybrid index for cross-modal semantic association is constructed based on a combined scheme of product quantization and graph indexing for aggregated low-altitude grid data features.

[0012] In some possible implementations of the first aspect, the mapped low-altitude grid data is divided into multiple dynamic sub-regions according to data distribution density and data access frequency, and a hybrid index corresponding to each dynamic sub-region is managed using the Loki log index sharding mechanism and tagged metadata, including:

[0013] Construct a dynamic grid partitioning strategy, whereby the dynamic grid partitioning strategy is characterized by ensuring that each dynamic sub-region satisfies the following condition:

[0014] Density sensitive: High data density regions are divided into finer-grained segments;

[0015] Popularity sensitive: High-frequency access areas retain lower index levels;

[0016] Dynamic balancing: Automatically adjusts partition boundaries based on real-time statistics;

[0017] The data distribution density is constructed by kernel density estimation, the data access popularity is updated by exponential smoothing, and a joint optimization objective function is constructed by combining the data distribution density and the data access popularity.

[0018] Based on the dynamic grid partitioning strategy and the joint optimization objective function, the mapped low-altitude grid data is divided into multiple dynamic sub-regions;

[0019] Based on the Loki log storage engine, we have made in-depth optimizations and achieved automatic tag extraction and encoding by rewriting its hybrid index building process. On the write path, we use the log structure merge tree, also known as the LSM tree structure, to batch process metadata updates. On the read path, we build a multi-layer caching system to intelligently prefetch the search results of hot tags.

[0020] Among the possible implementations of the first aspect, an EHCD hierarchical multi-level caching architecture is used to cache the mapped low-altitude grid data, including:

[0021] When caching the mapped low-altitude grid data, an EHCD hierarchical multi-level caching architecture is adopted, which divides the caching resources into three levels: on-chip cache, distributed memory cache, and cloud storage. Each level forms a vertical storage topology through an adaptive data migration mechanism, realizing the layered collaboration of on-chip cache, distributed memory cache, and cloud storage. Specifically, the on-chip cache is used to store hot low-altitude grid data, the distributed memory cache is used to store recently accessed low-altitude grid data, and the cloud storage is used to store cold low-altitude grid data.

[0022] Among the possible implementations of the first aspect, cross-modal low-altitude grid data retrieval combines SIMD instruction set parallelized vector similarity computation with GPU-accelerated complex spatial relationship operations, including:

[0023] In the vector similarity calculation layer, a memory-aligned batch memory access mode is constructed to address the high-dimensionality of cross-modal embedded vectors. Specifically, the cosine similarity calculation is decomposed into a parallel multiply-accumulate operation chain using the 512-bit wide vector register of the AVX-512 instruction set. Memory latency is hidden through loop unrolling and instruction reordering techniques, while mask registers are used to avoid redundant calculations of unaligned data. A dynamic task scheduler is used to achieve collaborative optimization of hardware resources. A computational cost prediction model is established to automatically select the execution path based on data dimension and index density: CPU-side SIMD acceleration is used for low-dimensional dense vectors, while high-dimensional sparse features are processed by the GPU. A static scheduling strategy is used for deterministic computation tasks, while a dynamic work-stealing queue is used for non-deterministic tasks.

[0024] Among some possible implementations of the first aspect, the method also includes:

[0025] The hybrid index is adaptively adjusted by dynamic index optimization. Drawing on the stream processing mechanism of time series databases, an incremental hybrid index is built for newly added multimodal low-altitude grid data, and invalid search ranges are filtered out by Bloom filter.

[0026] Among the possible implementations of the first aspect, adaptive adjustment of the hybrid index is achieved through dynamic index optimization. Drawing on the stream processing mechanism of time-series databases, an incremental hybrid index is constructed for newly added multimodal low-altitude grid data, including:

[0027] The architecture employs a windowed stream processing engine, automatically triggering the extraction of spatiotemporal grid metadata features when low-altitude grid data is written. It dynamically divides grid cells into hot and cold zones by statistically analyzing data arrival rate and spatial coverage using a sliding time window. For frequently updated hot zones, a hierarchical index is constructed using an LSM tree structure, temporarily storing real-time data in a memory table and performing merging optimization when batch-flushd to disk. Cold zones utilize a compression encoding strategy, transforming discrete data points into continuous interval representations on a space-filling curve.

[0028] A dual feedback adjustment mechanism is introduced: on the one hand, based on the analysis of historical retrieval patterns, a correlation model between retrieval popularity and index granularity is established to automatically improve the index resolution for frequently accessed dynamic sub-regions; on the other hand, an online learning algorithm is used to predict the evolution trend of low-altitude grid data distribution and inject a forward-looking offset into the grid splitting threshold.

[0029] Among the possible implementations of the first aspect, filtering invalid search ranges using a Bloom filter includes:

[0030] An evolvable Bloom filter is deployed at the retrieval routing layer. This Bloom filter dynamically adjusts the number of hash functions and the length of bit vectors based on the real-time distribution of low-altitude grid data. When the false positive rate of a specific grid cell is detected to be rising, the filter parameters are automatically reconfigured: the false positive probability is reduced by increasing the number of hash functions, or the space efficiency is maintained by extending the length of bit vectors.

[0031] Among some possible implementations of the first aspect, the method also includes:

[0032] By combining time-series databases with AI, a lightweight model is trained to predict data access patterns, and based on this, hotspot low-altitude grid data is prefetched into the on-chip cache.

[0033] Secondly, embodiments of this disclosure provide an improved caching mechanism for a dynamic indexing device for multimodal low-altitude grid data, the device comprising:

[0034] The mapping module is used to map multimodal low-altitude grid data to a unified vector space and to build a hybrid index for cross-modal semantic association of the mapped low-altitude grid data;

[0035] The partitioning module is used to divide the mapped low-altitude grid data into multiple dynamic sub-regions according to the data distribution density and data access frequency, and combines the Loki log index sharding mechanism and tagged metadata management to manage the hybrid index corresponding to each dynamic sub-region;

[0036] The caching module is used to cache the mapped low-altitude grid data using an EHCD hierarchical multi-level caching architecture;

[0037] The retrieval module is used to perform cross-modal low-altitude grid data retrieval by combining parallelized vector similarity calculations based on the SIMD instruction set with GPU-accelerated computation of complex spatial relationships.

[0038] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0039] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0040] Compared with the prior art, this disclosure has at least the following technical effects:

[0041] It can form a full-link optimization system from multimodal fusion representation, dynamic index optimization, intelligent caching to heterogeneous computing power acceleration, which can significantly improve the real-time retrieval efficiency, system scalability and energy efficiency of low-altitude grid data.

[0042] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0044] Figure 1 A flowchart is shown for a dynamic indexing method for multimodal low-altitude grid data with an improved caching mechanism provided by an embodiment of this disclosure;

[0045] Figure 2 A structural diagram of a multimodal low-altitude grid data dynamic indexing device with an improved caching mechanism provided by an embodiment of the present disclosure is shown.

[0046] Figure 3 A structural diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0048] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0049] To address the problems in the background art, embodiments of this disclosure provide a method, apparatus, device, and storage medium for dynamic indexing of multimodal low-altitude grid data with an improved caching mechanism. This can form a full-link optimization system from multimodal fusion characterization, dynamic index tuning, intelligent caching to heterogeneous computing acceleration, significantly improving the real-time retrieval efficiency, system scalability, and energy efficiency of low-altitude grid data.

[0050] The following description, in conjunction with the accompanying drawings, details an improved caching mechanism for dynamic indexing of multimodal low-altitude grid data, a method, apparatus, device, and storage medium provided by the present disclosure through specific embodiments.

[0051] Figure 1 A flowchart illustrating an improved caching mechanism for dynamic indexing of multimodal low-altitude grid data, as provided in an embodiment of this disclosure, is shown. Figure 1 As shown, method 100 may include the following steps:

[0052] S110 maps multimodal low-altitude grid data to a unified vector space and constructs a hybrid index for cross-modal semantic association of the mapped low-altitude grid data.

[0053] In some embodiments, an encoder corresponding to each modality of low-altitude grid data can be used to extract features from each modality of low-altitude grid data. Then, the extracted features of each modality of low-altitude grid data are aligned in a unified vector space, and a dynamic routing mechanism is used to optimize feature aggregation. Finally, a hybrid index for cross-modal semantic association is constructed for the aggregated low-altitude grid data features based on a joint scheme of product quantization and graph indexing.

[0054] S120 divides the mapped low-altitude grid data into multiple dynamic sub-regions according to data distribution density and data access frequency, and combines the Loki log index sharding mechanism and tagged metadata management to manage the hybrid index corresponding to each dynamic sub-region.

[0055] In some embodiments, a dynamic mesh partitioning strategy can be constructed first, wherein the dynamic mesh partitioning strategy is characterized by ensuring that each dynamic sub-region satisfies the following condition:

[0056] Density sensitive: High data density areas (i.e., areas where the data distribution density is greater than a preset threshold) are divided into more granular sections;

[0057] Popularity sensitive: High-access-frequency areas (i.e., areas where data access frequency exceeds a preset threshold) retain lower index levels;

[0058] Dynamic balancing: Automatically adjusts partition boundaries based on real-time statistics.

[0059] Then, the data distribution density is constructed by kernel density estimation, the data access popularity is updated by exponential smoothing, and a joint optimization objective function is constructed by combining the data distribution density and the data access popularity.

[0060] Then, based on the dynamic grid partitioning strategy and the joint optimization objective function, the mapped low-altitude grid data is divided into multiple dynamic sub-regions.

[0061] Ultimately, deep optimizations were made based on the Loki log storage engine. By rewriting its hybrid index building process, automatic extraction and encoding of tags were achieved. On the write path, an LSM tree structure was used to batch process metadata updates, and on the read path, a multi-layer caching system was built to intelligently prefetch the search results of hot tags.

[0062] The S130 uses an EHCD hierarchical multi-level caching architecture to cache the mapped low-altitude grid data.

[0063] In some embodiments, when caching the mapped low-altitude grid data, an EHCD hierarchical multi-level caching architecture can be adopted, dividing the caching resources into three levels: on-chip cache, distributed memory cache, and cloud storage. Each level forms a vertical storage topology through an adaptive data migration mechanism, realizing layered collaboration between on-chip cache, distributed memory cache, and cloud storage. Specifically, the on-chip cache is used to store hot low-altitude grid data, the distributed memory cache is used to store recently accessed low-altitude grid data, and cloud storage is used to store cold low-altitude grid data (also known as S3 cold data).

[0064] S140 combines parallelized vector similarity calculation using the SIMD instruction set with GPU-accelerated computation of complex spatial relationships to perform cross-modal low-altitude grid data retrieval.

[0065] In some embodiments, a memory-aligned batch memory access mode can be constructed at the vector similarity calculation layer to address the high-dimensionality of cross-modal embedded vectors. Specifically, the cosine similarity calculation is decomposed into a parallel multiply-accumulate operation chain using the 512-bit wide vector register of the AVX-512 instruction set. Memory latency is hidden through loop unrolling and instruction rearrangement techniques, while mask registers are used to avoid redundant calculations of unaligned data. Cooperative optimization of hardware resources is achieved through a dynamic task scheduler. A computational cost prediction model is established to automatically select the execution path based on data dimension and index density: CPU-side SIMD acceleration is used for low-dimensional dense vectors, while high-dimensional sparse features are processed by the GPU. Static scheduling strategies are used for deterministic computation tasks, while dynamic work-stealing queues are employed for non-deterministic tasks.

[0066] It is worth noting that, in addition to the steps described above, method 100 may also include the following steps:

[0067] The hybrid index is adaptively adjusted by dynamic index optimization. Drawing on the stream processing mechanism of time series databases, an incremental hybrid index is built for newly added multimodal low-altitude grid data, and invalid search ranges are filtered out by Bloom filter.

[0068] Specifically, a windowed stream processing engine can be used as the basic architecture, automatically triggering the extraction of spatiotemporal grid metadata features when low-altitude grid data is written. By using a sliding time window to statistically analyze the data arrival rate and spatial coverage of grid cells, hot and cold grids are dynamically divided. For frequently updated hot grids, a hierarchical index is constructed using an LSM tree structure, temporarily storing real-time data in an in-memory table, and performing merging optimization when batch-flushd to the disk layer. Cold grids employ a compression encoding strategy, transforming discrete data points into continuous interval representations on a space-filling curve.

[0069] A dual feedback adjustment mechanism is introduced: on the one hand, based on the analysis of historical retrieval patterns, a correlation model between retrieval popularity and index granularity is established to automatically improve the index resolution for frequently accessed dynamic sub-regions; on the other hand, an online learning algorithm is used to predict the evolution trend of low-altitude grid data distribution and inject a forward-looking offset into the grid splitting threshold.

[0070] An evolvable Bloom filter is deployed at the retrieval routing layer. This filter dynamically adjusts the number of hash functions and the bit vector length based on the real-time distribution of low-altitude grid data. When an increase in the false positive rate of a specific grid cell is detected, the filter parameters are automatically reconfigured: either by increasing the number of hash functions to reduce the false positive probability, or by extending the bit vector length to maintain space efficiency.

[0071] As an example, method 100 may also include the following steps:

[0072] By combining time-series databases with AI, a lightweight model is trained to predict data access patterns, and based on this, hotspot low-altitude grid data is prefetched into the on-chip cache.

[0073] To facilitate further understanding, the above steps will be described in detail below with reference to specific embodiments:

[0074] (1) Firstly, multimodal low-altitude grid data includes text, video, and image formats, and the content mainly covers aircraft information, pilot information, geographic information, airspace information, and meteorological information. The specific classification of multi-source heterogeneous data formats is shown in Table 1.

[0075] Table 1 Data Format Classification

[0076]

[0077] The data, after being organized according to content, are shown in Tables 2-8.

[0078] Table 2 Aircraft Information

[0079]

[0080] Table 3 Pilot Information

[0081]

[0082] Table 4 Geographic Information

[0083]

[0084]

[0085] Table 5 Airspace Information

[0086]

[0087] Table 6 Meteorological Information

[0088]

[0089] Table 7 Business Operation Data

[0090]

[0091] Table 8. Perception and Interaction Data

[0092]

[0093] The above describes the data format and content of multimodal low-altitude grid data. When processing this data, it is necessary to solve the semantic alignment problem of image-text-spatial data (such as the association between aerial images and airspace control text descriptions), and to meet certain real-time requirements.

[0094] (2) Next, based on the multimodal large model, multimodal heterogeneous low-altitude grid data such as text, images, and videos are mapped to a unified vector space, and a hybrid index with cross-modal semantic association is constructed for the mapped low-altitude grid data. The hybrid index can be used for efficient cross-modal retrieval and realize the unified representation of cross-modal features.

[0095] This stage primarily employs a phased, multi-task learning framework to achieve unified representation and efficient retrieval of cross-modal features. The specific steps are divided into multimodal joint embedding space construction and efficient retrieval system optimization. In the multimodal joint embedding space construction stage, a modal encoder is designed. The text modal information is processed using the Transformer-XL encoder, and the input sequence is denoted as T = {t1,…,t}. n The output feature is H. t =Transformer(W t T ·Embedding(T)+P), where W t ∈R d×d The learnable projection matrix is ​​P, where P is the positional encoding. Visual modal information is decomposed into a block sequence V = {v1,…,v...} using a Vision Transformer. m}, extract features as H v =MLP(Pooling(ViT(W) v T ·V))), where W v Let W be the parameters to be trained, and v be the pixel size of the image patch or video. Then, the extracted modal features are aligned to a unified vector space, and a dynamic routing mechanism is used to optimize feature aggregation. This feature aggregation process is further optimized using a cross-modal contrastive loss function, which is shown below:

[0096]

[0097] Among them, L cont To compare the loss values, B is the batch size, s() is the cosine similarity, and h i t From modal t, h i v The value comes from mode v, representing two different mode codes, and τ is the temperature coefficient.

[0098] A hybrid index is constructed for the aggregated features based on a joint scheme of product quantization and graph indexing. The joint scheme of product quantization and graph indexing can be represented as follows:

[0099]

[0100] Where d(x,y) is the hybrid index, m is the subspaced fraction, and d PQ (x (i) ,y (i) ) is the product quantization index, α is the balance coefficient, and d Graph (x,y) is the graph index.

[0101] This disclosure proposes a dynamic routing multimodal encoder, also known as a multimodal large model, which enhances fine-grained feature extraction and develops a hybrid quantization indexing strategy to balance retrieval accuracy and efficiency, thus achieving a unified representation of cross-modal features.

[0102] (3) Then, the mapped low-altitude grid data is divided into multiple dynamic sub-regions according to the data distribution density and data access popularity using dynamic grid partitioning. The Loki log index sharding mechanism and the tagged metadata management of the hybrid index corresponding to each dynamic sub-region are combined to reduce the redundant retrieval range.

[0103] Unlike static meshing, dynamic meshing updates the mesh structure in real time based on factors such as the complexity of physical phenomena, boundary changes, or material deformation. Dynamic meshing techniques are particularly crucial when dealing with large deformations, free surfaces, phase transitions, and complex physical field changes. In simulating such problems, dynamic meshing can provide more accurate and efficient solutions. In the dynamic mesh partitioning modeling process disclosed in this paper, the core step is to establish a partitioning strategy for the data space Ω, i.e., a dynamic mesh partitioning strategy, such that each dynamic sub-region R... k The following conditions must be met:

[0104] Density sensitive: High data density regions are divided into finer-grained segments;

[0105] Popularity sensitive: High-frequency access areas retain lower index levels;

[0106] Dynamic balancing: Automatically adjusts partition boundaries based on real-time statistics.

[0107] We define spatiotemporal coordinates (x,t)∈Ω, the data distribution density ρ(x) is constructed using kernel density estimation, and the data access popularity f(x) is updated using exponential smoothing. A joint optimization objective function is constructed by combining the data distribution density and data access popularity. Based on this, the mapped data is divided into multiple dynamic sub-regions according to the dynamic grid partitioning strategy and the joint optimization objective function. The joint optimization objective function is shown below:

[0108]

[0109] in, For density equilibrium, Let K be the number of dynamic subregions, and μ() be the joint optimization objective function, representing the heat normalization term. The density equilibrium term uses the coefficient of variation. Its physical meaning is to eliminate the influence of regional area dimensions and penalize partitions with large density fluctuations. When the data distribution density ρ(x) is in R k When the internal distribution is uniform, this term approaches zero. In the normalization term of heat, this is expressed as the coefficient of variation. The access frequency is compressed to the [0,1] interval to ensure that different dimensions can be jointly optimized. The other parameters in the formula mainly serve as weighting coefficients α∈[0,1] to adjust the relative importance of density and popularity. When α=1, it degenerates into a pure density-driven partition, and when α=0, it is completely dominated by access frequency.

[0110] Deeply optimized based on the Loki log storage engine, the automatic extraction and encoding of tags is achieved by rewriting its hybrid index construction process. On the write path, an LSM tree structure is used for batch processing of metadata updates to ensure high-throughput writes; on the read path, a multi-layered caching system is built to intelligently prefetch search results for hot tags.

[0111] (4) Then, the hybrid index is adaptively adjusted through dynamic index optimization. By drawing on the stream processing mechanism of time series databases, an incremental hybrid index is built for the newly added multimodal low-altitude grid data, and invalid search ranges are filtered through Bloom filters to reduce index maintenance overhead.

[0112] The core of this stage lies in establishing a co-evolution mechanism between the index structure and data distribution, transforming traditional batch index reconstruction into a continuous incremental optimization process to achieve Pareto optimality between retrieval efficiency and maintenance cost. A windowed stream processing engine is used as the basic architecture, automatically triggering metadata feature extraction from the spatiotemporal grid during data writing. By using a sliding time window to statistically analyze the data arrival rate and spatial coverage of grid cells, hot and cold grids are dynamically divided. For frequently updated hot grids, an LSM tree structure is used for hierarchical index construction, temporarily storing real-time data in an in-memory table. Merging optimization is performed during batch flushing to the disk layer. This design borrows from the write optimization characteristics of time-series databases, ensuring low-latency writes while avoiding random I / O overhead. Cold grids employ a compression encoding strategy, transforming discrete data points into continuous intervals on a space-filling curve, sacrificing a small amount of retrieval accuracy for an exponential reduction in index size.

[0113] A dual feedback adjustment mechanism is introduced during the incremental index construction process: on the one hand, based on historical retrieval pattern analysis, a correlation model between retrieval popularity and index granularity is established, automatically increasing the index resolution for frequently accessed areas; on the other hand, online learning algorithms predict data distribution evolution trends, injecting forward-looking offsets into the grid splitting threshold to prevent index failure due to data drift. This forward-looking-feedback composite regulation ensures that the index structure always adapts to changes in system state in advance, avoiding the lag effect of traditional passive adjustments.

[0114] To reduce the computational overhead caused by invalid searches, an evolvable Bloom filter is deployed at the search routing layer. This Bloom filter is not a static bit array, but rather dynamically adjusts the number of hash functions and the bit vector length based on real-time data distribution. When an increase in the false positive rate of a specific grid cell is detected, filter parameter reconfiguration is automatically triggered: either by increasing the number of hash functions to reduce the false positive probability, or by extending the bit vector length to maintain space efficiency. Bloom filter updates employ a hot-upgrade mechanism, utilizing double-buffering technology to ensure search continuity and guarantee that the parameter adjustment process is transparent to the business.

[0115] (5) When caching low-altitude grid data, the EHCD hierarchical multi-level caching architecture is adopted, which divides the caching resources into three levels: on-chip high-speed cache, distributed memory cache and cloud storage. Each level forms a vertical storage topology through an adaptive data migration mechanism, realizing the hierarchical collaboration of on-chip high-speed cache, distributed memory cache and cloud storage, so as to achieve a dynamic balance between data access efficiency and storage cost.

[0116] Specifically, the on-chip cache layer uses silicon-based nanoscale storage media. By collecting access pattern characteristics from the processor instruction stream in real time, a lightweight hotness prediction model is constructed, keeping high-frequency access data residing near the computing unit. Its replacement strategy integrates the dual weights of temporal locality and spatial correlation to ensure nanosecond-level response for hot data. The distributed memory cache layer adopts a partitioned consistent hash ring structure, dynamically mapping recently accessed data to cluster nodes according to a time decay factor. It maintains the cross-node replica status through an asynchronous heartbeat protocol and performs horizontal elastic scaling combined with data block hotness gradients, achieving access load balancing within sub-millisecond latency. The cloud storage layer builds a cold data archive based on object storage services. It uses erasure coding sharding and intelligent lifecycle strategies to compress and encrypt low-frequency data for persistent storage. At the same time, it establishes metadata indexes and prefetch links to support bidirectional conversion between cold and hot data states. Cross-level addressing optimization is achieved through a hierarchical directory service system among the three caches. The global metadata controller uses a probabilistic Bloom filter to quickly locate the data level and constructs the shortest response link combined with an access path prediction algorithm, effectively reducing cross-level communication overhead. This architecture achieves Pareto optimality for cache hit rate and resource utilization through joint optimization of the data heat propagation model and storage cost function.

[0117] Here, the time series database can also be combined with AI to train a lightweight model to predict data access patterns. Based on this, the low-altitude grid data that may be requested is prefetched into the on-chip cache to improve the hit rate. The core mechanism of the dynamic cache optimization framework that integrates the characteristics of the time series database and the machine learning model lies in establishing the spatio-temporal continuity hypothesis of the data access pattern and capturing the implicit access rules through an online learning algorithm. Assume that the time series data stream S = {(x i , t i )} i＝1 N , where x i is the data feature, t i is the timestamp, and N is the number of data points. Construct a probability prediction model f θ : (x, t) ∈ (0, 1). This model predicts the probability that the region x will be accessed within the future time window [t + Δt] by learning the spatio-temporal association patterns in the historical access sequences.

[0118] In the process of deriving the model architecture, a lightweight spatio-temporal attention network (LSTA-Net) is adopted. Its design draws on the time encoding mechanism of Transformer and the spatial locality perception characteristics of CNN. The input layer maps the spatio-temporal coordinates (x, t) into a joint embedding vector where PE() is the spatial position encoding function and TE() is the time encoding function. Local causal convolution is introduced in the shallow layer of the network to force the model to make predictions only relying on historical information and avoid future data leakage. The calculation of the attention weights adopts sparsification processing, and only the Top-K significantly relevant connections are retained, reducing the computational complexity from O(n 2 ) to O(nlogn). Then, a dual-threshold decision mechanism is adopted to establish a cache prefetching strategy. When the predicted probability p > α, the prefetching is triggered immediately. When β < p ≤ α, it is delayed until the cache is idle. The prefetching priority is determined by the hybrid weight w = p · e -γΔt , where Δt is the predicted time span and γ is the decay coefficient. This strategy implements a lazy eviction algorithm in the Redis cache layer. When the cache space is insufficient, the data blocks with high w values are preferentially retained.

[0119] (6) Finally, when retrieving the low-altitude grid data, cross-modal low-altitude grid data retrieval is performed by combining SIMD instruction set parallelization of vector similarity calculation and GPU acceleration of complex spatial relationship operations, and the cross-modal retrieval latency is controlled within milliseconds.

[0120] In detail, at the vector similarity calculation layer, a batch memory access mode based on memory alignment is constructed to address the high-dimensionality of cross-modal embedded vectors. Specifically, the 512-bit wide vector register of the AVX-512 instruction set is used to decompose the cosine similarity calculation into a parallel multiply-accumulate operation chain. Memory latency is hidden through loop unrolling and instruction rearrangement techniques, while mask registers are used to avoid redundant calculations of unaligned data. This design enables full similarity comparison of 32-dimensional floating-point vectors to be completed within a single instruction cycle, with a theoretical peak throughput several times that of traditional scalar computation. The hybrid computing framework achieves collaborative optimization of hardware resources through a dynamic task scheduler. A computational cost prediction model is established to automatically select the execution path based on data dimension and index density: CPU-side SIMD acceleration is used for low-dimensional dense vectors, while high-dimensional sparse features are handled by the GPU. Static scheduling strategies are used for deterministic computation tasks, while dynamic work-stealing queues are used for non-deterministic tasks. At the implementation level, hardware characteristics and algorithm logic are deeply integrated, and a heterogeneous computing backend is extended on the Faiss framework to control cross-modal retrieval latency to the millisecond level.

[0121] In summary, this disclosure achieves at least the following technical effects:

[0122] 1. Dual improvement in efficiency and accuracy of cross-modal retrieval

[0123] By mapping heterogeneous data such as text, images, and videos to a unified vector space through a multimodal large model, combined with a hybrid index, the semantic gap problem in traditional multimodal retrieval can be effectively eliminated.

[0124] 2. High adaptability in dynamic data environments

[0125] The dynamic grid partitioning strategy and incremental index update mechanism, through dynamic partitioning and Loki shard tagging management, can detect changes in data distribution in real time (such as sudden airspace control area updates). Combined with Bloom filters to quickly filter invalid query ranges, it can greatly reduce index maintenance overhead.

[0126] 3. Optimize cache resource utilization and hit rate

[0127] The EHCD hierarchical caching architecture, through the hierarchical collaboration of on-chip high-speed cache, distributed memory cache and cloud storage, combined with a lightweight AI model to predict spatiotemporal-modal joint access patterns, can predict data requests in the next few seconds.

[0128] 4. Collaborative acceleration of heterogeneous computing power and improvement of energy efficiency

[0129] By leveraging SIMD instructions to parallelize vector similarity calculations and GPU-accelerated spatial relationship operations, combined with an edge-cloud collaborative computing framework, the end-to-end latency of complex cross-modal queries can be controlled within milliseconds, while also meeting the low-power requirements of UAV onboard equipment.

[0130] In short, this approach maps heterogeneous data such as text, images, and videos to a unified vector space based on a multimodal large model, constructing a hybrid index structure with cross-modal semantic associations. A dynamic grid partitioning strategy divides the entire data domain into dynamic sub-regions based on density and access frequency. Fine-grained index management is achieved by combining tagged metadata and the Loki log index sharding mechanism. A streaming incremental index engine is designed, leveraging the real-time processing capabilities of the time-series database to build lightweight incremental indexes on new data. A Bloom filter quickly filters invalid query ranges, reducing index maintenance costs. An EHCD hierarchical multi-level caching architecture is constructed, utilizing on-chip high-speed... Layered collaboration between caching, distributed memory caching, and cloud storage, combined with lightweight AI models to predict spatiotemporal-modal joint access patterns, enables intelligent pre-fetching of hot data and hierarchical compressed storage of hot and cold data, improving cache hit rate and resource utilization. Parallelized vector similarity calculation using SIMD instruction set, combined with GPU-accelerated spatial topology relationship parsing, optimizes cross-modal retrieval latency to milliseconds through an edge-cloud collaborative computing framework. Ultimately, this forms a full-link optimization system from multimodal fusion representation, dynamic index tuning, intelligent cache scheduling to heterogeneous computing acceleration, significantly improving the real-time retrieval efficiency, system scalability, and energy efficiency of low-altitude grid data.

[0131] The following is a specific implementation case of applying the above method 100:

[0132] In the case of the river low-altitude logistics distribution system, text work orders submitted by cargo ships (“Container 12 in Zone A3 is damaged”), video streams from dock cameras, and drone inspection images are mapped into a unified vector (dimension = 1024) using a multimodal large model. Through hybrid index construction, the drone command recall speed is improved to within 50ms. Subsequently, the system matches the nearest available drone (error < 5 meters) and safe flight path within 10ms. In this way, through technological innovations such as multimodal fusion, dynamic grid optimization, and intelligent caching, second-level response and millimeter-level spatial accuracy of river low-altitude logistics are achieved, providing a reusable technical paradigm for the digital upgrade of the "golden waterway".

[0133] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0134] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0135] Figure 2 A structural diagram of a multimodal low-altitude grid data dynamic indexing device with an improved caching mechanism provided by an embodiment of this disclosure is shown, as follows: Figure 2 As shown, the device 200 may include:

[0136] The mapping module 210 is used to map multimodal low-altitude grid data to a unified vector space and to build a hybrid index for cross-modal semantic association of the mapped low-altitude grid data.

[0137] The partitioning module 220 is used to divide the mapped low-altitude grid data into multiple dynamic sub-regions according to the data distribution density and data access frequency, and to combine the Loki log index sharding mechanism and tagged metadata management to manage the hybrid index corresponding to each dynamic sub-region.

[0138] The cache module 230 is used to cache the mapped low-altitude grid data using an EHCD hierarchical multi-level cache architecture.

[0139] The retrieval module 240 is used to perform cross-modal low-altitude grid data retrieval by combining parallelized vector similarity calculation of the SIMD instruction set with GPU-accelerated complex spatial relationship calculation.

[0140] Understandable Figure 2 Each module / unit in the illustrated device 200 has the ability to implement Figure 1 The functions of each step in method 100 shown, and their corresponding technical effects, will not be elaborated here for the sake of brevity.

[0141] Figure 3 A structural diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 300 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0142] like Figure 3As shown, the electronic device 300 may include a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0143] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0144] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer program product, including a computer program tangibly contained in a computer-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0145] The various embodiments described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] It should be noted that this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute method 100 and achieve the corresponding technical effects achieved by the embodiments of this disclosure in executing the method. For the sake of brevity, these will not be elaborated here.

[0149] In addition, this disclosure also provides a computer program product including a computer program that implements method 100 when executed by a processor.

[0150] To provide interaction with a user, the embodiments described above can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0151] The embodiments described above can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0152] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0153] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for dynamic indexing of multi-modal low-altitude grid data with improved caching mechanism, characterized in that, The method comprises: mapping multi-modal low-altitude grid data to a unified vector space, and constructing a hybrid index of cross-modal semantic association for the mapped low-altitude grid data; dividing the mapped low-altitude grid data into multiple dynamic sub-regions according to data distribution density and data access heat, and managing the hybrid index of each dynamic sub-region in combination with a Loki log index fragmentation mechanism and labeled metadata; caching the mapped low-altitude grid data by using an EHCD hierarchical multi-level cache architecture; performing cross-modal low-altitude grid data retrieval by combining SIMD instruction set parallelization vector similarity calculation and GPU acceleration complex spatial relationship operation.

2. The method of claim 1, wherein, The mapping of multi-modal low-altitude grid data to a unified vector space, and the construction of a hybrid index of cross-modal semantic association for the mapped low-altitude grid data, comprises: extracting features of each modal low-altitude grid data by using an encoder corresponding to each modal low-altitude grid data; aligning the extracted features of each modal low-altitude grid data in a unified vector space, and optimizing feature aggregation by using a dynamic routing mechanism; constructing a hybrid index of cross-modal semantic association for the aggregated low-altitude grid data features based on a product quantization and graph index joint scheme.

3. The method of claim 1, wherein, The division of the mapped low-altitude grid data into multiple dynamic sub-regions according to data distribution density and data access heat, and the management of the hybrid index of each dynamic sub-region in combination with a Loki log index fragmentation mechanism and labeled metadata, comprises: constructing a dynamic grid partitioning strategy, wherein the dynamic grid partitioning strategy is characterized in that each dynamic sub-region satisfies the following conditions: density sensitive: high data density areas are divided into finer granularity; heat sensitive: high access frequency areas retain lower index levels; dynamic balance: automatically adjust the partition boundary according to real-time statistics; constructing data distribution density by kernel density estimation, updating data access heat by exponential smoothing method, and constructing a joint optimization objective function in combination with data distribution density and data access heat; dividing the mapped low-altitude grid data into multiple dynamic sub-regions according to the dynamic grid partitioning strategy and the joint optimization objective function; based on the Loki log storage engine, automatically extract and encode labels by modifying the hybrid index construction process; on the write path, use LSM tree structure to process metadata updates in batches, and on the read path, construct a multi-level cache system to intelligently prefetch the retrieval results of hot labels.

4. The method of claim 1, wherein, The caching of the mapped low-altitude grid data by using an EHCD hierarchical multi-level cache architecture, comprises: when caching the mapped low-altitude grid data, the EHCD hierarchical multi-level cache architecture is used to divide the cache resources into on-chip cache, distributed memory cache and cloud storage three levels, each level forms a vertical storage topology through an adaptive data migration mechanism, realizing the hierarchical cooperation of on-chip cache, distributed memory cache and cloud storage, wherein the on-chip cache is used to store hot low-altitude grid data, the distributed memory cache is used to store recent access low-altitude grid data, and the cloud storage is used to store cold low-altitude grid data.

5. The method of claim 1, wherein, The combination of the SIMD instruction set parallelizes the vector similarity calculation and the GPU accelerated complex spatial relationship operation for cross-modal low-altitude grid data retrieval, including: In the vector similarity calculation layer, a batch memory access mode based on memory alignment is constructed for the high-dimensional characteristics of cross-modal embedded vectors. Specifically, the cosine similarity calculation is decomposed into a parallel multiplication and addition operation chain using a 512-bit wide vector register of the AVX-512 instruction set, memory latency is hidden through loop unrolling and instruction rearrangement techniques, and redundant calculations of non-aligned data are avoided using a mask register. A dynamic task scheduler is used to optimize hardware resources. A calculation cost prediction model is established to automatically select an execution path based on data dimensions and index density: low-dimensional dense vectors are accelerated using the CPU-side SIMD, high-dimensional sparse features are processed by the GPU, and a static scheduling strategy is used for deterministic computing tasks, while a dynamic work-stealing queue is used for non-deterministic tasks.

6. The method of claim 1, wherein, The method further includes: A hybrid index adaptive adjustment is achieved through dynamic index optimization. By referring to the stream processing mechanism of a time series database, an incremental hybrid index is constructed for newly added multi-modal low-altitude grid data, and a Bloom filter is used to filter invalid retrieval ranges.

7. The method of claim 6, wherein, The hybrid index adaptive adjustment through dynamic index optimization includes: A windowed stream processing engine is used as the basic framework, and metadata feature extraction of the spatiotemporal grid is automatically triggered when low-altitude grid data is written. The data arrival rate and spatial coverage of grid cells are calculated through a sliding time window to dynamically divide hot grids and cold grids. For high-frequency updated hot grids, a LSM tree structure is used for hierarchical index construction, and real-time data is temporarily stored in a memory table and merged and optimized when batch flushing to the disk layer. Cold grids use a compression encoding strategy to convert discrete data points into continuous interval representations on a space-filling curve. A double feedback regulation mechanism is introduced: on the one hand, a correlation model between retrieval frequency and index granularity is established based on historical retrieval pattern analysis, and the index resolution of dynamic sub-regions with high access frequency is automatically improved; on the other hand, an online learning algorithm is used to predict the evolution trend of low-altitude grid data distribution, and a forward-looking offset is injected into the grid splitting threshold.

8. The method of claim 6, wherein, The Bloom filter is used to filter invalid retrieval ranges, including: An evolvable Bloom filter is deployed at the retrieval routing layer, which dynamically adjusts the number of hash functions and the length of the bit vector according to the real-time low-altitude grid data distribution. When the retrieval false positive rate of a specific grid cell is detected to be rising, the filter parameter reconfiguration is automatically triggered: the number of hash functions is increased to reduce the false positive probability, or the length of the bit vector is extended to maintain the space efficiency.

9. The method of claim 4, wherein, The method further includes: A time series database is combined with AI to train a lightweight model to predict data access patterns, and hot low-altitude grid data is pre-fetched to the on-chip cache based on the prediction.

10. A multi-modal low-altitude mesh data dynamic indexing device with improved caching mechanism, characterized in that, The device includes: A mapping module is configured to map multi-modal low-altitude grid data to a unified vector space and construct a hybrid index with cross-modal semantic association for the mapped low-altitude grid data. The division module is used for dividing the mapped low-altitude grid data into multiple dynamic sub-regions according to data distribution density and data access hotness, and combining a Loki log index fragmentation mechanism and a labeled metadata to manage a hybrid index corresponding to each dynamic sub-region; The cache module is used for caching the mapped low-altitude grid data by using an EHCD hierarchical multi-level cache architecture; The retrieval module is used for performing cross-modal low-altitude grid data retrieval by combining a SIMD instruction set parallelized vector similarity calculation and a GPU accelerated complex spatial relationship operation.

Images