A method, apparatus and storage medium for estimating the cardinality of high-dimensional sparse data

By employing lightweight neural radiation field modeling and adaptive integration methods, the cardinality estimation problem under high-dimensional sparse data is solved, achieving high-precision and efficient cardinality estimation, which is applicable to database query optimization and multi-table join scenarios.

CN123115237APending Publication Date: 2026-11-17HENAN CROSS SCIENCE & TECHNOLOGY RESEARCH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611238713.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing cardinality estimation methods suffer from problems such as the curse of dimensionality, severe overfitting, low inference efficiency, and insufficient support for multi-table joins in high-dimensional sparse data scenarios, making it difficult to meet the performance requirements of online database inference.

Method used

Lightweight Neural Radiation Field (L-NeRF) modeling is employed, and the continuous density field of high-dimensional sparse data is fitted by manifold-sensing Fourier position encoding and causal attention layer. Combined with adaptive integration and lightweight processing, high-precision cardinality estimation is achieved.

Benefits of technology

It achieves high-precision cardinality estimation in high-dimensional sparse data with an inference latency of less than 15ms, adapts to multi-table join scenarios, supports dynamic data environments, and significantly improves estimation accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure FT_2
    Figure FT_2
  • Figure FT_3
    Figure FT_3
Patent Text Reader

Abstract

This invention discloses a method, apparatus, and storage medium for cardinality estimation of high-dimensional sparse data, belonging to the interdisciplinary field of database query optimization and artificial intelligence. Addressing the shortcomings of traditional cardinality estimation in high-dimensional sparse scenarios, such as the curse of dimensionality, overfitting issues in deep learning methods, and low inference efficiency, this invention transforms the discrete cardinality estimation problem into a spatial integration problem of a continuous neural radiation field for the first time. This invention adapts the low-dimensional manifold distribution of high-dimensional sparse data to manifold-aware Fourier position encoding, constructs a lightweight neural radiation field network with causal attention enhancement to fit the continuous data density field, designs a hierarchical adaptive Monte Carlo integration algorithm to achieve efficient and high-precision estimation, and proposes a homomorphic field fusion mechanism to support multi-table join scenarios. The sample complexity of this invention increases polynomially with the environmental dimension, fundamentally alleviating the curse of dimensionality. The q-error is reduced by more than 50% compared to the existing best methods, while simultaneously meeting the low-latency and low-memory inference requirements of online databases. It can be widely applied in database query optimization, data governance, and data circulation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of database query optimization and artificial intelligence, specifically relating to a method, apparatus and storage medium for estimating the cardinality of high-dimensional sparse data. Background Technology

[0002] Cardinality estimation (CE) is a core component of the database query optimizer. Its output directly determines join sorting, index selection, and execution plan generation, making it a key technology affecting the overall performance of a database system. With the rapid development of recommendation systems, financial risk control, user behavior analysis, and other businesses, databases commonly contain high-dimensional sparse data of 50-200 dimensions, posing new challenges to cardinality estimation techniques.

[0003] Existing cardinality estimation methods are mainly divided into three categories, all of which are difficult to adapt to high-dimensional sparse scenarios:

[0004] The first category is traditional methods, including histograms, sampling methods, and probabilistic summary structures (such as HyperLogLog). These methods rely on the assumption of attribute independence, and the estimation error increases exponentially with increasing dimensionality, i.e., the "curse of dimensionality," with accuracy dropping sharply after the dimension exceeds 20. At the same time, sparse data downsampling methods have insufficient effective samples, rendering the estimation results invalid.

[0005] The second category is deep learning methods, including MLP, Transformer, Tree-GRU, FactorJoin, etc. These methods fit the data distribution through neural networks, but they are prone to overfitting under high-dimensional sparse data, and the model has a large number of parameters and high inference latency, which cannot meet the millisecond-level response requirements of online database inference; at the same time, most methods are only optimized for single-representation scenarios, and the accuracy is severely degraded in multi-table join scenarios.

[0006] The third category is hybrid methods, which combine traditional summarization and learning models, but still have not broken through the inherent limitations of discrete distribution modeling, and have limited performance improvement in scenarios with 100-dimensional sparse data.

[0007] Neural Radiance Fields (NeRF) is an emerging technology in computer vision that achieves breakthroughs in viewpoint synthesis tasks by implicitly modeling continuous radiance fields in 3D space using multilayer perceptrons. Its core advantage lies in its ability to model high-dimensional continuous manifold distributions with a compact network structure, theoretically possessing the potential to alleviate the curse of dimensionality. However, it has not yet been applied to the field of database cardinality estimation. Existing NeRF architectures are designed for visual rendering tasks and suffer from problems such as visual module redundancy, low efficiency in high-dimensional space integration, and insufficient multi-scene fusion capabilities, making them unsuitable for direct transfer to cardinality estimation scenarios involving high-dimensional sparse data.

[0008] Therefore, there is an urgent need to develop a cardinality estimation method that is suitable for high-dimensional sparse data and has both high accuracy and high efficiency, so as to break through the performance bottleneck of existing technologies. Summary of the Invention

[0009] The technical problem to be solved by the present invention The purpose of this invention is to address the shortcomings of existing cardinality estimation techniques in high-dimensional sparse scenarios, such as the curse of dimensionality, severe overfitting, low inference efficiency, and insufficient support for multi-table joins. This invention provides a cardinality estimation method, device, and storage medium for high-dimensional sparse data, achieving a breakthrough at the modeling paradigm level while balancing estimation accuracy, inference speed, and dimensional scalability.

[0010] Technical solution To achieve the above objectives, the present invention adopts the following technical solution, comprising the following steps:

[0011] S1 Data Preprocessing and Manifold Alignment

[0012] The input high-dimensional sparse data table is subjected to missing value imputation, outlier removal, numerical normalization and discrete field encoding to obtain a standardized high-dimensional data point set, and the intrinsic dimension of the low-dimensional manifold in which the data distribution is located is estimated.

[0013] Specifically, discrete fields are filled with the mode, and continuous fields are filled with the median; outliers are detected and removed using the 3σ criterion; all fields are mapped to the [0,1] interval using min-max normalization; discrete fields are one-hot encoded and then concatenated with continuous fields to form the final standardized data point set. The intrinsic dimension d of the data distribution is calculated using the local k-nearest neighbor intrinsic dimension estimation method. In real-world scenarios, d is usually much smaller than the environmental dimension N.

[0014] S2 Lightweight Attention-Enhancing Neural Radiation Field Modeling

[0015] First, a lightweight neural radiation field network, L-NeRF, is constructed. A manifold-aware Fourier position encoding adapted to the intrinsic dimension is used to feature map the high-dimensional coordinates. A multilayer perceptron with an embedded attention layer is used to fit the continuous data density field in the high-dimensional space. The network is then trained by combining mean squared error loss and density regularization loss to obtain the data density radiation field model.

[0016] Secondly, the manifold-aware Fourier positional encoding adaptively adjusts the encoding layers based on the intrinsic dimension. High-dimensional coordinates are first aligned to the low-dimensional manifold tangent space before encoding, improving the ability to capture high-frequency features while reducing redundant computation. The attention layer employs causal mask constraints to explicitly model the true dependencies between attributes, supplementing the local details of implicit modeling. The loss function introduces a full-space density regularization term to constrain distribution smoothness and suppress overfitting under sparse data.

[0017] S3 Predicate-Constraint Region Mapping

[0018] The predicate conditions of the SQL query are parsed, and the equivalence predicate, range predicate, and fuzzy predicate are transformed into point constraint regions, interval constraint regions, and neighborhood constraint regions in high-dimensional space, respectively. The multi-predicate query obtains the final query constraint region through the region intersection operation.

[0019] Specifically, equivalence predicates are mapped to a fixed set of values ​​for the corresponding dimension; range predicates are mapped to a range of values ​​for the corresponding dimension; fuzzy predicates are mapped to a neighborhood range centered on the target value; and the constraint region for multi-condition queries is the intersection of the Cartesian products of each single-condition region.

[0020] S4 Adaptive High-Dimensional Space Integral Estimation

[0021] Based on the volume of the constrained region, the internal density variance, and the query selectivity, the sampling scale is dynamically adjusted, and the density field within the constrained region is numerically integrated using the hierarchical importance Monte Carlo integral to output the cardinality estimation result.

[0022] Furthermore, by predicting the density distribution of the region through preliminary sampling, the constrained region is divided into high-density and low-density sub-regions, and sampling points are allocated proportionally to reduce the integral variance under the same sampling scale. The sampling scale is dynamically calculated based on the target variance threshold, balancing accuracy and efficiency. The inference latency under standard configuration does not exceed 15ms.

[0023] S5 Multi-table Join Homomorphic Field Fusion

[0024] For multi-table join queries, a homomorphic mapping is constructed with the join key as the common dimension. The independent radiation fields of each table are merged into a joint density radiation field. An integral operation is performed on the constrained region of the joint space to obtain the estimated value of the multi-table join cardinality.

[0025] Specifically, homomorphic mapping merges the join key dimension into a common dimension, avoiding the dimension explosion caused by directly splicing dimensions. The joint density field is the product of the density fields of the two tables after the join key is aligned, and the join key values ​​are constrained to be consistent through indicator functions.

[0026] Furthermore, the method also includes model lightweighting and incremental update steps: 8-bit quantization and structured pruning are used to compress the model, and the number of parameters is controlled within 50M; a high-frequency query cache is established to reduce the latency of repeated queries; new data is only fine-tuned for the corresponding local radiation field region, improving the update efficiency by more than 90%.

[0027] Corresponding to the above method, the present invention also provides a high-dimensional sparse data cardinality estimation device, including a data preprocessing module, a radiation field modeling module, a predicate parsing module, an integral estimation module, a multi-table fusion module, and a caching and updating module.

[0028] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cardinality estimation method described above.

[0029] Beneficial effects Compared with the prior art, the present invention has the following beneficial effects:

[0030] This paradigm innovation fundamentally alleviates the curse of dimensionality. For the first time, this invention transforms discrete cardinality estimation into a spatial integration problem of a continuous neural radiation field. By leveraging the low-dimensional manifold distribution characteristics of real high-dimensional data, it achieves a multinomial growth of sample complexity with environmental dimension, rather than the exponential growth of traditional methods, while maintaining stable high accuracy even in 250-dimensional scenarios.

[0031] The estimation accuracy is significantly improved. Through manifold-aware encoding, causal attention enhancement, and hierarchical integral optimization, the q-error is reduced to below 1.0 on a 100-dimensional sparse dataset, which is more than 50% lower than the existing state-of-the-art baseline; in multi-table join scenarios, the q-error is only 1.12, which is significantly better than existing methods.

[0032] Lightweight and efficient, adapted for online inference. A customized lightweight network architecture combined with quantization pruning results in a model memory of only 42MB; adaptive integration and caching mechanisms ensure query latency of ≤15ms, fully meeting the performance requirements of online database optimization.

[0033] It boasts strong scenario adaptability. It supports multiple predicate types such as equality, range, and fuzzy logic, and is compatible with single-table 100-dimensional queries and multi-table join scenarios. It also supports incremental updates, can adapt to dynamic data environments, and is widely used in various fields such as query optimization, data pricing, and circulation governance. Attached Figure Description

[0035] Figure 1 is a line graph comparing the scalability of cardinality estimation dimension in an embodiment of the present invention. The horizontal axis represents the data dimension, and the vertical axis represents q-error, showing the error change trend of different methods as the dimension increases.

[0036] Horizontal axis: Data dimensions (50, 100, 150, 200, 250); Vertical axis: q-error.

[0037] Curve Explanation: Traditional method: exhibits an exponential upward trend, with q-error exceeding 10 at 250 dimensions; Baseline deep learning methods exhibit an approximately linear upward trend, with a q-error of approximately 3-4 at 250 dimensions; NeRF-CE (this invention): The curve is flat, and the q-error remains within the range of 0.9-1.2 throughout the process.

[0038] Figure 2 is a visualization of the t-SNE dimensionality reduction density fitting effect of the embodiment of the present invention, showing the alignment effect between the actual data point distribution and the predicted density distribution of the neural radiation field.

[0039] Horizontal axis: t-SNE dimension 1; Vertical axis: t-SNE dimension 2.

[0040] Content Description: The background is a predicted density distribution in the form of a heat map (the darker the color, the higher the density), and the red dots are the actual data points; the concentrated areas of actual data points highly overlap with the high-density heat areas, and the sparse areas correspond to low-density heat values, which intuitively reflects the accurate fit of the radiation field to the actual distribution.

[0041] Figure 3 is an overall architecture diagram of the high-dimensional sparse data cardinality estimation system of the present invention, showing the composition of each module and the data flow.

[0042] It adopts a layered structure, from top to bottom: access layer, core processing layer, model layer, and data layer.

[0043] Figure 4 is a flowchart of the adaptive high-dimensional space integration algorithm of the present invention, showing the complete execution steps from region calculation to integration output. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] This embodiment provides a method for estimating the cardinality of high-dimensional sparse data. The complete process is as follows:

[0048] Step 1: Data Preprocessing and Manifold Alignment

[0049] Input a high-dimensional sparse data table, Where N is the number of dimensions (50 ≤ N ≤ 250), and M is the amount of data. Perform the following preprocessing operations:

[0050] (1) Filling missing values: For discrete fields, fill with the mode of the field; for continuous fields, fill with the median of the field.

[0051] (2) Outlier removal: Using the 3σ criterion, for each continuous field, values ​​that exceed the range are identified as outliers and removed.

[0052] (3) Normalization: Min-max normalization is used to map all continuous fields to the interval [0,1]. The formula is as follows:

[0053] in Let be the value of the j-th dimension for the i-th data point.

[0054] (4) Discrete field encoding: Perform one-hot encoding on discrete fields, and then concatenate the encoded vector with continuous fields to form a standardized data point set.

[0055] After preprocessing, the intrinsic dimension of the data distribution is estimated using the local k-nearest neighbor method: for each data point, the distance distribution of its k nearest neighbors is calculated, the local intrinsic dimension is fitted by logarithmic regression, and the average of all points is taken as the global intrinsic dimension d. The intrinsic dimension of real high-dimensional sparse data is usually 5-20, which is much smaller than the environmental dimension N.

[0056] Step 2: Modeling of Lightweight Attention-Enhancing Neural Radiation Fields

[0057] A lightweight neural radiation field network, L-NeRF, was constructed, consisting of three parts: a location encoding layer, an attention-enhanced MLP layer, and a density output layer.

[0058] Manifold-aware Fourier position encoding

[0059] The encoding level is adaptively set based on the intrinsic dimension d, with a value ranging from 8 to 15. The generated random projection matrix P projects the high-dimensional input coordinates x onto the tangent space of the manifold to obtain... Then perform Fourier encoding:

[0060] This encoding method maps low-dimensional manifold coordinates to a high-dimensional feature space, enabling the MLP to fit high-frequency density distributions, and the convergence speed is independent of the environment dimension N.

[0061] Attention-enhanced MLP structure

[0062] The MLP consists of four fully connected layers, with a causal attention layer inserted in between: The first and second fully connected layers have dimensions of 256 and 256 respectively, and use ReLU as the activation function to perform a preliminary nonlinear transformation on the encoded features.

[0063] Causal Attention Layer: The output of Layer 2 is mapped to the query matrix Q, key matrix K, and value matrix V, respectively. An attribute causal adjacency mask M (constructed from attribute dependencies obtained from prior knowledge or data mining) is introduced. Attention is calculated as follows:

[0064] in The dimension of the key vector. This is a Hadamard product. The masking mechanism filters out spurious correlations, explicitly models the true dependencies between attributes, and supplements the local details of NeRF's global implicit modeling.

[0065] Fully connected layers 3-4: dimensions are 256 and 1 respectively, and the final output density value is... , representing the probability density of a data point existing at coordinate x.

[0066] Model training

[0067] The training set consists of real data points (positive samples, density label 1) and randomly sampled points (negative samples, density label 0). The total loss function is:

[0068] Mean square error loss , For real labels, These are the model's predicted values; Density regularization term V is the high-dimensional spatial volume, used to constrain the smoothness of the density distribution and suppress overfitting; Regularization coefficient .

[0069] The Adam optimizer is used with a learning rate of 1e-4, a batch size of 1024, and 200 training epochs. An early stopping mechanism is set, and training is terminated if the validation set loss does not decrease after 20 epochs.

[0070] Step 3: Predicate-Constraint Region Mapping

[0071] Parse the set of predicates in the SQL query and map each type of predicate to a high-dimensional space constraint region:

[0072] (1) Equivalent predicates : Mapped to a point region with value v in dimension j. ;

[0073] (2) Scope predicate : Mapped to the interval region [a, b] of the j-th dimension. ;

[0074] (3) Fuzzy predicates : Maps to the j-th dimension neighborhood region centered at v with radius ε, default .

[0075] For a query containing k predicates, the final constraint region R is the intersection of the regions containing each single predicate: .

[0076] Step 4: Adaptive high-dimensional space integral estimation

[0077] The cardinal estimate is the integral of the density field over the constrained region R: The hierarchical adaptive Monte Carlo integration algorithm is adopted, and the steps are as follows:

[0078] (1) Calculate the volume of the constrained region ,in , Let be the upper and lower bounds of the j-th dimension;

[0079] (2) Pilot sampling: Randomly collect 100 samples within R, and calculate the mean and variance of the density within the region. ;

[0080] (3) Sampling volume calculation: based on the target variance threshold The number of samples required for calculation theory ;

[0081] (4) Sample size pruning: The final sample size is limited to the range [1000, 5000], i.e. ;

[0082] (5) Stratified sampling: Based on the density mean of the pilot sampling, the constrained region is divided into a high-density sub-region (density > mean) and a low-density sub-region, and sampling points are allocated in a 7:3 ratio;

[0083] (6) Integral Calculation: Uniformly sample from both sub-regions, substitute the samples into the density model to calculate the density value, and the integral estimate is: ;

[0084] (7) Simultaneously output integral variance .

[0085] Step 5: Multi-table join and homomorphic field fusion

[0086] For a two-table join query, let table A have N dimensions and a density field. Table B has a dimension of M and represents the density field. The connection key is the k-th dimension.

[0087] A homomorphic mapping is constructed to map the two radiation fields to an (N+M-1) dimensional joint space, merging the bond dimensions into a common dimension. The joint density function is:

[0088] Here, is an indicator function; it is 1 when the connection key values ​​are equal, and 0 otherwise.

[0089] Perform the above adaptive integration on the query constraint region in the union space to obtain the estimated value of the multi-table join cardinality.

[0090] Step 6: Lightweight Optimization and Incremental Updates

[0091] (1) Model compression: 8-bit quantization is performed on the fully connected layer and the attention layer, the model volume is compressed by 4 times and the accuracy loss is ≤2%; structured pruning is adopted to remove neurons with absolute weight values ​​< 0.001, and the final model parameter count is ≤50M.

[0092] (2) Query caching: Establish a high-frequency query area cache to store the integration results of commonly used constraint areas. The cache hit rate is ≥80%, further reducing inference latency.

[0093] (3) Incremental update: When adding new data, locate the local high-dimensional space corresponding to the data point and only fine-tune the network parameters corresponding to the region. The fine-tuning rounds are 10, and the update time is reduced by 90% compared with full retraining.

[0094] Example 2

[0095] This embodiment experimentally verifies the above method, and the experimental setup is as follows.

[0096] Datasets: Self-built Sparse-HD dataset (50-250 dimensions, 10 million data points each), KDD Cup 2023 recommendation dataset (128 dimensions, 8 million data points).

[0097] Baselines for comparison: Histogram, Sampling, HyperLogLog, MLP-CE, Transformer-CE, Tree-GRU-CE, FactorJoin.

[0098] Evaluation metrics: q-error, inference latency, memory usage, and dimensional scalability.

[0099] Experimental environment: NVIDIA RTX 4090 GPU, PyTorch 2.1 framework.

[0100] Partial experimental results

[0101] Accuracy comparison: On the 100-dimensional Sparse-HD dataset, the median q-error of the method in this invention is 0.98, which is 54.4% lower than the best baseline FactorJoin (2.15); on the KDD Cup dataset, the q-error is 1.05, which is 55.7% lower than FactorJoin (2.37).

[0102] Performance comparison: Single query inference latency is 15ms, and model memory is 42MB, which meets the requirements of online database inference.

[0103] Dimensional scalability: When the dimension increases from 50 to 250, the q-error of the method of this invention increases from 0.92 to 1.18, with a gradual increase; while the q-error of the traditional method increases from 3.2 to 10.7, showing an exponential increase, verifying that the present invention significantly alleviates the curse of dimensionality.

[0104] Multi-table join: In a scenario involving a 100-dimensional join between two tables, the q-error of the method in this invention is 1.12, which is far superior to the baseline method.

Claims

1. A method for estimating the cardinality of high-dimensional sparse data, characterized in that, Includes the following steps: S1 Data Preprocessing and Manifold Alignment: The input high-dimensional sparse data table is subjected to missing value imputation, outlier removal, numerical normalization and discrete field encoding to obtain a standardized high-dimensional data point set, and the intrinsic dimension of the low-dimensional manifold in which the data distribution is located is estimated. S2 Lightweight Attention-Enhanced Neural Radiation Field Modeling: A lightweight neural radiation field network L-NeRF is constructed. Manifold-aware Fourier position coding adapted to the intrinsic dimension is used to perform feature mapping on high-dimensional coordinates. A multilayer perceptron with an embedded attention layer is used to fit the continuous data density field in high-dimensional space. The network training is completed by combining mean square error loss and density regularization loss to obtain the data density radiation field model. S3 Predicate-Constraint Region Mapping: Parses the predicate conditions of SQL queries, transforming equivalence predicates, range predicates, and fuzzy predicates into point constraint regions, interval constraint regions, and neighborhood constraint regions in high-dimensional space, respectively. Multi-predicate queries obtain the final query constraint region through region intersection operations. S4 Adaptive High-Dimensional Spatial Integral Estimation: Based on the volume of the constrained region, the internal density variance, and the query selectivity, the sampling scale is dynamically adjusted, and the density field within the constrained region is numerically integrated using hierarchical importance Monte Carlo integration to output the cardinality estimation result; S5 Multi-Table Join Homomorphic Field Fusion: For multi-table join queries, a homomorphic mapping is constructed with the join key as the common dimension. The independent radiation fields of each table are fused into a joint density radiation field. An integral operation is performed on the constrained region of the joint space to obtain the estimated value of the multi-table join cardinality.

2. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S1, the intrinsic dimension of the low-dimensional manifold where the estimated data distribution is located is specifically calculated by using the local k-nearest neighbor intrinsic dimension estimation method, calculating the intrinsic dimension of each local region based on the k-nearest neighbor distance of the data points, and taking the global average value as the final intrinsic dimension d, where 5≤d≤20.

3. The method for estimating the cardinality of high-dimensional sparse data according to claim 2, characterized in that, In step S2, the manifold-aware Fourier position encoding specifically involves: The coding level L is adaptively set according to the intrinsic dimension d, where L = round (d / 2) + 5, and the coding level range is 8 to 15. For the input high-dimensional coordinates x∈R^N, first align them to the tangent space of the low-dimensional manifold using a random projection matrix, and then perform sine-cosine Fourier encoding. The encoding formula is as follows: in These are the aligned manifold coordinates.

4. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S2, the multilayer perceptron with the embedded attention layer is specifically: The multilayer perceptron consists of four fully connected layers and a ReLU activation function, with a causal attention layer inserted between the second and third fully connected layers. The causal attention layer employs a scaled dot product attention mechanism and introduces an attribute causal adjacency matrix as an attention mask, retaining only the attention weights of attribute pairs with causal dependencies. The attention calculation formula is as follows: Where M is the attribute causal adjacency mask matrix. For Hadama accumulation, The dimension of the key vector.

5. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S2, the total loss function combining the mean squared error loss and the density regularization loss is: in The mean square error of the data point density fitting is set to 1 for the true data point density and 0 for the negative sampling point density. This is the regularization term for the square integral of the density across the entire space, used to constrain the smoothness of the density distribution; This is the regularization coefficient, with a value of 0.

01.

6. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S4, the hierarchical importance Monte Carlo integral specifically includes: S41 Precursor Sampling: Randomly collect 100 precursor samples within the constrained region and calculate the mean density and gradient distribution within the region; S42 Region Stratification: The constrained region is divided into high-density sub-regions and low-density sub-regions based on the density threshold. Regions with a density greater than the global mean are high-density sub-regions. S43 Sampling Allocation: The total sampling points are allocated to the high-density sub-region and the low-density sub-region in a 7:3 ratio; S44 Integral Calculation: Uniformly sample and calculate the density integral in the two sub-regions respectively, and sum them to obtain the final integral result.

7. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S4, the dynamic adjustment of the sampling scale specifically refers to: Based on the volume of the constrained region Density variance within the region With target variance threshold Calculate the required theoretical number of samples ; Final sample number Limited to between 1000 and 5000, that is .

8. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, In step S5, the homomorphic mapping fusion specifically involves: Let the density functions of Table A and Table B be respectively , The connection key dimension is k; Construct a homomorphic mapping to map the two radiation fields to a (N+M-1) dimensional joint space, where N and M are the dimensions of the two tables, and the join key dimension is merged into a common dimension; The joint density function is: in This is an indicator function; it takes the value 1 when the connection keys are equal, and 0 otherwise.

9. The method for estimating the cardinality of high-dimensional sparse data according to claim 1, characterized in that, It also includes model lightweighting and incremental update steps: The L-NeRF network is compressed using 8-bit quantization and structured pruning, keeping the number of model parameters below 50M. Establish a high-frequency query result caching mechanism, with a cache hit rate of no less than 80%. For newly added data, locate its corresponding local radiation field region in high-dimensional space, and only make minor adjustments to this region, with 10 rounds of fine-tuning.

10. A device for estimating the cardinality of high-dimensional sparse data, characterized in that, include: The data preprocessing module is used to perform cleaning, encoding, and manifold intrinsic dimension estimation on high-dimensional sparse data tables. The radiation field modeling module is used to build and train a lightweight attention-enhancing neural radiation field network to generate a data density radiation field model. The predicate parsing module is used to parse SQL query predicates and map them to a high-dimensional space constraint region; The integral estimation module is used to perform adaptive hierarchical Monte Carlo integration and output single-table cardinality estimation results. The multi-table fusion module is used to calculate the multi-table connectivity cardinality estimate through homomorphic field fusion. The caching and update module is used for query result caching and incremental model updates.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the high-dimensional sparse data cardinality estimation method as described in any one of claims 1-9.