A wireless channel intelligent estimation method and system based on big data driving
Patent Information
- Application Number
- CN202610622595.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-08
- Publication Date
- 2026-08-18
AI Technical Summary
这些方法通常通过矩阵运算直接求解，但无法有效处理大规模数据集，计算复杂度高（如O(N3)），对噪声敏感，且未能充分利用历史数据的统计规律
[0033] (1) In this invention, the big data processing capability is greatly improved; the Hadoop+Spark distributed architecture is adopted, and the data throughput can reach 2GB/s, which is about 20 times higher than the traditional single-machine method (100MB/s); HDF5 compression storage is used to save about 70% of storage space; data index query is supported, and the query speed is 100 times faster than full table scan.
Smart Images
Figure CN122601422A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wireless communication technology, and in particular to a method and system for intelligent estimation of wireless channels based on big data. Background Technology
[0002] With the development of 5G / 6G communication technologies, new indoor communication systems (such as those using clip-on antenna systems) can generate several gigabytes of channel observation data per second. This data exhibits typical big data characteristics such as high dimensionality, strong correlation, and non-stationarity, posing a significant challenge to traditional channel estimation methods.
[0003] Currently, the relevant technical solutions in this field mainly include:
[0004] 1. Traditional statistical estimation methods, such as least squares (LS) and linear least mean square error estimation (LMMSE). These methods typically solve directly through matrix operations, but they cannot effectively handle large-scale datasets and have high computational complexity (e.g., O(N)). 3 It is sensitive to noise and fails to make full use of the statistical regularities of historical data.
[0005] 2. Traditional deep learning methods, such as using convolutional neural networks (CNNs) for end-to-end channel estimation mapping. These methods improve performance to some extent, but have limited generalization ability, poor adaptability to new scenarios, and are quite cumbersome in terms of data preprocessing and feature engineering. They also lack a systematic data quality management mechanism and are difficult to handle varying input data scales.
[0006] 3. Compressed sensing methods reconstruct data based on the assumption of signal sparsity. However, real-world channel data often does not meet strict sparsity conditions, and the iterative optimization of this method is time-consuming, making it unsuitable for big data processing scenarios with high real-time requirements. Furthermore, its performance is highly dependent on dictionary design.
[0007] In summary, there is currently a lack of an intelligent channel estimation scheme specifically designed for novel indoor communication systems that can efficiently process massive amounts of channel data, possess adaptive learning capabilities, and combine physical constraints with data-driven hybrid modeling. Existing technologies have significant shortcomings and technological gaps in addressing large data volumes, data quality issues, model generalization, and real-time requirements. Summary of the Invention
[0008] To address the above technical problems, this invention provides a big data-driven intelligent estimation method for wireless channels, comprising the following steps:
[0009] S1. Collect multi-user wireless channel data, which includes received signal data, antenna configuration data and environmental parameter data. The data is compressed and stored in HDF5 format and distributed storage is implemented using Hadoop HDFS.
[0010] S2. Use Apache Spark to perform parallel preprocessing on the collected data. The preprocessing operations include data cleaning, complex-to-real number conversion and Z-score standardization, and extract statistical features, frequency domain features and spatial features to form a fused feature vector.
[0011] S3. Calculate the data quality score Q, and perform data augmentation on data with a quality score greater than the threshold.
[0012] S4. Construct and train a generative model based on a diffusion model, which constructs noisy data through a forward diffusion process and trains the model by predicting the noise.
[0013] S5. In the inference phase, starting from random noise, iterative denoising is performed through the generative model, and gradient correction based on measurement consistency is introduced after each denoising step.
[0014] S6. Build a RESTful API service based on Flask, use Redis to cache the trained model, provide a real-time channel estimation interface, and integrate monitoring tools to visualize system performance;
[0015] S7. Compare the performance metrics of different model versions using an A / B testing framework, and make model iteration and deployment decisions based on the statistical test results.
[0016] The technical solution further defined in this invention is:
[0017] Furthermore, in step S1, channel samples are synthesized by calculating line-of-sight path loss and adding random non-line-of-sight scattering components, and then saved in HDF5 format.
[0018] As described above, in a big data-driven intelligent wireless channel estimation method, step S2 uses Pandas and NumPy libraries for preprocessing to convert complex channel data into real vectors, perform Z-score standardization, and extract statistical features. The statistical features include the signal's mean, variance, skewness, kurtosis, maximum value, minimum value, median, and energy. The frequency domain features include the signal's FFT coefficients, dominant frequency components, and spectral entropy. The spatial features include the signal's spatial correlation matrix, PCA dimensionality reduction results, and location coding.
[0019] As described above, in a big data-driven intelligent estimation method for wireless channels, step S3 includes data augmentation methods such as noise injection, configuration transformation, and time spread.
[0020] As described above, in a big data-driven intelligent estimation method for wireless channels, step S3, the formula for calculating the data quality score Q is: ,in, This represents the normalized signal-to-noise ratio, Completeness represents data integrity, and Consistency represents time consistency. , as well as These represent different weighting coefficients.
[0021] As described above, in a big data-driven intelligent estimation method for wireless channels, in step S4, the generating model is set to a multilayer perceptron containing 3-5 fully connected layers, and the diffusion step of the generating model is set to 50 steps.
[0022] As described above, in a big data-driven intelligent wireless channel estimation method, step S5 involves gradient correction based on an adaptive step size. The calculation method is as follows ,in, The initial step size is represented by t, the current denoising step number is represented by t, the total denoising step number is represented by T, and the signal-to-noise ratio is represented by SNR.
[0023] As described above, in a big data-driven intelligent estimation method for wireless channels, step S6 involves using the Grafana tool to visualize and monitor system performance, data quality, and estimation results.
[0024] As described above, in a big data-driven intelligent estimation method for wireless channels, step S7 evaluates the model performance by calculating the normalized mean square error and the correlation coefficient with the real channel.
[0025] This invention also provides a big data-driven intelligent wireless channel estimation system, including...
[0026] The data acquisition module is used to collect multi-user wireless channel data, which includes received signal data, antenna configuration data, and environmental parameter data.
[0027] The distributed storage module is used for compressed storage in HDF5 format and distributed storage is implemented using Hadoop HDFS.
[0028] The big data processing module is used to perform parallel preprocessing on the collected data using Apache Spark. The preprocessing operations include data cleaning, complex-to-real number conversion, and Z-score standardization, and extracting statistical features, frequency domain features, and spatial features to form a fused feature vector; it also calculates the data quality score Q, and performs data augmentation processing on data with quality scores greater than the threshold.
[0029] The deep learning module is used to build and train a generative model based on a diffusion model. This model constructs noisy data through a forward diffusion process and trains the model by predicting the noise. In the inference phase, starting from random noise, iterative denoising is performed through the generative model, and gradient correction based on measurement consistency is introduced after each denoising step.
[0030] The application service module is used to build RESTful API services based on Flask, use Redis to cache trained models, provide a real-time channel estimation interface, and integrate monitoring tools to visualize system performance.
[0031] The deployment decision module is used to compare the performance metrics of different model versions through an A / B testing framework and make model iteration and deployment decisions based on the statistical test results.
[0032] The beneficial effects of this invention are:
[0033] (1) In this invention, the big data processing capability is greatly improved; the Hadoop+Spark distributed architecture is adopted, and the data throughput can reach 2GB / s, which is about 20 times higher than the traditional single-machine method (100MB / s); HDF5 compression storage is used to save about 70% of storage space; data index query is supported, and the query speed is 100 times faster than full table scan.
[0034] (2) In this invention, the estimation accuracy is significantly improved; experiments show that under various signal-to-noise ratio (SNR) conditions, the normalized mean square error (NMSE) of this invention is better than that of traditional LS and CNN methods; for example, when SNR=10dB, the NMSE can reach -11.94dB, which is a significant improvement compared to the LS method (-6.15dB) and the CNN method (-4.94dB); under the extreme condition of low signal-to-noise ratio (-20dB), this invention can still maintain usable performance (NMSE=0.23dB), while traditional methods have basically failed;
[0035] (3) In this invention, the system has excellent performance and reduced cost; the single channel estimation inference latency is less than 20ms, and the system throughput can reach 50 requests / second (single GPU); the number of model parameters is controlled within 500,000, and the size is about 15MB, which can be run and trained on consumer-grade hardware (such as GTX1660); actual deployment cases show that compared with traditional solutions, the server cost is reduced by about 60%, the power consumption is reduced by about 65%, and the development cycle is shortened by about 67%;
[0036] (4) In this invention, the generalization and adaptability are strong; when the model is tested with zero samples in untrained antenna configurations (such as 8, 20, and 32 antennas), it can still maintain stable estimation performance (NMSE between -12.82dB and -13.38dB), showing good configuration adaptability. Attached Figure Description
[0037] Figure 1 This is a schematic diagram of the overall process of the present invention;
[0038] Figure 2 This is a schematic diagram of the data preprocessing process in an embodiment of the present invention;
[0039] Figure 3 This is a schematic diagram illustrating the training and inference process of the diffusion model in an embodiment of the present invention;
[0040] Figure 4 This is a schematic diagram of the HDFS storage structure in an embodiment of the present invention;
[0041] Figure 5 This diagram illustrates the performance comparison between the method of this invention and the traditional CNN method. Detailed Implementation
[0042] This embodiment provides a big data-driven intelligent estimation method for wireless channels, such as... Figure 1 As shown, the above objectives are achieved by constructing a four-layer architecture system comprising a data layer, a processing layer, a model layer, and an application layer. The data layer is responsible for the acquisition of petabyte-scale massive channel data and its HDFS distributed storage; the processing layer uses Apache Spark to implement parallel data cleaning, Z-score standardization, and 128-dimensional feature fusion; the model layer contains the core lightweight diffusion model, which is built based on a 3-5 layer MLP with the number of parameters controlled within 500,000; the application layer provides a RESTful API interface based on Flask and utilizes Redis to implement a high-concurrency cache with a response time of less than 50ms.
[0043] Data Layer: Responsible for the acquisition and storage of massive amounts of channel data. It collects multi-source data such as multi-user received signals, antenna configurations, and environmental parameters, compresses and stores them using the HDF5 format, and builds a distributed storage system based on Hadoop HDFS to achieve efficient management and fault-tolerant backup of petabyte-level data.
[0044] Processing Layer: Responsible for data preprocessing, quality assessment, and augmentation. Apache Spark is used for parallel data cleaning, standardization (e.g., Z-score), and feature extraction (including statistical, frequency, and spatial features), which are then fused into a unified feature vector, reducing the dimensionality to 128. A data quality scoring mechanism is designed to filter or alert on low-quality data. Strategies such as noise injection, configuration transformation, and time extension are employed to augment the training data, expanding it to 3-5 times the size of the original data.
[0045] like Figure 2 As shown, the entire processing flow from the raw signal to the training-ready dataset is described in detail. The core steps include a quality filtering node, which calculates a data quality score $Q$ and compares it to a preset threshold, only processing data that meets the score. For high-quality data, further data augmentation (noise injection, time expansion, etc.) is performed, expanding the training samples to 3-5 times their original size, significantly improving the model's generalization performance.
[0046] Model Layer: As the core intelligent estimation unit, a lightweight generative model (a simplified diffusion model) is employed for channel estimation. This lightweight generative model is a multilayer perceptron network containing 3 to 5 fully connected layers, with the total number of parameters controlled to within 500,000. This model has a simplified structure (e.g., a multilayer perceptron MLP), significantly reducing the total diffusion steps from the standard 1000 steps to 50 steps, thus significantly reducing computational complexity. During inference, an innovative gradient correction step based on measurement consistency constraints is introduced, fusing the results of data-driven learning with the physical observation equations, improving the accuracy and robustness of the estimation. The model also possesses the ability to adaptively adjust parameters based on conditions such as signal-to-noise ratio (SNR).
[0047] like Figure 3 As shown, the technical details of the diffusion model in the training and inference phases are illustrated. During training, the model learns to predict noise through a forward diffusion process; during inference, the system performs 50 iterative denoising steps starting from random noise. The lower right corner of the figure highlights the consistency gradient correction module, which adaptively adjusts the step size of each denoising step based on real-time observation data, thus integrating physical constraints into the data-driven inference process.
[0048] Application Layer: Provides engineered service interfaces and monitoring. A RESTful API built on Flask (with a response time requirement of less than 50 milliseconds) provides real-time estimation services. Redis is used to cache the model to accelerate inference, and tools such as Grafana are used to visualize and monitor system performance, data quality, and estimation results. An A / B testing framework is supported for model iteration.
[0049] This embodiment also provides alternative technical solutions. For example, data augmentation strategies may include oversampling and other methods; in addition to diffusion models, other lightweight generative network structures can be explored at the model layer. Application scenarios for this embodiment include multi-device collaborative communication and channel state management in smart homes, smart factories, VR / AR applications, and smart healthcare.
[0050] This embodiment also includes a big data-driven intelligent wireless channel estimation system, comprising:
[0051] The data acquisition module is used to access data from multiple sources and perform format unification and metadata management.
[0052] The distributed storage module, based on HDFS, implements block-based data storage, multi-replica fault tolerance, and compression. The HDFS block size configuration is 128MB, with a default replica count of 3. Using HDF5 compression reduces storage space by approximately 70%, supporting petabyte-level data storage and data read speeds exceeding 1GB / s. Figure 4 As shown, this reflects the file organization logic of the distributed storage module. The root directory is divided into three core subdirectories: raw_data / stores the raw HDF5 format samples; processed_data / stores the 128-dimensional feature vectors (such as Parquet format) processed by Spark; and models / stores the model weight files of different versions to support A / B testing and version iteration decisions at the application layer.
[0053] The big data processing module, based on the Spark engine, provides parallel data cleaning, feature extraction, quality assessment, and data augmentation functions.
[0054] The deep learning module provides model training, optimization inference, and version management functions.
[0055] The application services module provides real-time estimation APIs and system status monitoring based on Flask and Redis.
[0056] The operation and maintenance management module is used for system log collection, performance indicator monitoring, and anomaly alerts.
[0057] An indoor scenario experiment was conducted in one embodiment.
[0058] Experimental environment: The hardware configuration used was an Intel i5-10400 processor, an NVIDIA GTX 1660 Super graphics card, and 16GB of RAM. The software environment included Ubuntu 20.04, Hadoop 3.3.0, Spark 3.2.0, Python 3.8, and PyTorch 1.12.0.
[0059] Dataset generation: A simulated indoor scene of 10m×10m×3m was used to generate 10,000 channel data samples containing 4 users and 16 antennas. The channel samples were synthesized by calculating the line-of-sight (LoS) path loss and adding random non-line-of-sight (NLoS) scattering components, and saved in HDF5 format.
[0060] Data preprocessing: A preprocessing pipeline was implemented using the Pandas and NumPy libraries. This included converting complex channel data into real vectors, performing Z-score standardization, extracting statistical features such as mean, variance, and energy, calculating data quality scores based on simple rules, and filtering out samples with quality scores below 0.6.
[0061] Lightweight Model Training: Define a simplified diffusion model with a network consisting of 3-4 fully connected MLP layers. Load processed data using a custom dataset and train the model to predict noise from noisy states with a diffusion step count of 50. Train for approximately 50 epochs on a GTX 1660 graphics card.
[0062] Inference and Evaluation: A simplified inverse denoising inference function is implemented. The trained model is loaded, and channel estimation is performed on the test set. Model performance is evaluated by calculating the normalized mean square error (NMSE) and the correlation coefficient with the real channel. This embodiment achieves estimation accuracy superior to traditional methods, and the estimation results of the real and imaginary parts of the channel are plotted. Figure 5 As shown, the performance of the method in this embodiment is intuitively compared with that of the traditional convolutional neural network (CNN) scheme in six key dimensions. Experimental data shows that the method in this embodiment significantly outperforms existing technologies in terms of accuracy (NMSE up to -11.94dB), real-time performance (latency <20ms), throughput (50 requests / second), and generalization ability (stable performance in zero-sample testing), and also demonstrates superior performance in terms of deployment cost and storage efficiency.
[0063] In another embodiment, big data processing optimization is performed.
[0064] This example demonstrates a solution for handling massive data streams using Apache Spark in a production environment through distributed big data processing.
[0065] Distributed Processing: Initialize a Spark session to read large-scale channel data from the HDFS distributed file system. By defining partitioning functions, leverage Spark's parallel computing capabilities to perform feature extraction and quality scoring on each data partition. Distributed feature standardization is performed using tools such as StandardScaler in Spark MLlib. Finally, the processed high-quality data is written back to HDFS in Parquet format, achieving a linearly scalable large-scale data processing workflow.
[0066] This embodiment utilizes distributed big data processing technology and deep learning models to intelligently estimate and process massive, high-dimensional channel state information of novel indoor communication systems (such as 5G / 6G scenarios). It aims to address the problems of low big data processing efficiency, lack of data quality control, insufficient model generalization ability, and difficulty in meeting real-time requirements in existing wireless channel estimation technologies when dealing with massive, high-dimensional, and non-stationary communication big data. Specifically, this embodiment proposes an intelligent channel estimation system and method that integrates distributed storage computing, data quality management, lightweight deep learning models, and physical constraint fusion to achieve efficient, accurate, robust, and scalable estimation of channel states.
[0067] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.
Claims
1. A big data-driven intelligent estimation method for wireless channels, characterized in that: Includes the following steps: S1. Collect multi-user wireless channel data, which includes received signal data, antenna configuration data and environmental parameter data. The data is compressed and stored in HDF5 format and distributed storage is implemented using Hadoop HDFS. S2. Use Apache Spark to perform parallel preprocessing on the collected data. The preprocessing operations include data cleaning, complex-to-real number conversion and Z-score standardization, and extract statistical features, frequency domain features and spatial features to form a fused feature vector. S3. Calculate the data quality score Q, and perform data augmentation on data with a quality score greater than the threshold. S4. Construct and train a generative model based on a diffusion model, which constructs noisy data through a forward diffusion process and trains the model by predicting the noise. S5. In the inference phase, starting from random noise, iterative denoising is performed through the generative model, and gradient correction based on measurement consistency is introduced after each denoising step. S6. Build a RESTful API service based on Flask, use Redis to cache the trained model, provide a real-time channel estimation interface, and integrate monitoring tools to visualize system performance; S7. Compare the performance metrics of different model versions using an A / B testing framework, and make model iteration and deployment decisions based on the statistical test results.
2. The wireless channel intelligent estimation method based on big data as described in claim 1, characterized in that: In step S1, channel samples are synthesized by calculating line-of-sight path loss and adding random non-line-of-sight scattering components, and then saved in HDF5 format.
3. The wireless channel intelligent estimation method based on big data as described in claim 1, characterized in that: In step S2, Pandas and NumPy libraries are used for preprocessing to convert complex channel data into real vectors, perform Z-score standardization, and extract statistical features. Statistical characteristics include the signal's mean, variance, skewness, kurtosis, maximum value, minimum value, median, and energy; frequency domain characteristics include the signal's FFT coefficients, dominant frequency components, and spectral entropy. Spatial features include the spatial correlation matrix of the signal, the PCA dimensionality reduction result, and the location encoding.
4. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S3, the data augmentation methods include noise injection, configuration transformation, and time extension.
5. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S3, the formula for calculating the data quality score Q is: ,in, This represents the normalized signal-to-noise ratio, Completeness represents data integrity, and Consistency represents time consistency. , as well as These represent different weighting coefficients.
6. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S4, the generative model is set to a multilayer perceptron containing 3-5 fully connected layers, and the diffusion step of the generative model is set to 50 steps.
7. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S5, gradient correction is performed based on an adaptive step size. The calculation method is as follows ,in, The initial step size is represented by t, the current denoising step number is represented by t, the total denoising step number is represented by T, and the signal-to-noise ratio is represented by SNR.
8. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S6, Grafana is used to visualize and monitor system performance, data quality, and estimation results.
9. The intelligent wireless channel estimation method based on big data as described in claim 1, characterized in that: In step S7, the model performance is evaluated by calculating the normalized mean square error and the correlation coefficient with the real channel.
10. A big data-driven intelligent wireless channel estimation system, used to perform the method according to any one of claims 1-9, characterized in that: include The data acquisition module is used to collect multi-user wireless channel data, which includes received signal data, antenna configuration data, and environmental parameter data. The distributed storage module is used for compressed storage in HDF5 format and distributed storage is implemented using Hadoop HDFS. The big data processing module is used to perform parallel preprocessing on the collected data using Apache Spark. The preprocessing operations include data cleaning, complex-to-real number conversion, and Z-score standardization, and extracting statistical features, frequency domain features, and spatial features to form a fused feature vector; it also calculates the data quality score Q, and performs data augmentation processing on data with quality scores greater than the threshold. The deep learning module is used to build and train a generative model based on a diffusion model. This model constructs noisy data through a forward diffusion process and trains the model by predicting the noise. In the inference phase, starting from random noise, iterative denoising is performed through the generative model, and gradient correction based on measurement consistency is introduced after each denoising step. The application service module is used to build RESTful API services based on Flask, use Redis to cache trained models, provide a real-time channel estimation interface, and integrate monitoring tools to visualize system performance. The deployment decision module is used to compare the performance metrics of different model versions through an A / B testing framework and make model iteration and deployment decisions based on the statistical test results.