A data compression transmission storage method and system based on a radio cloud architecture

Abstract

The application provides a data compression transmission storage method and system based on a radio cloud architecture, and relates to the technical field of data compression transmission storage. The method comprises the following steps: collecting spectrum observation data; performing dynamic modeling and smoothing processing through frequency band division bidirectional optimization and envelope fusion technology to obtain a smooth noise base curve; performing normalization processing on the spectrum observation data based on the smooth noise base curve to obtain standardized spectrum observation data; performing compression coding on the standardized spectrum observation data through a lightweight K-means clustering algorithm to generate a compressed spectrum package; performing structured packaging of an additional parameter identification header information on the compressed spectrum package through a preset trigger mechanism to generate a compressed transmission data frame, and sending the compressed transmission data frame to a cloud platform; decompressing the compressed transmission data frame by the cloud platform to obtain restored smooth spectrum observation data; and performing block storage and management on the restored smooth spectrum observation data based on clustering parameters and in combination with time labels.

Description

Technical Field

[0001] This invention relates to the field of data compression, transmission, and storage technology, and in particular to a data compression, transmission, and storage method and system based on a radio cloud architecture. Background Technology

[0002] Existing radio monitoring systems primarily employ a centralized platform architecture. Remote monitoring nodes are responsible for data acquisition, uploading raw or simply compressed signal data to a central platform for unified storage and analysis. For data compression, common fixed-ratio algorithms (such as ZIP and GZIP) are commonly used to reduce storage and transmission pressure. Data transmission largely relies on 4G / 5G or low-power wide-area communication technologies (such as LoRa and NB-IoT). The central platform uses traditional database or file system storage structures to store all data uniformly and allows for querying via an indexing system.

[0003] However, existing methods, when faced with massive amounts of monitoring data, generally employ universal algorithms with fixed compression ratios. This makes it difficult to simultaneously achieve compression efficiency and the complete preservation of key signal features, resulting in large data transmission delays and high storage costs, failing to meet the real-time and long-term requirements of radio monitoring. Furthermore, remote monitoring nodes lack local intelligent preprocessing capabilities, and the central platform does not incorporate the spatiotemporal characteristics of monitoring data or machine learning methods for hierarchical management and efficient retrieval, leading to low feature data extraction efficiency and hindering the support of diverse and intelligent radio monitoring application scenarios. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a data compression, transmission, and storage method based on a radio cloud architecture. This method addresses the shortcomings of existing methods, which, when faced with massive amounts of monitoring data, generally employ fixed compression ratio algorithms. These methods struggle to simultaneously achieve compression efficiency and the complete preservation of key signal features, resulting in high data transmission latency, high storage costs, and an inability to meet the real-time and long-term requirements of radio monitoring. Furthermore, remote monitoring nodes lack local intelligent preprocessing capabilities, and the central platform fails to integrate the spatiotemporal characteristics of monitoring data with machine learning methods for hierarchical management and efficient retrieval. This leads to low feature data extraction efficiency and an inability to support diverse and intelligent radio monitoring application scenarios.

[0005] A first aspect of this invention provides a data compression, transmission, and storage method based on a radio cloud architecture, comprising: S1: Collect spectrum observation data; S2: By using frequency band bidirectional optimization and envelope fusion technology, the noise floor in the spectrum observation data is dynamically modeled and smoothed to obtain a smoothed noise floor curve; S3: Based on the smoothed noise baseline curve, normalization and outlier removal are performed on the spectrum observation data to obtain standardized spectrum observation data; S4: The standardized spectrum observation data is compressed and encoded using the lightweight Mini Batch K-Means algorithm to generate a compressed spectrum packet; S5: Through a preset triggering mechanism, the compressed spectrum packet is structurally encapsulated with additional parameter identification header information to generate a compressed transmission data frame, and the compressed transmission data frame is sent to the cloud platform; S6: The cloud platform decompresses the received compressed transmission data frames to obtain restored smooth spectrum observation data; S7: Based on the clustering parameters of the lightweight Mini Batch K-Means algorithm, combined with time labels, the restored smooth spectrum observation data is stored in blocks and managed with feature indexes.

[0006] A second aspect of the present invention provides a data compression transmission and storage system based on a radio cloud architecture, comprising: a processor and a memory; The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the data compression transmission and storage method based on a radio cloud architecture as described in the first aspect.

[0007] A third aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the data compression transmission and storage method based on a radio cloud architecture as described in the first aspect.

[0008] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: In this embodiment of the invention, by introducing dynamic modeling and preprocessing, normalization and outlier removal of spectrum data, combined with lightweight clustering compression algorithms and structured encapsulation methods, compression efficiency is significantly improved while retaining key signal features, reducing transmission latency and storage costs, thus meeting the real-time and long-term requirements of radio monitoring systems. Simultaneously, an intelligent triggering mechanism enables efficient uploading of compressed data, and the cloud platform combines clustering parameters and time tags to perform block storage and feature-based index management, constructing an end-cloud collaborative architecture with spatiotemporal awareness and machine learning capabilities. This significantly improves the retrieval efficiency of feature data and enhances the system's support capabilities for diverse radio monitoring application scenarios. Attached Figure Description

[0009] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0010] Figure 1 This is a schematic flowchart of a data compression, transmission, and storage method based on a radio cloud architecture provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a data compression transmission and storage system based on a radio cloud architecture provided in an embodiment of the present invention. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0012] The data compression, transmission, and storage method based on a radio cloud architecture provided by the present invention will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0013] Reference manual attached Figure 1 The diagram illustrates a flowchart of a data compression, transmission, and storage method based on a radio cloud architecture provided by an embodiment of the present invention.

[0014] This invention provides a data compression, transmission, and storage method based on a radio cloud architecture, which may include the following steps: S1: Collect spectrum observation data.

[0015] Specifically, data acquisition: The terminal station collects monitoring data by reusing existing monitoring equipment. The data types include conventional electromagnetic environment data, such as frequency scan spectrum observation data (B_FScan) and panoramic scan spectrum observation (BPScan) data required by the radio standard specification "Technical Specification for Integrated Platform for Ultra-Short Wave Monitoring and Management". The data stream is returned and collected according to the specification requirements.

[0016] It should be noted that in the traditional model, data is directly stored as files or uploaded to a decentralized system via data stream after acquisition. This invention proposes a novel preprocessing method for scanned spectrum observation data (B_FScan), aiming to solve the problems of high noise interference, difficult feature extraction, and low efficiency of subsequent analysis caused by direct storage or streaming of traditional spectrum data. Through bidirectional optimization of frequency bands and envelope fusion technology, it achieves accurate characterization of the noise floor.

[0017] S2: By using frequency band bidirectional optimization and envelope fusion technology, the noise floor in the spectrum observation data is dynamically modeled and smoothed to obtain a smooth noise floor curve.

[0018] Among them, the frequency-band bidirectional optimization technique is a refined processing method for modeling the noise floor in spectrum observation data. Its core idea is to divide the full-band data into multiple sub-bands with consistent physical or statistical characteristics, and independently construct a noise baseline curve within each band. This technique introduces the "lowest 21% signal sampling value" as an initial noise reference line, and combines a 6dB interference judgment threshold with a local 200kHz mean correction strategy to make differentiated adjustments to each sampling point, thereby achieving dynamic adaptation of noise estimation.

[0019] Envelope fusion technology is a key step in enhancing the integrity of the noise floor based on bidirectional noise optimization. It aims to construct envelopes in both frequency-increasing and frequency-decreasing directions and fuse them point-by-point maximum values ​​to form a unified noise envelope curve. This method effectively eliminates interference peak information that may be missed during unidirectional scanning, ensuring the noise model has coverage and robustness across the entire frequency band.

[0020] In one possible implementation, S2 specifically includes: S201: Divide the spectrum observation data into multiple frequency band intervals according to a predefined set of multiple frequency points.

[0021] Optionally, the predefined set of multiple frequency bands specifically includes: , among which, (such as , ,..., This method covers the entire frequency range of the signal and divides the scanned spectrum data into targeted processing intervals. Noise features are extracted independently for each frequency band, avoiding interference from differences in signal characteristics between different frequency bands on the noise floor and improving the targeting of processing.

[0022] S202: Using the lowest 21% signal sampling value as the initial noise base reference line, a 6dB interference judgment threshold and a local frequency band mean correction strategy are introduced to correct the spectrum observation data in each frequency band interval, thus obtaining the dynamic noise base curve.

[0023] In one possible implementation, S202 specifically includes: S2021: Count the frequency of occurrence of each level in the set of signal levels corresponding to each frequency band interval.

[0024] Optionally, the signal level set is: .in, D Represents a set of signal levels. d v Indicates the first v The signal level value at each sampling point v =1,2… m , m This indicates the total number of sampling points.

[0025] S2022: Sort by frequency of occurrence from high to low, and select the lowest level value of the top 21% of cumulative frequency as the initial noise floor reference line:

[0026] in, k Indicates the ranking index. Indicates level The number of times it appears, m Indicates the total number of sampling points. t This indicates the threshold number of voltage levels.

[0027] S2023: Determine whether the signal level at each sampling point is more than 6dB higher than the initial noise floor reference line. If so, select the set of initial noise floor values ​​within a 200kHz range before and after the current sampling point, calculate the corrected noise value, and use the corrected noise value as the actual noise value of the current sampling point. Otherwise, retain the minimum value between the signal level and the initial noise floor as the actual noise value of the current sampling point.

[0028] in, This represents the initial noise basis set within a local area. Represents the arithmetic mean. This represents the set of initial noise baseline reference values ​​within the left and right neighborhoods of the current sampling point.

[0029] Specifically, if the level at a certain sampling point is similar to the initial noise floor... satisfy This triggers local mean correction. The noise floor value within a 200kHz range before and after this point is selected and adjusted in frequency steps. Convert to index range.

[0030] S2024: Combine the actual noise values ​​of each sampling point in frequency order to form a dynamic noise baseline curve.

[0031] In this embodiment of the invention, adaptive modeling and fine-grained restoration of background noise in spectral observation data are achieved by constructing a dynamic noise baseline curve based on frequency statistics and a local correction mechanism. Simultaneously, a 6dB decision threshold and a 200kHz local mean correction mechanism are introduced to dynamically correct abnormal fluctuations, effectively avoiding deviations in noise modeling caused by sudden interference or short-term spikes. Finally, a continuous dynamic noise baseline curve is constructed using frequency as an index, ensuring the results possess spectral smoothness and physical consistency.

[0032] S203: Based on the dynamic noise base curve, construct frequency increasing envelope and frequency decreasing envelope, and fuse the point-by-point maximum values ​​of the frequency increasing envelope and frequency decreasing envelope to generate a noise envelope curve.

[0033] Among them, the frequency-increasing envelope refers to the noise envelope curve constructed by sequentially performing interference determination and local mean correction operations on each sampling point in the dynamic noise base curve in order of frequency from low to high.

[0034] Among them, the frequency decreasing envelope refers to another noise envelope curve constructed by traversing the dynamic noise base curve in reverse order from high to low frequency and applying the same correction strategy as the frequency increasing direction to each sampling point.

[0035] In one possible implementation, S203 specifically includes: S2031: Traverse each sampling point in the dynamic noise floor curve in ascending order of frequency, and correct the noise value of each sampling point by performing 6dB interference judgment and 200kHz local mean correction, thus constructing a frequency-increasing direction envelope curve:

[0036] in, This represents the noise envelope curve in the direction of increasing frequency. Indicates the first frequency in the direction of frequency increase. l The envelope noise value of each sampling point l =1,2… m , m This indicates the total number of sampling points.

[0037] S2032: After reversing the dynamic noise floor curves, perform the same correction logic as in S2031 on each sampling point in descending frequency order to construct a frequency-decreasing direction envelope curve:

[0038] in, This represents the noise envelope curve in the direction of frequency decrease. Indicates the first frequency in the direction of frequency increase.k The envelope noise value of each sampling point k =1,2… m , m This indicates the total number of sampling points.

[0039] S2033: The maximum values ​​at each sampling point in the frequency-increasing and frequency-decreasing direction envelope curves are fused point-by-point to generate a noise envelope curve.

[0040] Among them, table The envelope after fusion is shown in the first i Noise value at each sampling point max This indicates taking the maximum value.

[0041] It should be noted that in spectral data, since interference points may be asymmetrically distributed across certain frequency bands, using only a unidirectional envelope curve construction can easily lead to some high-amplitude interference not being correctly covered or corrected. Therefore, this invention innovatively constructs envelope curves in both frequency-increasing and frequency-decreasing directions, and fuses their maximum values ​​at a point-by-point level to generate a more robust noise envelope curve. Through bidirectional processing, a 6dB judgment and local mean correction are performed in each direction in a "complementary" manner, ensuring that points not fully corrected in one direction still have a chance to be accurately corrected in the other. The maximum value of both is taken during final fusion, ensuring that the resulting curve possesses "completeness, comprehensive coverage, and anti-interference capability," providing a more representative and fault-tolerant full-band noise benchmark for subsequent sliding smoothing and normalization.

[0042] S204: Apply a moving average to the noise envelope curve to obtain a smoothed noise base curve:

[0043] in, Indicates the first i The smoothed noise floor value corresponding to each sampling point Indicates the first k The noise envelope curve values ​​at each sampling point w Indicates the window length.

[0044] Moving average processing is a commonly used signal smoothing technique aimed at eliminating short-term fluctuations or high-frequency noise in data to enhance the continuity and stability of the curve. In this scheme, moving average processing is applied to the fused noise envelope curve. By calculating the mean within a fixed window length before and after each sampling point, local abrupt changes and jitter are effectively suppressed, making the final noise base curve smoother, more stable, and consistent with the noise distribution trend in the actual electromagnetic environment.

[0045] In this embodiment of the invention, through the collaborative innovation of frequency band fine processing, dynamic noise reference, bidirectional envelope fusion and smoothing optimization, the original spectrum data is transformed into structured noise feature data, which effectively improves the accuracy of noise floor characterization, reduces the computational complexity of subsequent data analysis, and provides key technical support for the efficient storage, rapid analysis and accurate application of ultra-shortwave monitoring data.

[0046] S3: Based on the smooth noise baseline curve, normalization and outlier removal are performed on the spectrum observation data to obtain standardized spectrum observation data.

[0047] The normalization process maps the spectral data to a uniform interval based on a smoothed noise basis, eliminating amplitude differences between different frequency bands or time periods and improving the consistency of feature representation. Outlier removal is based on the 3σ principle, removing extreme points far from the main noise distribution to ensure the stability of the input samples and the quality of clustering.

[0048] In one possible implementation, S3 specifically includes: S301: Estimate the normalization boundary based on the smoothed noise baseline curve, perform interval mapping normalization on the spectrum observation data, and obtain normalized spectrum observation data.

[0049] Optionally, normalization can be achieved using min-max normalization.

[0050] S302: Based on the 3σ rule, outlier sampling points that are far from the noise floor distribution in the normalized spectrum observation data are removed to obtain standardized spectrum observation data.

[0051] In this embodiment of the invention, a smoothed noise baseline curve is first used as a reference to uniformly map the original spectral data to a standard interval, effectively eliminating the amplitude inconsistency caused by different frequency bands or time periods, and giving each sample a uniform feature scale. Subsequently, outlier sampling points far from the main noise distribution are removed based on the 3σ rule to avoid extreme values ​​interfering with subsequent clustering and compression. Overall, this step not only improves the clustering separability and compression efficiency of the data, but also enhances the system's robustness and generalization ability in complex signal environments, serving as a crucial bridge connecting preprocessing and machine learning compression.

[0052] S4: The standardized spectrum observation data is compressed and encoded using the lightweight Mini Batch K-Means algorithm to generate compressed spectrum packets.

[0053] The Mini Batch K-Means algorithm is a highly efficient clustering algorithm designed for large-scale data, and is a lightweight improvement on the classic K-Means algorithm. Its core idea is to introduce a mini-batch sampling mechanism, using only a subset of samples (rather than the entire dataset) to update cluster centers in each iteration, thus significantly reducing computational complexity and memory usage. In each iteration, the algorithm randomly selects a small batch of samples from the original data, calculates their Euclidean distance to the current cluster centers, assigns them to classes, and then incrementally updates the cluster centers based on the mean of each class. Because it processes only a small amount of data in each round, Mini Batch K-Means exhibits significant computational speedup, real-time performance, and resource adaptability.

[0054] It should be noted that the core of this invention lies in providing a lightweight cloud-based collaborative spectrum data compression transmission and processing method. By modifying the K-Means algorithm for specific scenarios (achieving high compression with low computing power consumption through adaptive parameter configuration, iterative optimization, and feature encoding), and combining it with cloud-based collaborative design, the entire chain of optimization is achieved, from lightweight compression at the terminal station to efficient transmission to cloud-based adaptive decompression and feature-based storage.

[0055] In one possible implementation, S4 specifically includes: S401: Initialize the algorithm parameters of the lightweight Mini Batch K-Means algorithm, including the number of clusters, batch, maximum number of iterations, and initial cluster centers.

[0056] Specifically, Elbow's rule combined with 5-fold cross-validation is used to calculate the sum of squared clustering errors (SSE) for different numbers of clusters K, and the K value with the smallest SSE is selected as the optimal number of clusters. The batch size B is set according to the terminal station's memory capacity and real-time requirements to ensure that the time taken for a single clustering calculation does not exceed 10ms. An initial number of iterations T is set, and an early stopping mechanism is introduced: clustering is terminated early when the change in cluster centers is less than 0.001 in three consecutive iterations. Initial cluster centers are selected from standardized spectral observation data based on the K-Means++ strategy to improve the clustering convergence speed.

[0057] S402: Using the initialized algorithm parameters, perform clustering iteration on the standardized spectrum observation data, update the cluster center of each sample in the standardized spectrum observation data, and form a set of cluster centers.

[0058] In one possible implementation, S402 specifically includes: S4021: Randomly sample B samples from standardized spectrum observation data to form a small batch subset.

[0059] S4022: Calculate the Euclidean distance between each sample in the mini-batch subset and the initial cluster center, and assign each sample to the category of the nearest cluster center:

[0060] in, Indicates sample With cluster center The Euclidean distance between them M Indicates the number of feature dimensions. Indicates the first i The sample at the th m Values ​​in each dimension Indicates the first j The cluster centers at the in m The values ​​can be taken in each dimension.

[0061] S4023: Based on the classification results of each sample in the mini-batch subset, calculate the mean of the feature vectors for samples belonging to the same cluster center category, and update the corresponding cluster centers based on the mean of the feature vectors.

[0062] in, Indicates the updated number j Cluster center vectors This indicates that the current mini-batch subset is classified into the first... j A sample set of classes.

[0063] S4024: If the number of iterations reaches the maximum number of iterations or if the change in all cluster centers is less than the preset change in 3 consecutive iterations, stop the iteration and output the updated set of cluster centers.

[0064] In this embodiment of the invention, by introducing a mini-batch random sampling and incremental center update mechanism, the efficiency and terminal adaptability of clustering computation are significantly improved. Processing only B samples in each iteration not only greatly reduces computational resource consumption but also ensures that the algorithm can run efficiently on resource-constrained terminal devices.

[0065] S403: Based on the cluster center numbers in the cluster center set, encode each sample with a category index to obtain a category index sequence.

[0066] S404: According to the preset format, the clustering parameters and category index sequence are structured and encapsulated to generate a compressed spectrum package.

[0067] It should be noted that the default format is to create a new compressed data frame header. The frame header compression includes "parameter header + cluster centers + category index sequence", where the parameter header records... K , BNumber of iterations (for easy cloud adaptation and decompression), cluster center is K 32-bit floating-point data (4 K byte), index sequence is n × b byte( n For the sample size, b (Index bit length / 8), the new data format ensures both data integrity and lightweight design.

[0068] In this embodiment of the invention, by introducing the lightweight Mini Batch K-Means algorithm, efficient compression and structured encoding of spectrum data are achieved. This not only significantly reduces the computational burden and storage requirements of the terminal equipment, but also significantly improves data transmission efficiency, laying a stable and efficient foundation for subsequent cloud decompression and feature-based management.

[0069] S5 uses a preset triggering mechanism to perform structured encapsulation of compressed spectrum packets with additional parameter identifier header information, generates compressed transmission data frames, and sends the compressed transmission data frames to the cloud platform.

[0070] In one possible implementation, S5 specifically includes: S501: Determine whether the acquired compressed spectrum packet meets the preset transmission trigger condition. If yes, proceed to the next step. Otherwise, continue determining the transmission trigger condition.

[0071] It should be noted that the default triggering mechanism is either a sample number threshold trigger: data upload is triggered when the cumulative number of compressed samples on the terminal reaches 100–500. Alternatively, it can be a time interval trigger: even if the sample number has not been reached, an upload can be forcibly triggered every 5–10 minutes.

[0072] S502: Construct the message header information of the compressed spectrum packet, and set the cloud algorithm identifier field in the message header to identify the radio cloud compression storage method used for the data.

[0073] S503: Records the algorithm parameters used for data parsing in the message header. The algorithm parameters include the number of clusters and the batch size.

[0074] S504: Perform Base64 encoding on the clustered compressed data in the compressed spectrum packet to generate a Base64 format message body.

[0075] Base64 encoding is an encoding method that converts binary data into ASCII character format and is often used to securely transmit non-text data in text protocols.

[0076] S505: Combines and encapsulates the message header and message body to construct a complete compressed transmission data frame.

[0077] S506: The compressed data frames are sent to the cloud platform via the network for cloud parsing and storage.

[0078] In this embodiment of the invention, by setting a reasonable preset trigger mechanism, the real-time performance of data transmission and the resource consumption of the terminal station are effectively balanced, meeting the monitoring timeliness requirements while avoiding the computational burden caused by frequent communication. Combined with a structured encapsulation data frame design, algorithm identifiers, clustering parameters, and compressed data are uniformly integrated and Base64 encoded to improve transmission stability. This ensures that the compressed data has good integrity, resolvability, and protocol compatibility during transmission to the cloud platform via the wireless link, laying a reliable foundation for subsequent efficient decompression and storage in the cloud.

[0079] S6: The cloud platform decompresses the received compressed transmission data frames to obtain restored smooth spectrum observation data.

[0080] In one possible implementation, S6 specifically includes: S601: Perform protocol verification and integrity parsing on the received compressed transmission data frame to determine if there are any errors in the frame header format or incomplete data bodies. If so, request the end station to retransmit. Otherwise, proceed to the next step.

[0081] S602: Extract the cloud algorithm identifier field and parameter header information from the compressed transmission data frame, and identify the core parameter configuration of the clustering compression algorithm.

[0082] It should be noted that the cluster center C is extracted from the compressed file. final Stored in a cloud-based machine learning model library with a 24-hour expiration period, subsequent data from the same site and batch can be directly accessed from the library after decompression. final This eliminates the need to repeatedly receive cluster center data, reducing transmission redundancy and avoiding duplicate transmissions.

[0083] S603: Parse and extract the final set of cluster centers and the sequence of category indices contained in the compressed frame.

[0084] S604: Retrieve the corresponding cluster center vector from the cluster center set according to the category index sequence, and combine each cluster center vector as an approximate reconstruction value of the spectrum sample in sequence to obtain the restored smooth spectrum observation data.

[0085] Optionally, the category index value is used to locate the corresponding vector in the cluster center set, realizing the mapping and reconstruction of the sample feature vector. Based on Cfinal, the category index is mapped to the cluster center value of the original data dimension.

[0086] It should be noted that during the restoration process, a decompression deviation correction model trained based on historical data will be invoked to fine-tune the restored spectrum samples in order to improve decompression accuracy.

[0087] In this embodiment of the invention, by performing protocol verification, parameter parsing, and cluster decoding on compressed transmission data frames in the cloud, efficient restoration and accuracy assurance of spectrum observation data are effectively achieved. Centralized extraction and caching of cluster centers avoids redundant data transmission and reduces communication burden. Combined with the rapid mapping between category index sequences and cluster centers, the original sample features can be efficiently restored. Simultaneously, a bias correction model trained on historical data is introduced to fine-tune and compensate for the restoration results, significantly improving the accuracy and consistency of the decompressed data, providing an accurate and stable spectrum foundation for subsequent data analysis and feature retrieval.

[0088] S7: Based on the clustering parameters of the lightweight Mini Batch K-Means algorithm, combined with time labels, the restored smooth spectrum observation data is stored in blocks and managed with characteristic indexes.

[0089] Specifically, the decompressed data is divided into blocks (each 1-5MB in size) according to the dimension of "endpoint ID - data date - number of machine learning clusters K". The block name includes the clustering feature identifier (e.g., "ID_20240520_K12") and stored in the cloud monitoring data platform to achieve a strong correlation between data and clustering features. The cloud database records each data block's "block ID, endpoint ID, data time range, machine learning parameters (K / B), storage path, and cluster center summary", supporting rapid retrieval of similar feature data based on clustering parameters, improving retrieval efficiency by over 60%. Nearly 30 days of "hot data" (high-frequency queries) is stored on SSDs (solid-state drives), while "cold data" older than 30 days (only archived, quickly decompressable via historical queries) is migrated to low-cost archive storage, reducing storage costs by 80% compared to traditional full-volume SSD storage.

[0090] In this embodiment of the invention, by combining clustering parameters and time tags to perform block storage and feature-based index management of the restored spectrum data, not only is a high correlation between the data and its machine learning features achieved, but retrieval efficiency and storage utilization are also significantly improved. Simultaneously, storing recent "hot data" on high-speed SSDs and migrating historical "cold data" to low-cost storage media effectively balances access performance and economy, enabling the overall system to have both high responsiveness and low long-term operating costs.

[0091] Reference manual attached Figure 2 The diagram shows a schematic representation of a data compression transmission and storage system based on a radio cloud architecture provided by an embodiment of the present invention.

[0092] This invention provides a data compression transmission and storage system 20 based on a radio cloud architecture, including: a processor 201 and a memory 202; The memory 202 stores programs or instructions that can run on the processor 201. When the program or instructions are executed by the processor 201, they implement the steps of the above-described data compression transmission and storage method based on a radio cloud architecture and achieve the same technical effect. To avoid repetition, the present invention will not elaborate further.

[0093] It should be understood that the processor 201 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0094] It should also be understood that the memory 202 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0095] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0096] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0099] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0101] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0102] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] This invention provides a readable storage medium comprising: storing a program or instructions on the readable storage medium, wherein when the program or instructions are executed by a processor, the program or instructions implement the steps of the above-described data compression transmission and storage method based on a radio cloud architecture, and can achieve the same technical effect. To avoid repetition, this invention will not elaborate further.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A data compression, transmission, and storage method based on a radio cloud architecture, characterized in that, include: S1: Collect spectrum observation data; S2: By using frequency band bidirectional optimization and envelope fusion technology, the noise floor in the spectrum observation data is dynamically modeled and smoothed to obtain a smooth noise floor curve; S2 specifically includes: S201: Divide the spectrum observation data into multiple frequency band intervals according to a predefined set of multiple frequency points; S202: Using the lowest 21% signal sampling value as the initial noise floor reference line, a 6dB interference judgment threshold and a local frequency band mean correction strategy are introduced to correct the spectrum observation data in each frequency band interval to obtain the dynamic noise floor curve. S203: Based on the dynamic noise base curve, construct a frequency increasing envelope and a frequency decreasing envelope, and fuse the point-by-point maximum values ​​of the frequency increasing envelope and the frequency decreasing envelope to generate a noise envelope curve. S204: Apply a moving average to the noise envelope curve to obtain the smoothed noise base curve: in, Indicates the first i The smoothed noise floor value corresponding to each sampling point Indicates the first k The noise envelope curve values ​​at each sampling point w Indicates the window length; S3: Based on the smoothed noise baseline curve, normalization and outlier removal are performed on the spectrum observation data to obtain standardized spectrum observation data; S3 specifically includes: S301: Estimate the normalization boundary based on the smoothed noise basis curve, and perform interval mapping normalization on the spectrum observation data to obtain normalized spectrum observation data; S302: Based on the 3σ rule, remove the abnormal sampling points in the normalized spectrum observation data that are far from the noise floor distribution to obtain the standardized spectrum observation data; S4: The standardized spectrum observation data is compressed and encoded using the lightweight Mini Batch K-Means algorithm to generate a compressed spectrum packet; S4 specifically includes: S401: Initialize the algorithm parameters of the lightweight Mini Batch K-Means algorithm, wherein the algorithm parameters include the number of clusters, batch, maximum number of iterations, and initial cluster centers; S402: Using the initialized algorithm parameters, perform clustering iteration on the standardized spectrum observation data, update the cluster center of each sample in the standardized spectrum observation data, and form a set of cluster centers; S403: Based on the cluster center numbers in the cluster center set, each sample is coded with a category index to obtain a category index sequence; S404: The clustering parameters and the category index sequence are structured and encapsulated according to a preset format to generate the compressed spectrum package; S5: Through a preset triggering mechanism, the compressed spectrum packet is structurally encapsulated with additional parameter identification header information to generate a compressed transmission data frame, and the compressed transmission data frame is sent to the cloud platform; S6: The cloud platform decompresses the received compressed transmission data frames to obtain restored smooth spectrum observation data; S7: Based on the clustering parameters of the lightweight Mini Batch K-Means algorithm, combined with time labels, the restored smooth spectrum observation data is stored in blocks and managed with feature indexes. Specifically, S7 involves dividing the decompressed data into blocks according to the dimensions of terminal station ID, data date, and number of machine learning clusters. The block name contains the clustering feature identifier, and the data is stored in the cloud monitoring data platform.

2. The data compression, transmission, and storage method based on a radio cloud architecture according to claim 1, characterized in that, S202 specifically includes: S2021: Count the frequency of occurrence of each level in the signal level set corresponding to each frequency band interval; S2022: Sort by frequency of occurrence from high to low, and select the lowest level value of the top 21% of cumulative frequency as the initial noise floor reference line: in, k Indicates the ranking index. Indicates level The number of times it appears, m Indicates the total number of sampling points. t Indicates the threshold number of voltage levels; S2023: Determine whether the signal level of each sampling point is higher than the initial noise floor reference line by more than 6dB; if so, select the set of initial noise floor values ​​within a 200kHz range before and after the current sampling point, calculate the corrected noise value, and use the corrected noise value as the actual noise value of the current sampling point; otherwise, retain the minimum value between the signal level and the initial noise floor as the actual noise value of the current sampling point. in, This represents the initial noise basis set within a local area. Represents the arithmetic mean. This represents the set of initial noise floor reference values ​​within the left and right neighborhoods of the current sampling point; S2024: Combine the actual noise values ​​of each sampling point in frequency order to form the dynamic noise base curve.

3. The data compression, transmission, and storage method based on a radio cloud architecture according to claim 1, characterized in that, S203 specifically includes: S2031: Traverse each sampling point in the dynamic noise baseline curve in ascending frequency order, and correct the noise value of each sampling point by performing 6dB interference judgment and 200kHz local mean correction, thereby constructing a frequency-increasing direction envelope curve: in, This represents the noise envelope curve in the direction of increasing frequency. Indicates the first frequency in the direction of frequency increase. l The envelope noise value of each sampling point l =1,2… m , m Indicates the total number of sampling points; S2032: After reversing the dynamic noise baseline curves, perform the same correction logic as in S2031 on each sampling point in descending frequency order to construct a frequency decreasing direction envelope curve: in, This represents the noise envelope curve in the direction of frequency decrease. Indicates the first frequency in the direction of frequency increase. k The envelope noise value of each sampling point k =1,2… m , m Indicates the total number of sampling points; S2033: The maximum values ​​at each sampling point in the frequency-increasing direction envelope curve and the frequency-decreasing direction envelope curve are fused point-by-point to generate the noise envelope curve. Among them, table The envelope after fusion is shown in the first i Noise value at each sampling point max This indicates taking the maximum value.

4. The data compression, transmission, and storage method based on a radio cloud architecture according to claim 1, characterized in that, Specifically, S402 includes: S4021: Randomly sample B samples from the standardized spectrum observation data to form a small batch subset; S4022: Calculate the Euclidean distance between each sample in the mini-batch subset and the initial cluster center, and classify each sample into the category of the nearest cluster center: in, Indicates sample With cluster center The Euclidean distance between them M Indicates the number of feature dimensions. Indicates the first i The sample at the th m Values ​​in each dimension Indicates the first j The cluster centers at the in m Values ​​in each dimension; S4023: Based on the classification results of each sample in the mini-batch subset, calculate the mean of the feature vectors for samples belonging to the same cluster center category, and update the corresponding cluster centers based on the mean of the feature vectors. in, Indicates the updated number j Cluster center vectors, This indicates that the current mini-batch subset is classified into the first... j A sample set of classes; S4024: When the number of iterations reaches the maximum number of iterations or in three consecutive iterations, if the change in all cluster centers is less than the preset change, stop the iteration and output the updated set of cluster centers.

5. The data compression, transmission, and storage method based on a radio cloud architecture according to claim 1, characterized in that, S5 specifically includes: S501: Determine whether the collected compressed spectrum packet meets the preset transmission trigger condition; if yes, proceed to the next step; otherwise, continue to determine the transmission trigger condition. S502: Construct the message header information of the compressed spectrum packet, and set a cloud algorithm identifier field in the message header to identify the radio cloud compression storage method used for the data; S503: The message header records the algorithm parameters used for data parsing, including the number of clusters and the batch size; S504: Perform Base64 encoding on the clustered compressed data in the compressed spectrum packet to generate a message body in Base64 format; S505: Combine and encapsulate the message header and the message body to construct a complete compressed transmission data frame; S506: The compressed transmission data frame is sent to the cloud platform via the network for cloud parsing and storage.

6. The data compression, transmission, and storage method based on a radio cloud architecture according to claim 1, characterized in that, S6 specifically includes: S601: Perform protocol verification and integrity parsing on the received compressed transmission data frame to determine whether there is a frame header format error or incomplete data body; if so, request the end station to retransmit; otherwise, proceed to the next step. S602: Extract the cloud algorithm identifier field and parameter header information from the compressed transmission data frame, and identify the core parameter configuration of the clustering compression algorithm; S603: Parse and extract the final cluster center set and category index sequence contained in the compressed frame; S604: Retrieve the corresponding cluster center vector from the cluster center set according to the category index sequence, and sequentially combine each of the cluster center vectors as approximate reconstruction values ​​of the spectrum sample to obtain the restored smooth spectrum observation data.

7. A data compression, transmission, and storage system based on a radio cloud architecture, characterized in that, include: Processor and memory; The memory stores programs or instructions that can run on the processor, which, when executed by the processor, implement the steps of the data compression transmission and storage method based on a radio cloud architecture as described in any one of claims 1 to 6.

Images