An edge-computing-based data compression and feature extraction method for internet of things devices

Abstract

The present application relates to the field of data processing, and more particularly to a kind of data compression and feature extraction method of Internet of Things equipment based on edge computing.First, based on the obtained multi-source data stream, the multi-source data of uniform structure is obtained;Then, respectively, the structured event type data, discrete enumeration type data and image type data in the multi-source data of uniform structure are compressed by type, the structured numerical type data and time series type data in the multi-source data of uniform structure are compressed by time-frequency coupling entropy compression transformation algorithm, and the compressed integer symbol stream is obtained;At the same time, the multi-source data of uniform structure is subjected to feature extraction, and different type domain features are obtained, and finally, fusion feature vector is obtained based on different type domain features.The technical problems of poor adaptive ability of compression strategy to signal complexity change, low compression efficiency and large information loss under high compression rate in the existing compression process are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method for data compression and feature extraction of Internet of Things (IoT) devices based on edge computing. Background Technology

[0002] With the rapid development of IoT technology, numerous sensor devices continuously generate multi-source and multi-type data, including numerical data, time-series data, event-based data, and image data. This data typically features high sampling frequency, large scale, and significant structural differences, requiring real-time processing and transmission in practical applications. Traditional IoT architectures often employ a "terminal acquisition—cloud processing" approach, directly uploading raw data to the cloud for storage and analysis. However, with the rapid increase in the number of devices and the continuous expansion of data volume, the direct transmission of massive amounts of raw data not only consumes significant network bandwidth but also increases transmission latency, resulting in low compression efficiency and substantial information loss even at high compression rates. This makes it difficult to meet the low latency and high reliability requirements of industrial control, intelligent manufacturing, and real-time monitoring. Summary of the Invention

[0003] This invention provides a method for data compression and feature extraction of IoT devices based on edge computing, in order to solve the problem that existing IoT multi-type data compression at the edge has poor adaptability to changes in signal complexity, resulting in low compression efficiency and significant information loss under high compression ratio conditions.

[0004] The present invention provides a method for data compression and feature extraction of IoT devices based on edge computing, comprising the following steps: S1. Based on the acquired multi-source data stream, perform unified structuring processing to obtain multi-source data with a unified structure; compress the structured event data, discrete enumeration data, and image data in the unified structured multi-source data to obtain compressed event data, discrete enumeration data, and image data; S2. In the data compression process, a time-frequency coupled entropy compression transformation algorithm is introduced to compress structured numerical data and time series data in multi-source data with a unified structure, so as to obtain compressed numerical data and time series data. S3. While performing compression processing, based on multi-source data with a unified structure, preliminary feature functions of the type domain are extracted to obtain features of different type domains; based on the features of different type domains, dimensionality reduction and normalization processing are performed to obtain normalized multi-domain features; based on the normalized multi-domain features, weighted combination is performed to form a fused feature vector.

[0005] Preferably, S2 specifically includes: Structured time series data are analyzed using Fast Fourier Transform, and a fixed window length is calculated. Structured numerical data and time series data are segmented using a fixed window.

[0006] Preferably, S2 specifically includes: The time-domain composite energy of the fixed window is obtained by squaring and summing the data within the fixed window; the data within the same fixed window is converted to the frequency domain by Fourier transform, and the frequency domain energy of the fixed window is obtained by squaring and summing the frequency domain data.

[0007] Preferably, S2 specifically includes: The time-frequency coupling weighting factor is calculated by multiplying the ratio of the time-domain integrated energy to the frequency-domain energy by a fixed length; the time-domain integrated energy of the fixed window is divided by 1 and the frequency-domain energy of the fixed window is added, and then multiplied by the time-frequency coupling weighting factor to obtain the compensation term of the time-domain integrated energy after spectral scale suppression.

[0008] Preferably, S2 specifically includes: By introducing complex spectral coefficients, the square of the absolute value of the complex spectral coefficients is added to the compensation term of the time-domain comprehensive energy after spectral scale suppression to obtain the spectral enhancement weight.

[0009] Preferably, S2 specifically includes: Based on the compensation term after spectral scale suppression of the time-domain integrated energy, the fixed length, and the frequency-domain energy, the normalization factor of the total effective energy after time-frequency coupling is calculated; the ratio of the spectral enhancement weight to the normalization factor of the total effective energy after time-frequency coupling is calculated to obtain the time-frequency coupling normalized spectral weight with a fixed window.

[0010] Preferably, S2 specifically includes: Based on the time-domain integrated energy and frequency-domain energy of a fixed window, the time-domain energy coupling weight and frequency-domain energy coupling weight are calculated respectively; based on the time-frequency coupling normalized spectral weight, time-domain energy coupling weight and frequency-domain energy coupling weight of a fixed window, the time-frequency coupling entropy of the fixed window is calculated.

[0011] Preferably, S2 specifically includes: Based on the time-frequency coupling entropy of a fixed window, an exponential smoothing enhancement term is constructed using an exponential function. Based on the exponential smoothing enhancement term, the time-frequency coupling entropy of the fixed window, and the time-domain integrated energy, the compression adjustment factor corresponding to the fixed window is calculated.

[0012] Preferably, S2 specifically includes: Based on the compression adjustment factor, a nonlinear mapping is performed on the data in the fixed window to obtain the compressed data of the fixed window; based on all the compressed data of the fixed window, a window compression sequence of the fixed window is constructed; all window compression sequences are concatenated in chronological order to obtain the compressed data, including compressed numerical data and compressed time series data.

[0013] The beneficial effects of the technical solution of the present invention are: 1. A categorized compression strategy was adopted for multi-source data with different types of unified structures. This included run-length encoding, differential encoding, and Huffman or Golomb encoding for structured event-based and discrete enumeration-based data, as well as block-based transform coding and entropy coding for structured image data. While preserving the original statistical characteristics of the multi-source data with unified structures across different types, efficient data compression was achieved. This significantly saved bandwidth and storage resources during data storage and transmission at edge computing nodes, while also reducing network load.

[0014] 2. For structured numerical and time-series data, a time-frequency coupled entropy compression transformation algorithm is introduced. Through fixed-length sliding window partitioning, coupled calculation of time-domain comprehensive energy and frequency-domain energy, and nonlinear mapping, the compression intensity is adaptively adjusted. The time-frequency coupled entropy compression transformation algorithm can dynamically control the compression ratio according to the signal complexity, taking into account both the fine preservation of high-complexity windows and the high compression ratio of low-complexity windows, improving the data utilization efficiency of edge computing nodes in real-time processing scenarios, and reducing information loss.

[0015] 3. During the compression process, preliminary feature extraction of the type domain is performed simultaneously. Statistical and time-frequency features are extracted from structured numerical, time-series, event, image, and discrete enumeration data respectively. After dimensionality reduction and normalization, a unified mapping of features from different type domains is achieved. Combined with a linear weighted fusion mechanism with preset weights, the generated fusion feature vector can comprehensively and accurately reflect the operating status of the equipment or the environmental status, providing a reliable feature basis for anomaly detection, status determination, and data uploading. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for data compression and feature extraction of IoT devices based on edge computing, as described in this invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for a data compression and feature extraction method for IoT devices based on edge computing provided by this invention.

[0020] See attached document Figure 1 The diagram illustrates a flowchart of a data compression and feature extraction method for IoT devices based on edge computing, provided by an embodiment of the present invention. The method includes the following steps: S1. Based on the acquired multi-source data stream, perform unified structuring processing to obtain multi-source data with a unified structure; compress the structured event data, discrete enumeration data, and image data in the unified structured multi-source data to obtain compressed event data, discrete enumeration data, and image data.

[0021] At edge computing nodes, multi-source data streams from IoT devices such as cameras and laser scanners are received, including numerical data, time-series data, event-based data, discrete enumeration data, and image data. These multi-source data streams undergo unified structuring processing. Specifically, based on existing technologies such as a unified data description model (e.g., key-value pair structure or TLV encapsulation structure based on field mapping), data serialization mechanisms (e.g., binary serialization or lightweight object representation structure), unified timestamp calibration mechanisms (e.g., system clock synchronization and time standard conversion technology), and device registration information mapping tables, data from different sources and in different formats is standardized. Simultaneously, normalization preprocessing is performed according to data type to eliminate dimensional issues; for example, the maximum-minimum method is used for numerical data. Finally, a unified structured multi-source data is obtained, specifically including structured data units containing device identifiers, timestamps, type labels, and data block content. This unified structured multi-source data includes structured numerical data, structured time-series data, structured event-based data, structured image data, and structured discrete enumeration data. The unified structured processing process uses existing mature technologies, which will not be elaborated here.

[0022] Through the above unified structuring process, multi-source data streams from different sources and in different formats are unified in terms of logical structure and field organization, thus establishing a consistent data organization form for subsequent compression encoding and feature extraction.

[0023] After unified structuring, the multi-source data with the unified structure undergoes corresponding categorized compression based on type labels. Specifically, for structured event-type data and structured discrete enumeration data, run-length encoding is primarily used to compress repetitive state segments, and differential encoding is applied to the event occurrence time intervals. This is then combined with Huffman or Golomb coding for efficient integer compression, resulting in compressed event-type data and compressed discrete enumeration data. For structured image-type data, a block-based transform coding method is employed, such as discrete cosine transform or integer wavelet transform. After quantizing the transform coefficients, entropy coding is used for compression, resulting in compressed image-type data. Through this categorized compression process, data from different sources and with different structures achieves high compression efficiency while preserving their respective statistical characteristics. The final output is a unified compressed integer symbol stream for subsequent storage or transmission.

[0024] S2. In the data compression process, a time-frequency coupled entropy compression transformation algorithm is introduced to compress structured numerical data and time series data in multi-source data with a unified structure, so as to obtain compressed numerical data and time series data.

[0025] In the data compression process, for structured numerical data and structured time-series data, a time-frequency coupled entropy compression transform algorithm is introduced for compression processing to obtain compressed data. The specific implementation process is as follows: First, to ensure the real-time performance of edge computing nodes and address the constraints of limited computing power, structured numerical and time-series data are segmented using a fixed-length sliding window. The purpose of this segmentation technique is to transform globally non-stationary sequences into locally approximately stationary segments, establishing a controllable analysis range for subsequent frequency-domain coupled computations. Specifically, structured time-series data is analyzed using a Fast Fourier Transform to determine the dominant frequency, and four times the ratio of the known sampling frequency to the dominant frequency is used as the fixed length of the sliding window. .

[0026] In the Within a fixed-length sliding window, i.e. the first... Within a fixed window, the time-domain composite energy is first calculated to characterize the amplitude and variation intensity. The result is obtained by summing the squares of the data within the fixed window. The combined energy of the time domain within a fixed window The time-domain integrated energy is used as the basis for subsequent calculations of the time-frequency coupled entropy. Input.

[0027] After obtaining the time-domain composite energy, in order to calculate the frequency-domain energy, the data within the same fixed window are transformed to the frequency domain using the existing Fourier transform method, and the frequency-domain data are then summed by squares to obtain the first energy. Frequency domain energy of a fixed window and integrated energy in the time domain It constitutes the basis of joint energy.

[0028] Based on this, a normalized coupling spectrum probability distribution is constructed: ; in, It is the first A fixed window at frequency Time-frequency coupling normalized spectral weights at the location; Indicates the first Each frequency is determined by Fourier transform; It is a frequency index; It is the first A fixed window at frequency The complex spectrum coefficients at position 1 are determined by Fourier transform and used to represent the 1st... A fixed window at frequency Amplitude and phase information at the location; It is the first A fixed window at frequency Spectral energy at the location; The time-frequency coupling weighting factor is adaptively determined based on the ratio of the combined energy in the time domain to the energy in the frequency domain, and the formula is as follows: , Represents a small constant to prevent the denominator from being zero, such as ; This represents the time-domain composite energy of the i-th fixed window; Indicates the first Frequency domain energy of a fixed window; It is the compensation term for the time-domain integrated energy after spectral scale suppression; This represents uniform and smooth energy in the time domain. Indicates the spectrum enhancement weights; It is the normalization factor of the total effective energy after time-frequency coupling, which represents the global effective energy after superimposing the time-domain uniform smooth energy on the frequency domain energy, and serves as the normalization benchmark.

[0029] The above formula guarantees This forms a probability distribution, providing a basis for entropy calculation.

[0030] Furthermore, the time-frequency coupling entropy is calculated using the following formula: ; in, It is the first The time-frequency coupling complexity index for a fixed window, i.e., time-frequency coupling entropy; It is the time-domain energy coupling weight, used to control the contribution of the time-domain comprehensive energy to the overall time-frequency coupling complexity index. The reference value range is ; It is the frequency domain energy coupling weight, used to control the contribution of frequency domain energy to the overall time-frequency coupling complexity index. The reference value range is ; It represents the spectral information entropy, used to measure the complexity of the frequency structure; It is a time-domain coupling weighted contribution term used to represent the modulation intensity of the time-fluctuation structure in the frequency domain context; It is a frequency-domain coupling weighted contribution term used to represent the modulation intensity of the spectral distribution structure in the time-domain integrated energy environment.

[0031] Furthermore, based on the time-frequency coupling entropy, a compression adjustment factor is constructed, and the specific calculation formula is as follows: ; in, It is the first A compression adjustment factor corresponding to a fixed window is used to control the compression intensity, quantization step size scaling ratio, or amplitude nonlinear mapping intensity. It is an exponential smoothing enhancement term used to suppress miscompression in low-entropy regions and improve stability in high-entropy regions; This is used to measure the weight of the time-frequency coupling complexity index during the compression and regulation process.

[0032] Finally, regarding the first Any of the first fixed windows Data Perform a nonlinear mapping, the specific expression is: ; in, It is the first Any of the first fixed windows Data The compressed data; Indicates the first Any of the first fixed windows One data point; Used to maintain the direction of the symbol; It is the main compression term, used to describe the equivalent magnitude after complexity adjustment; It is an enhancement factor used to prevent small-value data from being excessively weakened after compression; It is a rounding operation.

[0033] Furthermore, forming the first A window compression sequence of a fixed window All window compression sequences are concatenated in chronological order to obtain the compressed data. ,in, Indicates the total number of fixed windows. Compressed data. This includes compressed numerical data and compressed time-series data.

[0034] The compressed event-type data, compressed discrete enumeration-type data, compressed image-type data, compressed numerical-type data, and compressed time-series-type data together constitute the compressed integer symbol stream.

[0035] S3. While performing compression processing, based on multi-source data with a unified structure, preliminary feature functions of the type domain are extracted to obtain features of different type domains; based on the features of different type domains, dimensionality reduction and normalization processing are performed to obtain normalized multi-domain features; based on the normalized multi-domain features, weighted combination is performed to form a fused feature vector.

[0036] While performing compression, edge computing nodes extract preliminary features from multi-source data with a unified structure, obtaining features from different type domains. Specifically, for structured numerical data, existing statistical analysis methods are used to calculate statistical features such as mean, variance, skewness, kurtosis, rate of change, and trend slope as preliminary features; for structured time-series data, existing digital signal processing and statistical analysis methods such as Fast Fourier Transform are used to extract frequency and time domain features such as dominant frequency, spectral energy distribution, zero-crossing rate, peak density, and period length as preliminary features; for structured event-type data, the number of triggers per unit time, average duration, maximum duration, and event density are statistically analyzed as preliminary features; for structured image data, existing OpenCV methods are used to calculate grayscale histogram distribution features, edge density, texture energy, and local contrast as preliminary features; for structured discrete enumeration data, existing counting methods are used to statistically analyze category distribution ratios and frequency of change as preliminary features. All of the above feature extraction processes are based on deterministic mathematical formulas and rule calculations, which are mature technologies in this field and will not be elaborated upon here.

[0037] After obtaining the preliminary features of various data types, the edge computing nodes perform dimensionality reduction on the features of different domains using methods such as principal component analysis, followed by normalization using methods such as minimum-maximum normalization to map them to a unified numerical range, resulting in normalized multi-domain features. Furthermore, the normalized multi-domain features are weighted and combined using existing linear weighted fusion methods based on pre-defined weights, such as those used in Transformer attention mechanisms, to form a fused feature vector. The final fused feature vector serves as a comprehensive expression of the device's operating status or environmental status and can be used for subsequent data uploading, status determination, or anomaly detection.

[0038] In summary, a method for data compression and feature extraction of IoT devices based on edge computing has been developed.

[0039] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0040] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0041] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for data compression and feature extraction of IoT devices based on edge computing, characterized in that, Includes the following steps: S1. Based on the acquired multi-source data stream, perform unified structuring processing to obtain multi-source data with a unified structure; compress the structured event data, discrete enumeration data, and image data in the unified structured multi-source data to obtain compressed event data, discrete enumeration data, and image data; S2. In the data compression process, a time-frequency coupled entropy compression transformation algorithm is introduced to compress structured numerical data and time series data in multi-source data with a unified structure, so as to obtain compressed numerical data and time series data. S3. While performing compression processing, based on multi-source data with a unified structure, preliminary feature extraction of type domains is performed to obtain features of different type domains; based on the features of different type domains, dimensionality reduction and normalization processing are performed to obtain normalized multi-domain features. Based on the normalized multi-domain features, a weighted combination is performed to form a fused feature vector.

2. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 1, characterized in that, S2 specifically includes: Structured time series data are analyzed using Fast Fourier Transform, and a fixed window length is calculated. Structured numerical data and time series data are segmented using a fixed window.

3. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 2, characterized in that, S2 specifically includes: The time-domain composite energy of the fixed window is obtained by squaring and summing the data within the fixed window; the data within the same fixed window is converted to the frequency domain by Fourier transform, and the frequency domain energy of the fixed window is obtained by squaring and summing the frequency domain data.

4. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 3, characterized in that, S2 specifically includes: The time-frequency coupling weighting factor is calculated by multiplying the ratio of the time-domain integrated energy to the frequency-domain energy by a fixed length; the time-domain integrated energy of the fixed window is divided by 1 and the frequency-domain energy of the fixed window is added, and then multiplied by the time-frequency coupling weighting factor to obtain the compensation term of the time-domain integrated energy after spectral scale suppression.

5. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 4, characterized in that, S2 specifically includes: By introducing complex spectral coefficients, the square of the absolute value of the complex spectral coefficients is added to the compensation term of the time-domain comprehensive energy after spectral scale suppression to obtain the spectral enhancement weight.

6. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 5, characterized in that, S2 specifically includes: Based on the compensation term after spectral scale suppression of the time-domain integrated energy, the fixed length, and the frequency-domain energy, the normalization factor of the total effective energy after time-frequency coupling is calculated; the ratio of the spectral enhancement weight to the normalization factor of the total effective energy after time-frequency coupling is calculated to obtain the time-frequency coupling normalized spectral weight with a fixed window.

7. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 6, characterized in that, S2 specifically includes: Based on the time-domain integrated energy and frequency-domain energy of a fixed window, the time-domain energy coupling weight and frequency-domain energy coupling weight are calculated respectively; based on the time-frequency coupling normalized spectral weight, time-domain energy coupling weight and frequency-domain energy coupling weight of a fixed window, the time-frequency coupling entropy of the fixed window is calculated.

8. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 7, characterized in that, S2 specifically includes: Based on the time-frequency coupling entropy of a fixed window, an exponential smoothing enhancement term is constructed using an exponential function. Based on the exponential smoothing enhancement term, the time-frequency coupling entropy of the fixed window, and the time-domain integrated energy, the compression adjustment factor corresponding to the fixed window is calculated.

9. The method for data compression and feature extraction of IoT devices based on edge computing according to claim 8, characterized in that, S2 specifically includes: Based on the compression adjustment factor, a nonlinear mapping is performed on the data in the fixed window to obtain the compressed data of the fixed window; based on all the compressed data of the fixed window, a window compression sequence of the fixed window is constructed; all window compression sequences are concatenated in chronological order to obtain the compressed data, including compressed numerical data and compressed time series data.

Images