Data compression method and device, equipment and storage medium

Abstract

The invention provides a data compression method, apparatus and device, and a storage medium. The method comprises the steps of obtaining an original data stream transmitted through a serial port; determining the data type of the original data flow according to the original data flow; determining a corresponding target compression algorithm according to the data type; compressing the original data stream according to the target compression algorithm to obtain compressed data; the defects of low efficiency and even data expansion during data abrupt change in a fixed compression mode are effectively overcome, and the data transmission efficiency and the bandwidth utilization rate of serial port communication in a resource limited environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of serial port data compression, and in particular to a data compression method, apparatus, device and storage medium. Background Technology

[0002] Existing serial port data compression methods generally use fixed compression algorithms to process all transmitted data uniformly, lacking a mechanism for sensing and responding to real-time changes in data content characteristics. This compression strategy suffers a significant drop in efficiency when faced with drastic or sudden changes in data characteristics. For example, a temperature sensor outputs a slowly changing numerical sequence under normal conditions, which can be efficiently processed by a time-series compression algorithm. However, when an abnormal event occurs (such as device overheating), the data may suddenly shift to a high-frequency, highly fluctuating pattern, making the original compression method inadequate. In this case, the fixed algorithm not only fails to achieve effective compression but may even lead to an increase in the compressed data volume. Therefore, existing fixed compression algorithms result in the system consuming a large amount of transmission bandwidth in dynamic environments, making it difficult to meet the requirements of efficient and reliable transmission in resource-constrained serial communication scenarios. Summary of the Invention

[0003] This application provides a data compression method, apparatus, device, and storage medium, which solves the problem of low efficiency caused by the diversity of data types and the limited computing power of edge nodes in serial port transmission, and provides an efficient and lightweight data transmission method for scenarios such as industrial IoT and smart homes.

[0004] Firstly, this application provides a data compression method, including:

[0005] Obtain the raw data stream transmitted via serial port;

[0006] The data type of the original data stream is determined based on the original data stream;

[0007] Determine the corresponding target compression algorithm based on the data type;

[0008] The original data stream is compressed according to the target compression algorithm to obtain compressed data.

[0009] In one or more possible embodiments, the data type includes numeric, text, and binary types.

[0010] In one or more possible embodiments, determining the corresponding target compression algorithm based on the data type includes:

[0011] When the data type of the original data stream is numerical, the target compression algorithm is determined to be the first compression algorithm, which is used to process data sequences that are continuous and correlated.

[0012] When the data type of the original data stream is text, the target compression algorithm is determined to be the second compression algorithm, which is determined based on a dictionary encoding mechanism;

[0013] When the data type of the original data stream is binary, the target compression algorithm is determined to be the third compression algorithm, which is a general lossless compression algorithm.

[0014] In one or more possible embodiments, when the target compression algorithm is determined to be a first compression algorithm, the step of compressing the original data stream according to the target compression algorithm includes:

[0015] Based on the differences between adjacent data in the data sequence, a difference sequence is obtained;

[0016] For any one of the differences, based on the magnitude of the difference, select a corresponding bit width from multiple preset bit widths for encoding to obtain the encoded value;

[0017] When it is determined that there are consecutively repeated encoded values ​​in the encoded value sequence corresponding to the difference sequence, the consecutively repeated encoded values ​​are compressed into a combination of the number of repetitions and the encoded value.

[0018] In one or more possible embodiments, when the target compression algorithm is determined to be a second compression algorithm, the step of compressing the original data stream according to the target compression algorithm includes:

[0019] Determine the string in the original data stream and match the string with a preset dictionary and a dynamic dictionary;

[0020] Based on the matching results, determine to replace the string with the matching encoding;

[0021] The preset dictionary contains predefined characters and a fixed-length short code corresponding to each predefined character; the dynamic dictionary is used to acquire and store newly appearing temporary characters during the compression process, and dynamically generate corresponding index codes for the temporary characters according to the LZW algorithm.

[0022] In one or more possible embodiments, determining the data type of the original data stream based on the original data stream includes:

[0023] Feature extraction is performed on the original data stream to obtain the feature parameters of the original data stream;

[0024] Based on the feature parameters and the pre-trained classification model, the data type corresponding to the original data stream is determined.

[0025] In one or more possible embodiments, after obtaining the compressed data, the method further includes:

[0026] Generate a metadata header based on the data type and the target compression algorithm;

[0027] The metadata header and the compressed data are concatenated and encapsulated into a data packet, which is then sent to the gateway.

[0028] Secondly, this application provides a data compression apparatus, the apparatus comprising:

[0029] The data acquisition module is used to acquire the raw data stream transmitted via the serial port;

[0030] A data type confirmation module is used to determine the data type of the original data stream based on the original data stream.

[0031] The target compression algorithm confirmation module is used to determine the corresponding target compression algorithm based on the data type.

[0032] The compressed data acquisition module is used to compress the original data stream according to the target compression algorithm to acquire compressed data.

[0033] Thirdly, this application provides an electronic device, the electronic device comprising:

[0034] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods in the first aspect.

[0035] Fourthly, this application provides a computer storage medium storing a computer program for causing a computer to perform any of the methods described in the first aspect.

[0036] This application provides a data compression method, apparatus, device, and storage medium that, by analyzing data stream characteristics in real time and intelligently switching compression algorithms, effectively overcomes the shortcomings of fixed compression modes in terms of low efficiency or even data expansion when data changes abruptly, and significantly improves the data transmission efficiency and bandwidth utilization of serial communication in resource-constrained environments. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application, and do not constitute an undue limitation of this application.

[0038] Figure 1This is a schematic diagram of an application scenario provided according to an embodiment;

[0039] Figure 2 This is a flowchart of a data compression method provided according to an embodiment;

[0040] Figure 3 This is a schematic diagram of a data compression device according to an embodiment;

[0041] Figure 4 This is a schematic diagram of an electronic device according to an embodiment;

[0042] Figure 5 This is a schematic diagram of a computer-readable storage medium provided according to an embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0044] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0045] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0046] For ease of understanding, the terms used in the embodiments of this invention are explained below:

[0047] Serial port (serial communication port): A communication interface that transmits data bit by bit. Common types include RS-232, RS-485, etc.

[0048] Edge Controller: An embedded processing device deployed on an edge node, integrating data processing, algorithm execution and other functions. It typically uses chips such as STM32, Raspberry Pi, and Pico. In this application, it is used to receive sensor data in real time, perform feature recognition, hierarchical compression and metadata encapsulation, and realize localized and efficient processing of edge data.

[0049] Sensor: A device that converts physical quantities (such as temperature, pressure, sound, light intensity) or chemical quantities (such as gas concentration) into transmittable and processable electrical signals (usually digital signals).

[0050] Existing serial port data compression methods generally use fixed compression algorithms to process all transmitted data uniformly, lacking a mechanism for sensing and responding to real-time changes in data content characteristics. This compression strategy suffers a significant drop in efficiency when faced with drastic or sudden changes in data characteristics. For example, a temperature sensor outputs a slowly changing numerical sequence under normal conditions, which can be efficiently processed by a time-series compression algorithm. However, when an abnormal event occurs (such as device overheating), the data may suddenly shift to a high-frequency, highly fluctuating pattern, making the original compression method inadequate. In this case, the fixed algorithm not only fails to achieve effective compression but may even lead to an increase in the compressed data volume. Therefore, existing fixed compression algorithms result in the system consuming a large amount of transmission bandwidth in dynamic environments, making it difficult to meet the requirements of efficient and reliable transmission in resource-constrained serial communication scenarios.

[0051] To address the aforementioned problems, this application proposes a solution based on dynamic data type awareness and adaptive compression strategy. By analyzing data stream characteristics in real time and intelligently switching compression algorithms, it effectively overcomes the shortcomings of fixed compression modes, such as low efficiency or even data expansion, when data changes drastically. This significantly improves the data transmission efficiency and bandwidth utilization of serial communication in resource-constrained environments.

[0052] For ease of understanding, the data compression method provided in this application embodiment will be described in detail below with reference to the accompanying drawings:

[0053] like Figure 1The diagram illustrates an application scenario of a data compression method provided in this application. The diagram includes multiple sensors, an RS485 communication interface, an edge controller, and an industrial gateway. The sensors, acting as the data acquisition source, transmit the raw data stream to the edge controller via the RS485 bus. The edge controller has a built-in data feature recognition and hierarchical compression module. First, it performs real-time parsing and feature extraction on the input raw data stream to identify its data type and structural features. Then, based on the recognition results, it adaptively selects the optimal target compression algorithm and performs hierarchical compression processing, effectively reducing data volume while retaining key information. Finally, it encapsulates the compressed data stream and sends it to the gateway via the RS485 interface. The gateway decompresses the compressed data stream to restore the data content, then converts the decompressed data into a protocol format suitable for upper-layer systems or network transmission. Finally, the data is uploaded to a higher-level monitoring platform, cloud server, or data center via the edge controller or industrial gateway, or it can be used for local real-time control and decision-making.

[0054] It should be noted that the sensors involved in the embodiments of this application can be various commonly used industrial sensors such as temperature, pressure, and flow. The RS485 communication interface is suitable for long-distance, interference-resistant transmission in industrial environments. The edge controller and industrial gateway, as edge computing nodes, have certain data caching, protocol conversion, and device control capabilities. Figure 1 Although the description uses a single edge controller and industrial gateway as an example, those skilled in the art should understand that the architecture can be extended to scenarios where multiple controllers, gateways, or distributed nodes work together. Figure 1 The units shown can be integrated into the same hardware device or distributed among different devices, communicating via wired or wireless means; in addition, data storage and caching can be located at the edge layer, gateway layer or cloud, and the specific deployment method can be flexibly adjusted according to actual application needs.

[0055] Although Figure 1 While the network and server are not explicitly specified, it is understood that the compressed data stream can be accessed through various means such as Industrial Ethernet, 5G, and LoRa to enter a wider network environment, and ultimately transmitted to a server or cloud platform for storage and further analysis. The layered compression and processing mechanism proposed in this application is not only applicable to industrial IoT scenarios, but can also be extended to other fields that require efficient processing of large amounts of sensor data, such as smart cities, environmental monitoring, and energy management, which will not be elaborated upon here.

[0056] This application provides a data compression method, specifically as follows: Figure 2 As shown, the edge controller applied in this application includes:

[0057] Step 201: Obtain the raw data stream transmitted via the serial port;

[0058] In one or more possible embodiments, the raw data stream described above may come from a sensor, which is connected to the edge controller as a raw data stream provider to continuously output physical quantity data such as temperature and pressure, or device status information, and then send it to the edge controller via an RS485 communication interface.

[0059] Step 202: Determine the data type of the original data stream based on the original data stream described above;

[0060] In one or more possible embodiments, after receiving the raw data stream, the edge controller performs feature extraction on the received raw data to obtain feature parameters of the raw data stream; based on the feature parameters and a pre-trained classification model, it determines the data type corresponding to the raw data stream; after receiving the raw data stream transmitted via the RS485 bus, the edge controller first initiates a data preprocessing and feature analysis process to obtain feature parameters of the raw data stream. These feature parameters include, but are not limited to: printable character ratio, ASCII byte ratio, number of zero-value bytes, byte entropy value, number of numeric characters (i.e., the number of bytes with ASCII codes between 0x30 and 0x39), average byte value, maximum byte value, and whether it is a valid UT. Features such as F-8 encoding format, together, constitute the feature vector of the data stream, effectively characterizing its statistical distribution, regularity, and semantic structure. Taking data sent by a temperature sensor as an example, when the edge device receives the raw data stream reported by the temperature sensor, it performs feature extraction on the raw data stream. If the data is a slowly changing temperature value sequence, its features are usually characterized by a high proportion of numeric characters, low byte entropy, fewer zero bytes, and a relatively stable range of byte values. If there is an abnormally high-frequency jump caused by device overheating, the entropy value increases, the numerical fluctuation intensifies, and the feature vector will change significantly. The extracted feature parameters will be organized into a structured vector and input into a pre-trained classification model, such as the Random Forest used in this application. Random Forest (RFR) models are trained on a large amount of labeled sample data and can accurately output corresponding type identifiers based on input features. For example, 0x01 represents numerical data and 0x02 represents text data. The type identifier can determine the data type of the current raw data stream, thus providing a decision basis for subsequent adaptive selection of compression algorithms. It is worth noting that in addition to the Random Forest model, other machine learning methods such as Support Vector Machine (SVM), Naive Bayes, or Lightweight Neural Networks can also be used for this classification task. The specific model selection can be adjusted according to the actual computing power and recognition accuracy requirements of the equipment.

[0061] Step 203: Determine the corresponding target compression algorithm based on the above data types;

[0062] In one or more possible embodiments, sensors (such as pressure sensors, temperature sensors, etc.) collect physical quantities or device status information in a timed (e.g., every 100 milliseconds) or event-triggered (e.g., when the measured value exceeds a preset threshold) manner. The data generated by the sensors can generally be divided into three types: numerical, text, and binary. Numerical data is usually represented as a continuous or discrete floating-point sequence with high temporal correlation, such as [25.1, 25.2, 25.3, 26.8] output by a temperature sensor, with an accuracy of ±0.1℃. Text data generally refers to log information or status reports generated during device operation, such as "Device_001:WARNING:Temperature exceeds 35℃". This type of data has obvious semantic structure and character repetition characteristics. Binary data includes unstructured byte streams such as firmware upgrade packages and raw waveforms collected by sensors, which usually lack explicit statistical regularities.

[0063] Based on the identified data type, the target compression algorithm is adaptively determined: if the original data stream is numerical, the first compression algorithm is selected. This first compression algorithm is used to process numerical sequences with continuity and high correlation, such as a combination of Delta coding and run-length encoding (RLE), or a lightweight temporal compression method; if the data type is text, the second compression algorithm is used, usually based on dictionary encoding mechanisms (such as the LZ77 / LZ78 series algorithms), which can effectively utilize repeated words and pattern redundancy in the text; if the data is identified as binary, the third compression algorithm is activated, often using general lossless compression methods, such as the DEFLATE algorithm, which can still maintain good compression efficiency in unknown or mixed data structures. Through the above type-aware compression strategy, the system can achieve optimal compression results in different types of sensor data streams, improving bandwidth utilization and transmission reliability.

[0064] Step 204: Compress the original data stream according to the target compression algorithm to obtain compressed data.

[0065] In one or more possible embodiments, when the target compression algorithm is determined to be the first compression algorithm, the compression of the original data stream according to the target compression algorithm includes: obtaining a difference sequence based on the difference between adjacent data in the data sequence; for any one of the differences, selecting a corresponding bit width from multiple preset bit widths for encoding based on the magnitude of the difference to obtain an encoded value; when it is determined that there are continuously repeated encoded values ​​in the encoded value sequence corresponding to the difference sequence, compressing the continuously repeated encoded values ​​into a combination of the repetition count and the encoded value; the first compression algorithm is specifically used to process numerical data sequences, such as continuous floating-point data collected by sensors such as temperature and pressure in an industrial environment. The compression process first calculates the difference between adjacent values ​​in the data sequence to generate a difference sequence. Then, for each difference in the difference sequence, selecting a suitable bit width from multiple preset bit widths for encoding based on its magnitude to obtain an encoded value. Finally, if continuously repeated encoded values ​​are detected in the encoded value sequence, these continuously repeated segments are compressed into a combination of the repetition count and the corresponding encoded value to further reduce the amount of data.

[0066] In one or more possible embodiments, taking the numerical sequence output by an industrial temperature sensor as an example, the original data is [25.1, 25.2, 25.3, 26.8]. First, the adjacent differences are calculated to obtain the difference sequence [0.1, 0.1, 1.5]. Then, dynamic bit width allocation is performed: for small and stable differences (such as 0.1), 5 bits can be selected for encoding and stored as 0b00010. However, for suddenly increasing differences (such as 1.5), since it exceeds the representation range of 5 bits, ... If the value exceeds the preset threshold range, the system automatically switches to 16-bit floating-point encoding, represented as 0x3F8C0000. Next, the Run-Length Encoding (RLE) mechanism detects that the first two 5-bit encoded values ​​are the same and compresses them into [2, 0b00010], indicating that the value appears twice consecutively. Finally, the compressed data is [25.1 encoded value 2, 0b00010, 0x3F8C0000], which only requires four bytes, while the original four floating-point values ​​require 16 bytes.

[0067] Traditional industrial sensor data is often transmitted via serial port without compression or using only a single compression algorithm, which cannot adapt to dynamic changes in data. This application proposes a method based on dynamically selecting compression algorithms according to data characteristics, significantly improving both compression and transmission efficiency. For numerical data, the dynamic bit-width allocation strategy adjusts the encoding bit depth in real time according to data fluctuations. For example, 5-bit encoding is used for small differences, and 16-bit encoding is used for abrupt changes (large differences). During stable sensor data phases, such as when temperature fluctuations remain within ±0.1℃, even if abrupt changes occur, such as a sudden temperature rise due to equipment failure, the compression ratio decrease can still be controlled within 5%. This effectively overcomes the significant reduction in compression efficiency caused by fixed bit-width encoding when data characteristics change. The method addresses several issues. First, the dynamic bit-width allocation mechanism reduces the overhead of encoding outliers. Traditional compression schemes often assign a uniform bit width (e.g., 32 bits) to all differences, while this method only uses 16-bit encoding for mutation values, reducing storage requirements. Second, the run-length encoding compresses consecutively repeating differences into a concise representation of "repetition count + value". For example, 20 consecutive identical differences can be compressed into 2 bytes, significantly reducing the amount of data transmitted. Overall, the memory footprint of the edge controller can be reduced by about 40%, for example, from 1KB to 600 bytes, and the computational cost is reduced by 30%. This enables the stable operation of efficient compression algorithms in embedded systems with limited memory and computing power, resolving the contradiction that traditional compression technologies are difficult to deploy in resource-constrained environments.

[0068] In one or more possible embodiments, when the target compression algorithm is determined to be the second compression algorithm, the compression of the original data stream according to the target compression algorithm includes: determining the string in the original data stream, matching the string with a preset dictionary and a dynamic dictionary; determining to replace the string with a matching code based on the matching result; wherein the preset dictionary contains predefined characters and a fixed-length short code corresponding to each predefined character; the dynamic dictionary is used to acquire and store newly appearing temporary characters during the compression process, and dynamically generate corresponding index codes for the temporary characters according to the LZW algorithm; taking an application case in the field of smart homes as an example, the data transmitted by the communication module of smart home appliances usually includes both numerical data (such as temperature, humidity, etc.) and textual data (such as device operation logs); the edge controller first performs feature recognition on the received data, determines the data type, and then performs layered compression processing. For numerical data, it adopts a method similar to... The same compression algorithm as the aforementioned embodiments, namely a combination of Delta encoding, Run-Length Encoding (RLE), and dynamic bit-width allocation, is used. For text data, a dual-dictionary co-coding algorithm is employed for compression. This algorithm specifically includes: a pre-built dictionary in ROM for storing high-frequency system keywords (such as "WARNING", "ERROR", etc.), with a fixed encoding length of 4 bits, ranging from 0x00 to 0x0F; and a dynamic dictionary maintained in RAM for real-time learning and storage of temporarily appearing fields (such as the device number "Device_001"), and dynamically generating index codes according to the LZW algorithm. For example, the log text "WARNING:Device_001" can be compressed into 0x01 (representing "WARNING"), 0x02 (representing the colon ":"), and 0x80 (pointing to the index corresponding to "Device_001" in the dynamic dictionary), thereby achieving significant compression.

[0069] In existing smart home systems, sensors and edge controllers typically do not perform data compression and directly transmit raw data streams. This application introduces the aforementioned compression method, which significantly improves compression efficiency with minimal computational resources. Especially for text data compression, the traditional LZW algorithm relies entirely on the dynamic update of the global dictionary and cannot distinguish between high-frequency fixed words and low-frequency temporary fields, resulting in limited compression efficiency. However, the dual-dictionary collaborative mechanism adopted in this application compresses high-frequency system words (such as "WARNING") from 7 bytes to 1 byte using a pre-set ROM dictionary, while the dynamic RAM dictionary compresses temporary fields (such as the 20-byte device ID "Device_001") into a 5-byte index, achieving a compression rate of 75%. The overall text log compression ratio has been improved from 1.5:1 in the traditional LZW algorithm to 3:1, effectively solving the problem of low compression efficiency caused by the coexistence of high-frequency words and temporary fields in mixed text types. When the compressed data (e.g., "Device_001:WARNING" is compressed from 20 bytes to 3 bytes) is transmitted through the RS485 bus, the transmission time is reduced by about 80%, which not only reduces the pressure on network bandwidth, but also improves the system response speed and overall energy efficiency, making it very suitable for resource-constrained smart home environments.

[0070] In one or more possible embodiments, after acquiring compressed data, the process further includes: generating a metadata header based on the data type and the target compression algorithm; concatenating the metadata header with the compressed data to encapsulate a data packet and sending it to the gateway; after acquiring compressed data, a data encapsulation step is also required. First, a metadata header is generated based on the data type and the target compression algorithm used. The metadata header typically contains key information such as a data type identifier, a compression algorithm identifier, and the original data length. The data type identifier is used to distinguish between numeric, text, or binary data; the compression algorithm identifier is used to determine the compression method used, such as 0x02 representing a combination of Delta encoding and Run-Length Encoding (RLE); the original data length field records the size of the data before compression, such as 0x04 representing 4 floating-point numbers; after generating the metadata header, the metadata header is concatenated with the compressed data according to a predefined fixed format to encapsulate a complete data transmission packet. For example, a complete transmission packet may appear as: 0x01 (data type identifier - numeric), 0x0... The data packet contains 2 (compression algorithm identifier - Delta + RLE), 0x04 (original data length - 4 values), 2 (RLE repetition count), 0b00010 (encoded difference), 0x3F8C0000 (mutation value encoding), and an encoded value of 25.1 (or 26.8; the encoded values ​​of 25.1 and 26.8 are used to obtain the complete original data stream during decoding. The floating-point data 25.1 or 26.8 can use 32-bit floating-point encoding or other encoding methods, which can be determined according to the actual application). The total length is 7 bytes. This data packet is then sent to the industrial gateway for processing.

[0071] In one or more possible embodiments, the industrial gateway, as a key node between the edge and the cloud, integrates a dynamic decompression unit and a protocol conversion unit. The gateway receives layered compressed data packets from the edge controller through a wired communication interface (such as RS485) and performs decompression and protocol adaptation operations. The dynamic decompression unit is responsible for parsing the received data packets: first, extracting the metadata header, reading the compression algorithm number, and calling the corresponding decompression algorithm accordingly. For example, if the algorithm number is 0x02, the inverse operation of the Delta and RLE combination algorithm is executed; if it is a text data compression algorithm number, a dual dictionary restoration mechanism is activated; if it is a general binary compression algorithm, the DEFLATE decompression algorithm is called, and the decompression process completely restores the original data stream, making it consistent with the data directly output by the sensor in terms of format and content, such as restoring it to the original floating-point sequence or text string; the protocol conversion unit further converts the decompressed original data into a standard protocol format suitable for IoT communication, such as MQTT or CoAP; the protocol conversion unit also supports integrated secure transmission mechanisms, such as TLS or DTLS encryption, to ensure the security of data transmission in public networks. Finally, the converted data is uploaded to the cloud monitoring platform or data center via wireless methods such as 4G, 5G or Wi-Fi, completing the entire data processing flow.

[0072] This application provides a data compression method aimed at solving the challenge of efficient transmission of various data types, such as industrial sensor data and equipment logs, under the limited computing power of edge devices. The specific process is as follows: a data feature recognition module analyzes the type of input data (e.g., numerical, text, or binary) in real time and dynamically selects the optimal compression strategy accordingly. For continuous numerical data, a combination of differential coding (Delta), run-length encoding (RLE), and dynamic bit-width allocation is used to effectively eliminate adjacent redundancy in the data sequence. For text data, a coding mechanism that combines a pre-set static dictionary with a dynamic dictionary enables rapid matching and compression of high-frequency words. For random binary data, Huffman coding and the LZ77 algorithm are integrated to achieve a balance between compression efficiency and computational overhead. A good balance is achieved between compression and data processing. The compression result, along with metadata such as the type identifier and algorithm ID describing the compression parameters, is encapsulated into a data packet, facilitating self-adaptive decompression by the receiving end. This solves the limitations of traditional general-purpose compression algorithms in terms of poor data type adaptability and high computational consumption. Practical tests on typical embedded platforms such as STM32 show that the average compression ratio is improved by more than 2.5 times, and the transmission latency is reduced by 40%. This effectively solves the inefficiency problem caused by the diversity of data types and the limited computing power of edge nodes in serial port transmission, providing an efficient and lightweight data transmission solution for scenarios such as industrial IoT and smart homes.

[0073] Corresponding to the above-mentioned data compression method, the present invention also proposes a data compression apparatus, specifically as follows: Figure 3 As shown, it includes:

[0074] The data acquisition module 301 is used to acquire the raw data stream transmitted via the serial port;

[0075] The data type confirmation module 302 is used to determine the data type of the original data stream based on the original data stream.

[0076] The target compression algorithm confirmation module 303 is used to determine the corresponding target compression algorithm based on the above data type.

[0077] The compressed data acquisition module 304 is used to compress the original data stream according to the target compression algorithm to acquire compressed data.

[0078] Since the device embodiments of the present invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated in the present invention.

[0079] This application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described data compression method.

[0080] like Figure 4 As shown, the device includes a processor 401, a memory 402, a communication interface 403, and a bus 404. The processor 401, memory 402, and communication interface 403 are interconnected via the bus 404.

[0081] Processor 401 is configured to read instructions from memory 402 and execute them, so that at least one processor can perform the data compression method provided in the above embodiments.

[0082] The memory 402 is used to store various instructions and programs for the data compression method provided in the above embodiments.

[0083] Bus 404 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0084] Processor 401 can be a central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), or any combination of CPU, NP, and GPU. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0085] In addition, this application also provides a computer-readable storage medium, such as Figure 5 As shown, the computer storage medium stores a computer program that is used to cause the computer to perform any of the methods described in the above embodiments.

[0086] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) 501 and / or cache memory 502, and may further include read-only memory (ROM) 503.

[0087] The memory may also include a program / utility 505 having a set (at least one) of program modules 504, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data compression method characterized by, The method comprises the following steps: acquiring an original data stream transmitted through a serial port; determining a data type of the original data stream according to the original data stream; determining a target compression algorithm corresponding to the data type according to the data type; compressing the original data stream according to the target compression algorithm to acquire compressed data.

2. The method of claim 1, wherein, The data type comprises a numerical type, a text type and a binary type.

3. The method of claim 2, wherein, The step of determining the target compression algorithm corresponding to the data type according to the data type comprises the following steps: when the data type of the original data stream is the numerical type, determining that the target compression algorithm is a first compression algorithm, and the first compression algorithm is used to process a data sequence with continuity and correlation; when the data type of the original data stream is the text type, determining that the target compression algorithm is a second compression algorithm, and the second compression algorithm is determined based on a dictionary coding mechanism; when the data type of the original data stream is the binary type, determining that the target compression algorithm is a third compression algorithm, and the third compression algorithm is a general lossless compression algorithm.

4. The method of claim 3, wherein, When the target compression algorithm is determined to be the first compression algorithm, the step of compressing the original data stream according to the target compression algorithm comprises the following steps: obtaining a difference value sequence according to the difference values between adjacent data in the data sequence; for any one of the difference values, selecting a corresponding bit width from a plurality of preset bit widths according to the size of the any one difference value to encode and acquire an encoded value; when there are continuously repeated encoded values in the encoded value sequence corresponding to the difference value sequence, compressing the continuously repeated encoded values into a combination of the number of repetitions and the encoded values.

5. The method of claim 3, wherein, When the target compression algorithm is determined to be the second compression algorithm, the step of compressing the original data stream according to the target compression algorithm comprises the following steps: determining a string in the original data stream, and matching the string with a preset dictionary and a dynamic dictionary; determining to replace the string with a matched code according to the matching result. The preset dictionary comprises predefined characters and fixed-length short codes corresponding to each predefined character; and the dynamic dictionary is used to acquire and store a temporary character newly appearing in a compression process, and dynamically generate an index code corresponding to the temporary character according to an LZW algorithm.

6. The method of claim 1, wherein, The step of determining the data type of the original data stream according to the original data stream comprises the following steps: performing feature extraction on the original data stream to acquire a feature parameter of the original data stream; determining the data type corresponding to the original data stream according to the feature parameter and a pre-trained classification model.

7. The method of any of claims 1-6, wherein, After acquiring the compressed data, the method further comprises the following steps: generating a metadata header according to the data type and the target compression algorithm; splicing and packaging the metadata header and the compressed data into a data packet, and sending the data packet to a gateway.

8. A data compression device, characterized by, The device comprises: a data acquisition module configured to acquire an original data stream transmitted through a serial port; a data type confirmation module configured to determine a data type of the original data stream according to the original data stream; a target compression algorithm confirmation module configured to determine a target compression algorithm corresponding to the data type according to the data type; and a compression module configured to compress the original data stream according to the target compression algorithm to acquire compressed data. The compressed data acquisition module is configured to acquire compressed data by compressing the original data stream according to the target compression algorithm.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, and the computer program is used to enable a computer to perform the method of any one of claims 1-7.

Images