Efficient data compression and transmission method based on computing power host management operation platform

Abstract

The invention discloses an efficient data compression and transmission method based on a computing power host management operation platform, and relates to a large-scale data concurrent management technology. Comprising the steps of 1, integrating a lightweight data feature analyzer at a Kafka producer end, extracting feature indexes of data streams in real time through the data feature analyzer, and generating compression strategy parameters according to the feature indexes, and 2, performing feature recognition on structured data and unstructured data, extracting data features, and generating compression strategy parameters according to the compression strategy parameters. And step 3, based on data characteristics, CPU load of producers / consumers and network bandwidth prediction, constructing a multi-objective optimization model, dynamically selecting a compression algorithm and a compression level, selecting compression strategy parameters according to the compression algorithm, introducing a compression income ratio index, calculating a ratio of compression rate increase to compression time consumption, and performing compression strategy optimization. And preferentially selecting a compression algorithm with a high gain ratio to perform data compression and transmission.

Description

Technical Field

[0001] The present invention discloses an efficient data compression and transmission method based on a computing power host management and operation platform, which relates to the technology of large-scale data concurrent management. Background Art

[0002] As a core component of a distributed stream processing platform, Apache Kafka is widely used in scenarios of high throughput and low latency data pipelines. However, there are performance bottlenecks in the following scenarios. For example, there is a contradiction between compression efficiency and real-time performance: Kafka producers (Producers) natively support compression algorithms such as Gzip and Snappy, but static compression strategies cannot dynamically adjust according to data characteristics, such as text, binary, and time-series data, resulting in increased compression time or insufficient compression ratio; waste of network resources: Data transmission between Kafka Brokers and consumers (Consumers) usually relies on fixed compression strategies and does not combine real-time network status, such as bandwidth fluctuations and congestion levels, to dynamically optimize transmission parameters, resulting in high latency or packet loss; end-to-end resource overhead: After the producer compresses the data, the consumer needs to decompress it. If the data needs to be forwarded multiple times, such as cross-cluster replication, repeated decompression / compression operations significantly increase CPU and memory consumption. However, existing improvement schemes, such as hierarchical compression or multi-protocol compatibility, still have problems of poor dynamic adaptability and insufficient end-to-end coordination. Summary of the Invention

[0003] Aiming at the problems of the existing technology, the present invention provides an efficient data compression and transmission method based on a computing power host management and operation platform, which has the characteristics of strong generality and simple implementation, and has broad application prospects.

[0004] The specific solution proposed by the present invention is as follows:

[0005] The present invention provides an efficient data compression and transmission method based on a computing power host management and operation platform, including:

[0006] Step 1: Integrate a lightweight data feature analyzer at the Kafka producer side, and use the data feature analyzer to extract the feature metrics of the data stream in real time, and generate compression strategy parameters according to the feature metrics.

[0007] Step 2: Identify the features of structured and unstructured data, and extract the data features.

[0008] Step 3: Based on data features, producer / consumer CPU load, and network bandwidth prediction, construct a multi-objective optimization model, dynamically select the compression algorithm and compression level, and select compression strategy parameters according to the compression algorithm. Among them, a compression benefit ratio index is introduced to calculate the ratio of the compression rate improvement to the compression time, and a compression algorithm with a high benefit ratio is preferentially selected for data compression and transmission.

[0009] Furthermore, in step 1 of the efficient data compression and transmission method based on the computing power host management and operation platform, it includes:

[0010] Integrate a custom interceptor at the Kafka producer side to implement a lightweight data feature analyzer. The lightweight data feature analyzer provides the ProducerInterceptor interface, inserts analysis logic in the onSend method, extracts feature metrics from the data stream. The feature metrics include basic feature metrics and advanced feature metrics. The basic feature metrics include message size, number of fields, and null value rate. The advanced feature metrics include field type distribution and regular matching conditions. Generate compression strategy parameters according to the feature metrics.

[0011] Furthermore, in step 2 of the efficient data compression and transmission method based on the computing power host management and operation platform, it includes:

[0012] Use predefined rules or patterns for feature recognition of structured data, where check whether the data conforms to the predefined pattern, verify the field types of the data, and check whether the fields meet the constraints;

[0013] For unstructured data, use natural language processing (NLP) methods to perform lexical analysis, syntactic analysis, and semantic analysis on text data to extract keywords and entities. Use computer vision methods to perform edge detection, feature point extraction, and object detection on image data. Use speech recognition to extract voiceprint features from audio data. Use time series analysis methods to extract action categories, scenes, and motion trajectories from video data.

[0014] Furthermore, in step 3 of the efficient data compression and transmission method based on the computing power host management and operation platform, dynamically select a compression algorithm, including:

[0015] Select a lossless compression algorithm, and the lossless compression algorithms include the Gzip algorithm, the Zstandard algorithm, and the LZ4 algorithm, or select a lossy compression algorithm, and the lossy compression algorithms include the JPEG algorithm and the MP3 algorithm.

[0016] The present invention also provides an efficient data compression and transmission device based on the computing power host management and operation platform, including a data feature dynamic perception module, a feature extraction module, and a compression strategy dynamic decision-making module.

[0017] The data feature dynamic perception module integrates a lightweight data feature analyzer at the Kafka producer side, and extracts the feature metrics of the data stream in real time through the data feature analyzer, and generates compression strategy parameters according to the feature metrics.

[0018] The feature extraction module performs feature recognition on structured data and unstructured data, and extracts data features.

[0019] Based on data characteristics, producer / consumer CPU load, and network bandwidth prediction, the compression strategy dynamic decision-making module constructs a multi-objective optimization model, dynamically selects a compression algorithm and compression level, and selects compression strategy parameters according to the compression algorithm. Among them, a compression benefit ratio index is introduced to calculate the ratio of compression rate improvement to compression time consumption, and a compression algorithm with a high benefit ratio is preferentially selected for data compression and transmission.

[0020] Furthermore, the data characteristic dynamic perception module of the high-efficiency data compression and transmission device based on the computing power host management and operation platform integrates a custom interceptor at the Kafka producer side to implement a lightweight data characteristic analyzer. The lightweight data characteristic analyzer provides the ProducerInterceptor interface, inserts analysis logic in the onSend method, extracts characteristic indicators from the data stream, and the characteristic indicators include basic characteristic indicators and advanced characteristic indicators. The basic characteristic indicators include message size, number of fields, and null value rate, and the advanced characteristic indicators include field type distribution and regular matching conditions. Compression strategy parameters are generated according to the characteristic indicators.

[0021] Furthermore, the feature extraction module of the high-efficiency data compression and transmission device based on the computing power host management and operation platform uses predefined rules or patterns for feature recognition of structured data, where it checks whether the data conforms to the predefined pattern, verifies the field types of the data, and checks whether the fields meet the constraints;

[0022] For unstructured data, use natural language processing (NLP) methods to perform lexical analysis, syntactic analysis, and semantic analysis on text data to extract keywords and entities, use computer vision methods to perform edge detection, feature point extraction, and object detection on image data, use speech recognition to extract voiceprint features from audio data, and use time series analysis methods to extract action categories, scenes, and motion trajectories from video data.

[0023] Furthermore, the compression strategy dynamic decision-making module of the high-efficiency data compression and transmission device based on the computing power host management and operation platform dynamically selects a compression algorithm, including:

[0024] Select a lossless compression algorithm, where the lossless compression algorithms include the Gzip algorithm, the Zstandard algorithm, and the LZ4 algorithm, or select a lossy compression algorithm, where the lossy compression algorithms include the JPEG algorithm and the MP3 algorithm.

[0025] The beneficial effects of the present invention are:

[0026] Using the method of the present invention saves bandwidth. The dynamic compression strategy improves the compression rate by 20%-50% compared with the static compression, and reduces the cross-network transmission cost;

[0027] Optimize end-to-end latency: By chunking and path selection, reduce transmission jitter caused by network congestion, and the average latency is reduced by 30%;

[0028] Improve resource utilization: Zero-copy decompression and cache reuse reduce CPU and memory occupancy, and the resource overhead is reduced by 40%. Brief Description of the Drawings

[0029] Figure 1 It is a schematic flowchart of the method of the present invention. Detailed Embodiments

[0030] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments given are not intended to limit the present invention.

[0031] Embodiment 1

[0032] The present invention provides an efficient data compression and transmission method based on a computing power host management and operation platform, including: Step 1: Integrate a lightweight data feature analyzer at the Kafka producer side, and use the data feature analyzer to extract the feature indicators of the data stream in real time, and generate compression policy parameters according to the feature indicators.

[0033] Among them, in Step 1, it may specifically include:

[0034] Integrate a custom interceptor at the Kafka producer side to implement a lightweight data feature analyzer. The lightweight data feature analyzer provides a ProducerInterceptor interface, inserts analysis logic in the onSend method, extracts feature indicators from the data stream. The feature indicators include basic feature indicators and advanced feature indicators. The basic feature indicators include message size, number of fields, null value rate, and the advanced feature indicators include field type distribution, regular matching conditions, and generate compression policy parameters according to the feature indicators.

[0035] Step 2: Perform feature recognition on structured data and unstructured data, and extract data features.

[0036] Among them, in Step 2, it may specifically include:

[0037] Use predefined rules or patterns for feature recognition of structured data, where check whether the data conforms to the predefined pattern, verify the field types of the data, and check whether the fields meet the constraints;

[0038] For unstructured data, use natural language processing (NLP) methods to perform lexical analysis, syntactic analysis, and semantic analysis on text data, extract keywords and entities, use computer vision methods to perform edge detection, feature point extraction, and object detection on image data, use speech recognition to extract voiceprint features from audio data, and use time series analysis methods to extract action categories, scenes, and motion trajectories from video data.

[0039] Step 3: Based on data characteristics, producer / consumer CPU load, and network bandwidth prediction, construct a multi-objective optimization model, dynamically select a compression algorithm and compression level, and select compression strategy parameters according to the compression algorithm. Among them, introduce a compression benefit ratio index, calculate the ratio of compression rate improvement to compression time consumption, and preferentially select a compression algorithm with a high benefit ratio for data compression and transmission.

[0040] The dynamic selection of the compression algorithm in Step 3 may include:

[0041] Select a lossless compression algorithm, where the lossless compression algorithms include the Gzip algorithm, Zstandard algorithm, Brotli algorithm, and LZ4 algorithm, or select a lossy compression algorithm, where the lossy compression algorithms include the JPEG algorithm and MP3 algorithm.

[0042] Embodiment 2

[0043] The present invention also provides an efficient data compression and transmission device based on a computing power host management and operation platform, including a data feature dynamic perception module, a feature extraction module, and a compression strategy dynamic decision module.

[0044] The data feature dynamic perception module integrates a lightweight data feature analyzer at the Kafka producer side, and uses the data feature analyzer to extract the feature indicators of the data stream in real time, and generate compression strategy parameters according to the feature indicators.

[0045] The feature extraction module performs feature recognition on structured and unstructured data, and extracts data features.

[0046] The compression strategy dynamic decision module constructs a multi-objective optimization model based on data characteristics, producer / consumer CPU load, and network bandwidth prediction, dynamically selects a compression algorithm and compression level, and selects compression strategy parameters according to the compression algorithm. Among them, introduce a compression benefit ratio index, calculate the ratio of compression rate improvement to compression time consumption, and preferentially select a compression algorithm with a high benefit ratio for data compression and transmission.

[0047] For the information interaction, execution process, etc. among the modules in the above device, since they are based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention, and will not be elaborated here.

[0048] Similarly, the advantages of the device of the present invention are:

[0049] Save bandwidth. The dynamic compression strategy can improve the compression ratio by 20% - 50% compared with the static compression, reducing the cross-network transmission cost.

[0050] Optimize end-to-end latency: By chunking and path selection, reduce the transmission jitter caused by network congestion, and the average latency is reduced by 30%.

[0051] Improve resource utilization: Zero-copy decompression and cache reuse reduce CPU and memory occupancy, and the resource overhead is reduced by 40%.

[0052] It should be noted that not all steps and modules in the above processes and device structures are necessary. Some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities respectively, or some components in multiple independent devices may be jointly implemented.

[0053] The above embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims

1. An efficient data compression and transmission method based on a computing power host management and operation platform, characterized in that including: Step 1: Integrate a lightweight data feature analyzer at the Kafka producer side. The data feature analyzer extracts the feature metrics of the data stream in real time and generates compression strategy parameters based on the feature metrics. Step 2: Conduct feature recognition on structured and unstructured data to extract data features. Step 3: Based on data features, producer / consumer CPU load, and network bandwidth prediction, build a multi-objective optimization model to dynamically select the compression algorithm and compression level, and select compression strategy parameters according to the compression algorithm. Among them, introduce the compression benefit ratio index, calculate the ratio of compression rate improvement to compression time consumption, and preferentially select the compression algorithm with a high benefit ratio for data compression and transmission.

2. The high - efficiency data compression and transmission method based on the computing power host management and operation platform according to claim 1, characterized in that In Step 1, it includes: Integrate a custom interceptor at the Kafka producer side to implement a lightweight data feature analyzer. The lightweight data feature analyzer provides the ProducerInterceptor interface, inserts analysis logic in the onSend method, extracts feature metrics from the data stream. The feature metrics include basic feature metrics and advanced feature metrics. The basic feature metrics include message size, number of fields, null value rate, and the advanced feature metrics include field type distribution, regular expression matching conditions, and generate compression strategy parameters based on the feature metrics.

3. An efficient data compression and transmission method based on a computing power host management and operation platform according to claim 1, characterized in that In Step 2, it includes: Use predefined rules or patterns for feature recognition of structured data, where check whether the data conforms to the predefined pattern, verify the field types of the data, and check whether the fields meet the constraints. For unstructured data, use natural language processing (NLP) methods to perform lexical analysis, syntactic analysis, and semantic analysis on text data to extract keywords and entities, use computer vision methods to perform edge detection, feature point extraction, and object detection on image data, use speech recognition to extract voiceprint features from audio data, and use time series analysis methods to extract action categories, scenes, and motion trajectories from video data.

4. An efficient data compression and transmission method based on a computing power host management and operation platform according to claim 1, characterized in that Dynamically select the compression algorithm in Step 3, including: Select a lossless compression algorithm, and the lossless compression algorithms include the Gzip algorithm, Zstandard algorithm, and LZ4 algorithm, or select a lossy compression algorithm, and the lossy compression algorithms include the JPEG algorithm, MP3 algorithm.

5. An efficient data compression and transmission device based on a computing power host management and operation platform, characterized in that including a data feature dynamic perception module, a feature extraction module, and a compression strategy dynamic decision-making module. The data feature dynamic perception module integrates a lightweight data feature analyzer at the Kafka producer side, extracts the feature metrics of the data stream in real time through the data feature analyzer, and generates compression strategy parameters based on the feature metrics. The feature extraction module conducts feature recognition on structured and unstructured data to extract data features. The compression strategy dynamic decision-making module builds a multi-objective optimization model based on data features, producer / consumer CPU load, and network bandwidth prediction, dynamically selects the compression algorithm and compression level, and selects compression strategy parameters according to the compression algorithm. Among them, introduce the compression benefit ratio index, calculate the ratio of compression rate improvement to compression time consumption, and preferentially select the compression algorithm with a high benefit ratio for data compression and transmission.

6. The high-efficiency data compression and transmission device based on the computing power host management and operation platform according to claim 5, wherein the data feature is dynamic The perception module integrates a custom interceptor at the Kafka producer side to implement a lightweight data feature analyzer. The lightweight data feature analyzer provides the ProducerInterceptor interface, inserts analysis logic in the onSend method, extracts feature metrics from the data stream. The feature metrics include basic feature metrics and advanced feature metrics. The basic feature metrics include message size, number of fields, and null value rate. The advanced feature metrics include field type distribution and regular matching conditions. Compression strategy parameters are generated based on the feature metrics.

7. The efficient data compression and transmission device based on the computing power host management and operation platform according to claim 5 is characterized by feature extraction The module uses predefined rules or patterns for feature recognition of structured data, where it checks whether the data conforms to the predefined pattern, verifies the field types of the data, and checks whether the fields meet the constraints. For unstructured data, natural language processing (NLP) methods are used to perform lexical analysis, syntactic analysis, and semantic analysis on text data to extract keywords and entities. Computer vision methods are used to perform edge detection, feature point extraction, and object detection on image data. Speech recognition is used to extract voiceprint features from audio data. Time series analysis methods are used to extract action categories, scenes, and motion trajectories from video data.

8. An efficient data compression and transmission device based on a computing power host management and operation platform according to claim 5, characterized in that The compression strategy dynamic decision module dynamically selects compression algorithms, including: Selecting a lossless compression algorithm, which includes the Gzip algorithm, the Zstandard algorithm, and the LZ4 algorithm, or selecting a lossy compression algorithm, which includes the JPEG algorithm and the MP3 algorithm.

Images