A high-frequency data compression management system and method based on artificial intelligence

Through the high-frequency data compression management system based on artificial intelligence, historical data and real-time monitoring are analyzed, and early warning models are built, which solves the problem of high-frequency data transmission quality detection in smart stores, and realizes efficient and automated data compression transmission, avoiding information loss and Transmission delay.

CN118075144BActive Publication Date: 2024-11-19EXANDS INFORMATION TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410180255.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-18
Publication Date
2024-11-19
Estimated Expiration
2044-02-18

AI Technical Summary

Technical Problem

During the compression and transmission of high-frequency data in smart stores, it is difficult for the existing technology to detect the quality of data transmission in real time, resulting in the possible problem of missing data information or excessive transmission time, and the transmission end fails to adjust in time when selecting the compression ratio, which affects the impact of Transmission efficiency.

Method used

Using a high-frequency data compression management system based on artificial intelligence, the compression ratio range, compression speed, transmission speed and information transmission efficiency of each high-frequency data are constructed, and the compressed transmission quality function is used to collect historical data, and the compression ratio range, compression speed, transmission speed and information transmission efficiency of each high-frequency data are analyzed and calculated. The neural network builds an early warning model, monitors and adjusts the compression transmission process in real time, and ensures the integrity and efficiency of data transmission.

Benefits of technology

Real-time early warning and automatic adjustment of high-frequency data during compression transmission is realized, reducing the risk of data loss and error, improving transmission efficiency and automation, and reducing human resources consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118075144B_ABST
    Figure CN118075144B_ABST
Patent Text Reader

Abstract

The present invention discloses a high-frequency data compression management system and method based on artificial intelligence, and belongs to the technical field of data compression and transmission. The present invention collects high-frequency data that has been successfully compressed and transmitted in history, marks and records each high-frequency data and constructs a data set; analyzes the constructed data set, extracts data features and marks them, and constructs a feature set of high-frequency data; calculates the compression ratio range of each high-frequency data according to the obtained data set and feature set; collects the compression transmission time of historical high-frequency data, and calculates the compression speed, transmission speed and information transmission efficiency of high-frequency data at different compression ratios; then calculates the compression transmission quality function of historical high-frequency data according to the three functions obtained; constructs a high-frequency data compression transmission warning model using the high-frequency data compression transmission quality obtained by calculation, and implements real-time warning of high-frequency data compression transmission according to the warning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data compression transmission, and in particular to a high-frequency data compression management system and method based on artificial intelligence. Background Art

[0002] With the invention of computers, the Internet has entered every household, and what follows is the information age. People no longer use the older methods of sending letters, pigeons, etc. to transmit information. The transmission of information has become quick and convenient. In the tide of the times, people have gradually become familiar with and proficient in using various information transmission methods, such as: making phone calls, sending text messages, electromagnetic waves, etc., and according to the different types of signals, various signal transmission media have evolved, among which digital signals, analog signals, mechanical signals, etc. require different transmission methods; and data compression transmission is more and more widely used in life, such as: intelligent manufacturing, medical care, smart stores, etc.; in smart stores, the frequency of data generation is relatively high. In order to improve transmission efficiency, save bandwidth, reduce costs, and improve data security, the high-frequency data generated is processed using compressed transmission methods.

[0003] With the use of computers and the Internet, more and more information is generated in smart stores, and the amount of data that needs to be transmitted also increases. In order to enable faster and more efficient transmission of information, data compression is increasingly used in smart stores. Data containing information is compressed before being transmitted to make it smaller and faster to transmit to the client. Gradually, with the application and research of data compression, many data compression algorithms have emerged, including: Huffman coding, Lempel-Ziv algorithm series, etc. Different algorithms set different compression parameters, and the compression ratio of the compressed data is also different. Different compression ratios affect various efficiencies in data compression, including compression speed, transmission speed, information transmission efficiency, etc. Therefore, when people use data compression, in order to achieve Different effects may use different compression ratios, but this may greatly reduce other compression effects and fail to achieve the purpose of data compression transmission. In addition, when compressing and transmitting data, the transmission end often selects the compression ratio and sends the data without considering whether the data information transmitted to the client is complete. If the data information transmitted to the client is missing or the transmission time is too long, the compression ratio can be modified and retransmitted only after waiting for the client's feedback. Especially in smart stores, because a large amount of data is generated and the frequency is high, there is often no dedicated person to detect the quality of transmitted data, so there is often a problem of low data transmission quality. Therefore, when compressing and transmitting data, it is very important to automatically detect the effect and quality of data compression on the transmission end, so that the transmission end can find problems in time before the client feedbacks and re-compress the data for transmission. Summary of the invention

[0004] The purpose of the present invention is to provide a high-frequency data compression management system and method based on artificial intelligence to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A high-frequency data compression management method based on artificial intelligence, the method comprising the following steps:

[0007] S100, collecting high-frequency data that has been successfully compressed and transmitted in historical records, marking and recording each high-frequency data and constructing a data set;

[0008] Furthermore, for all the compressed and transmitted data frequencies in the collection history, all data frequencies are sorted from small to large, the median is calculated, and the data with a frequency greater than the median frequency is defined as high-frequency data;

[0009] All data in the high-frequency data are distinguished according to the tags, so that each data in the high-frequency data can be displayed separately, and the data set is constructed as {s1, s2, s3...s n}, s1, s2, s3...s n is the 1st, 2nd, 3rd...nth data in the high-frequency data, where n is a positive integer.

[0010] In the collected historical high-frequency data, each data to be transmitted is individually marked, which can better realize the search for data features;

[0011] S200, analyzing the constructed data set, extracting data features for labeling, and constructing a feature set of high-frequency data;

[0012] Furthermore, the specific steps for constructing the feature set of data are:

[0013] S201, classify the same data in the data set as the same type of data, and after classification, obtain the type of data in the data set as x; classify the same data in the high-frequency data according to the mark, and use the type of data in the high-frequency data to indirectly reflect the necessary information contained in the data;

[0014] S202, taking x types of data in the data set as the features of high-frequency data, and finally obtaining the feature set of high-frequency data as {t1, t2, t3...t m}, t1, t2, t3...t m is the 1st, 2nd, 3rd...mth feature in the high-frequency data, where m is a positive integer. <n。

[0015] S300, calculating the maximum compression ratio and the minimum compression ratio of each high-frequency data according to the analyzed data set and feature set, and obtaining the compression ratio range of the high-frequency data;

[0016] Furthermore, the specific steps for obtaining the compression ratio range of high-frequency data are as follows:

[0017] S301, collect the compression ratios of the k-times high-frequency data compression transmission success records, specifically {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio when the 1st, 2nd, 3rd...kth high-frequency data collected are successfully compressed and transmitted, where k is a positive integer;

[0018] S302, sorting the compression ratios of the k high-frequency data compression transmissions collected in ascending order, setting the minimum compression ratio θ_min at which the high-frequency data can be successfully compressed and transmitted, and setting the maximum compression ratio θ_max at which the high-frequency data can be successfully compressed and transmitted;

[0019] S303. The compression ratio range of the high-frequency data for compressed transmission is obtained according to the sorting calculation: (θ_min, θ_max).

[0020] The minimum compression ratio in the compression ratio range indicates the compression ratio that can be selected to speed up the transmission speed after ensuring the transmission of the most basic information in high-frequency data. The maximum compression ratio indicates the compression ratio that speeds up the transmission speed in order to make the information of the transmitted data the most complete. Compression ratios outside this range may result in missing information in the transmitted data or too slow transmission speed.

[0021] S400, by collecting the compression transmission time of historical high-frequency data, the compression speed, transmission speed and information transmission efficiency of the high-frequency data at different compression ratios are calculated;

[0022] Furthermore, the specific steps for calculating the compression speed of high-frequency data at different compression ratios are as follows:

[0023] S411, collect the compression ratio and compression time of the historical k high-frequency data compression transmission successes, the compression ratio is {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected, k is a positive integer; the compression time is {T1, T2, T3...T k}, T1, T2, T3...T k is the compression time of the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer;

[0024] S412, calculating the compression speed according to the collected compression time and compression ratio, the formula is:

[0025] V_y=(n*θ i ) / T i

[0026] In the formula, V_y represents the compression speed, θ i and T i represents the corresponding compression ratio and compression time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression;

[0027] S413, assuming compression ratio as an independent variable and compression speed as a variable, a relationship curve graph between compression ratio and compression speed is drawn, and a functional relationship between compression ratio and compression speed is calculated using a segmentation method according to the curve graph as V_y=f(θ).

[0028] The specific steps for calculating the transmission speed of high-frequency data at different compression ratios are as follows:

[0029] S421, collect the compression ratios of the k-times high-frequency data compression transmission successes in history, the compression ratios are {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected during compression transmission, k is a positive integer; and the transmission time of the high-frequency data collected after compression is {T_c1, T_c2, T_c3...T_c k}, T_c1, T_c2, T_c3...T_c k is the transmission time after the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer;

[0030] S422. Calculate the transmission speed according to the collected compression ratio and transmission time, using the formula:

[0031] V_c=(nn*θ i ) / T_c i

[0032] In the formula, V_c represents the transmission speed, θ i and T_c i represents the corresponding compression ratio and transmission time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression;

[0033] S423, assuming compression ratio as an independent variable and transmission speed as a variable, a relationship curve between compression ratio and transmission speed is plotted, and the functional relationship between compression ratio and transmission speed is calculated using the segmentation method according to the curve, which is V_c=g(θ).

[0034] The specific steps for calculating the information transmission efficiency of high-frequency data at different compression ratios are as follows:

[0035] S431, collect the compression ratios of the k-times high-frequency data compression transmission successes in history, the compression ratios are {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected, where k is a positive integer; and the probability that the information described in the high-frequency data is true is collected;

[0036] S432 calculates the amount of information contained in the high-frequency data before compression and after compression transmission, respectively, and the calculation formula is:

[0037] L=-log a p(x i )

[0038] In the formula, L represents the amount of information calculated, a represents the unit of selected information, and p(x i ) represents the probability that the information described in the high-frequency data is true;

[0039] In the above formula, L is calculated to be the amount of information in the high-frequency data. After the high-frequency data is compressed, the amount of information in the high-frequency data will also change. The probability p(x i ) indicates the complexity of the information in the high-frequency data. The more complex the information is, the more information it contains, and vice versa. For example, when the event in the high-frequency data is "the sun rises from the east", the probability of the event occurring is 100%, so the information contained is relatively simple and the amount of information is relatively small. If the probability of the event occurring in the high-frequency data is small, it proves that the uncertainty of the event is large, the complexity of the information is large, and the amount of information is large; and the "information is true" means the probability of the event occurring in the high-frequency data. According to the probability of the event occurring, the amount of information before and after the compression and transmission of the high-frequency data can be calculated. By comparing the amount of information before and after compression, the efficiency of the compression and transmission of the high-frequency data can be judged.

[0040] S433, according to the above formula, the information amounts contained in the data before and after high-frequency data compression and transmission are calculated to be {L_q1, L_q2, L_q3...L_q k} and {L_h1, L_h2, L_h3...L_h k}, L_q1, L_q2, L_q3...L_q k is the amount of information contained in the data before the 1st, 2nd, 3rd, ...kth high-frequency data compression, L_h1, L_h2, L_h3, ... L_h k is the amount of information contained in the data after the 1st, 2nd, 3rd...kth high-frequency data compression transmission, where k is a positive integer;

[0041] S434, calculating the information transmission efficiency of the high-frequency data at different compression ratios according to the information amount contained in the data before and after compression and transmission obtained by the above calculation, the formula is:

[0042]

[0043] In the formula, ρ represents the information transmission efficiency of high-frequency data at different compression ratios, L_q represents the amount of information contained in the data before high-frequency data compression, and L_h represents the amount of information contained in the data after high-frequency data compression transmission;

[0044] S435, assuming compression ratio as an independent variable and information transmission efficiency as a variable, a relationship curve between compression ratio and information transmission efficiency is plotted, and the functional relationship between compression ratio and information transmission efficiency is calculated using the segmentation method according to the curve:

[0045] S500, calculating a historical high-frequency data compression transmission quality function according to the calculated compression speed, transmission speed, and compression efficiency;

[0046] Furthermore, the specific steps for calculating the high-frequency data compression transmission quality are as follows:

[0047] S501, collect the three functional relationships calculated in S400, which are compression speed function V_y=f(θ), transmission speed function V_c=g(θ), and information transmission efficiency function

[0048] S502, calculating the high-frequency data compression transmission quality according to the above three functional relationships, the formula is:

[0049] M=V_y+V_c+V_ρ

[0050] In the formula, M is the high-frequency data compression transmission quality, V_y is the compression speed function value, V_c is the transmission speed function value, and V_ρ is the information transmission efficiency function value;

[0051] The formula obtained by the above calculation combines the compression speed, transmission speed and information transmission efficiency when high-frequency data is compressed to comprehensively judge the efficiency of high-frequency data in compression transmission. Only one value, the compression transmission quality, needs to be calculated to judge whether the compressed and transmitted data is valid.

[0052] S503. A functional relationship is obtained according to the above calculation, in which the compression ratio is an independent variable and the high-frequency data compression transmission quality is a variable. A relationship curve between the compression ratio and the high-frequency data compression transmission quality is drawn. The definition domain of the independent variable of the high-frequency data compression transmission quality function is restricted according to the value range of the compression ratio calculated in S300. The calculated high-frequency data compression transmission quality function M=V_y+V_c+V_ρ is set as the normal quality function of high-frequency data compression transmission, and the definition domain of the independent variable θ is θ∈(θ_min, θ_max).

[0053] S600: construct a high-frequency data compression transmission warning model using the high-frequency data compression transmission quality obtained by calculation, and implement real-time warning for the compression transmission of high-frequency data according to the warning model.

[0054] Furthermore, the specific steps for real-time warning of compressed transmission of high-frequency data are as follows:

[0055] S601, transmitting the historical high-frequency data compression transmission quality function M calculated in S500 to the convolutional neural network algorithm to build an early warning model;

[0056] S602. When the system compresses and transmits high-frequency data, it collects the compression ratio, compression time, and transmission time during the compression transmission in real time, and finally calculates the high-frequency data compression transmission quality \(M_x\) in real time;

[0057] S603. Input the high-frequency data compression transmission quality \(M_x\) calculated in real time into the warning model for judgment. When either \(\theta\in(\theta_{min},\theta_{max})\) or \(M_x < M\), it is determined that the high-frequency data compression transmission quality is abnormal and there is a risk of data loss. A warning is sent to the transmission end, and it is prompted to re-perform high-frequency data compression transmission.

[0058] A high-frequency data compression management system based on artificial intelligence. The high-frequency data compression management system includes a data collection module, a data processing module, a function processing module, a quality function calculation module, a model construction module, and a real-time warning module;

[0059] The data collection module is used to collect historical high-frequency data compression transmission data, mark and record each high-frequency data, and construct a data set;

[0060] The data processing module is used to perform calculation processing on the historical high-frequency data compression data collected by the data collection module to obtain the compression ratio range during high-frequency data compression;

[0061] The function processing module is used to collect the compression ratio, compression time, transmission time, and the probability of event occurrence in high-frequency data during historical high-frequency data compression transmission, calculate the compression speed, transmission speed, and information transmission efficiency for the collected historical data, and the data processing module includes a compression speed calculation unit, a transmission speed calculation unit, and an information transmission efficiency calculation unit;

[0062] The quality function calculation module is used to calculate the high-frequency data compression transmission quality function based on the calculated compression speed function, transmission speed function, and information transmission efficiency function;

[0063] The model construction module is used to transmit the calculated historical high-frequency data compression transmission quality function \(M\) to the convolutional neural network algorithm to construct a warning model;

[0064] The real-time warning module is used to input the high-frequency data compression transmission quality \(M_x\) calculated in real time into the warning model for judgment, and perform real-time warning on the high-frequency data compression transmission according to the warning model.

[0065] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0066] 1. The present invention calculates the compression range when compressing high-frequency data, and obtains the minimum and maximum compression ratios when compressing high-frequency data. This can prevent the high-frequency data from being compressed by setting a compression ratio that is too large and fails to achieve the compression effect, resulting in slower data transmission after compression, or by setting a compression ratio that is too small, resulting in serious data loss and inability to communicate effective information after data transmission.

[0067] 2. The present invention calculates three functions: compression speed, transmission speed and information transmission efficiency, and finally performs comprehensive processing to obtain the compression transmission quality of high-frequency data compression. The effect of high-frequency data compression is judged by the compression transmission quality, real-time warning is achieved, and the risk of missing and error in high-frequency data transmission is reduced.

[0068] 3. The present invention constructs an early warning module based on the calculated high-frequency data compression transmission quality, realizes the system's automatic judgment and early warning of high-frequency data, can ensure the high efficiency of high-frequency data during compression and transmission, is more automated, and reduces the consumption of human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0070] Figure 1 It is a schematic diagram of the steps of a high-frequency data compression management method based on artificial intelligence of the present invention;

[0071] Figure 2 It is a module distribution diagram of a high-frequency data compression management system based on artificial intelligence of the present invention;

[0072] Figure 3 It is a compression speed and transmission speed curve diagram of a high-frequency data compression management method based on artificial intelligence of the present invention;

[0073] Figure 4 It is an information transmission efficiency diagram of a high-frequency data compression management method based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1-Figure 4 , the present invention provides a technical solution:

[0076] A high-frequency data compression management method based on artificial intelligence, the method comprising the following steps:

[0077] S100, collecting high-frequency data that has been successfully compressed and transmitted in historical records, marking and recording each high-frequency data and constructing a data set;

[0078] Furthermore, for all the compressed and transmitted data frequencies in the collection history, all data frequencies are sorted from small to large, the median is calculated, and the data with a frequency greater than the median frequency is defined as high-frequency data;

[0079] All data in the high-frequency data are distinguished according to the tags, so that each data in the high-frequency data can be displayed separately, and the data set is constructed as {s1, s2, s3...s n}, s1, s2, s3...s n is the 1st, 2nd, 3rd...nth data in the high-frequency data, where n is a positive integer.

[0080] In the collected historical high-frequency data, each data to be transmitted is individually marked, which can better realize the search for data features;

[0081] S200, analyzing the constructed data set, extracting data features for labeling, and constructing a feature set of high-frequency data;

[0082] Furthermore, the specific steps for constructing the feature set of data are:

[0083] S201, classify the same data in the data set as the same type of data, and after classification, obtain the type of data in the data set as x; classify the same data in the high-frequency data according to the mark, and use the type of data in the high-frequency data to indirectly reflect the necessary information contained in the data;

[0084] S202, taking x types of data in the data set as the features of high-frequency data, and finally obtaining the feature set of high-frequency data as {t1, t2, t3...t m}, t1, t2, t3...t m is the 1st, 2nd, 3rd...mth feature in the high-frequency data, where m is a positive integer. <n。

[0085] S300, calculating the maximum compression ratio and the minimum compression ratio of each high-frequency data according to the analyzed data set and feature set, and obtaining the compression ratio range of the high-frequency data;

[0086] Furthermore, the specific steps for obtaining the compression ratio range of high-frequency data are as follows:

[0087] S301, collect the compression ratios of the k-times high-frequency data compression transmission success records, specifically {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio when the 1st, 2nd, 3rd...kth high-frequency data collected are successfully compressed and transmitted, where k is a positive integer;

[0088] S302, sorting the compression ratios of the k high-frequency data compression transmissions collected in ascending order, setting the minimum compression ratio θ_min at which the high-frequency data can be successfully compressed and transmitted, and setting the maximum compression ratio θ_max at which the high-frequency data can be successfully compressed and transmitted;

[0089] S303. The compression ratio range of the high-frequency data for compressed transmission is obtained according to the sorting calculation: (θ_min, θ_max).

[0090] The minimum compression ratio in the compression ratio range indicates the compression ratio that can be selected to speed up the transmission speed after ensuring the transmission of the most basic information in high-frequency data. The maximum compression ratio indicates the compression ratio that speeds up the transmission speed in order to make the information of the transmitted data the most complete. Compression ratios outside this range may result in missing information in the transmitted data or too slow transmission speed.

[0091] S400, by collecting the compression transmission time of historical high-frequency data, the compression speed, transmission speed and information transmission efficiency of the high-frequency data at different compression ratios are calculated;

[0092] Furthermore, the specific steps for calculating the compression speed of high-frequency data at different compression ratios are as follows:

[0093] S411, collect the compression ratio and compression time of k high-frequency data compression transmissions in history, the compression ratio is {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected, k is a positive integer; the compression time is {T1, T2, T3...T k}, T1, T2, T3...T k is the compression time of the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer;

[0094] S412, calculating the compression speed according to the collected compression time and compression ratio, the formula is:

[0095] V_y=(n*θ i ) / T i

[0096] In the formula, V_y represents the compression speed, θ i and T i represents the corresponding compression ratio and compression time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression;

[0097] S413, assuming compression ratio as an independent variable and compression speed as a variable, a relationship curve graph between compression ratio and compression speed is drawn, and a functional relationship between compression ratio and compression speed is calculated using a segmentation method according to the curve graph as V_y=f(θ).

[0098] The specific steps for calculating the transmission speed of high-frequency data at different compression ratios are as follows:

[0099] S421, according to the compression ratio of the historical k high-frequency data compression transmission collected in S411, the compression ratio is {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected during compression transmission, k is a positive integer; and the transmission time of the high-frequency data collected after compression is {T_c1, T_c2, T_c3...T_c k}, T_c1, T_c2, T_c3...T_c k is the transmission time after the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer;

[0100] S422. Calculate the transmission speed according to the collected compression ratio and transmission time, using the formula:

[0101] V_c=(nn*θ i ) / T_c i

[0102] In the formula, V_c represents the transmission speed, θ i and T_c i represents the corresponding compression ratio and transmission time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression;

[0103] S423, assuming compression ratio as an independent variable and transmission speed as a variable, a relationship curve between compression ratio and transmission speed is plotted, and the functional relationship between compression ratio and transmission speed is calculated using the segmentation method according to the curve, which is V_c=g(θ).

[0104] The specific steps for calculating the information transmission efficiency of high-frequency data at different compression ratios are as follows:

[0105] S431, according to S411, the compression ratio and compression time of the historical k high-frequency data compression transmission are collected, and the compression ratio is {θ1, θ2, θ3...θ k}, θ1, θ2, θ3...θk is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected, where k is a positive integer; and the probability that the information described in the high-frequency data is true is collected;

[0106] S432 calculates the amount of information contained in the high-frequency data before compression and after compression transmission, respectively, and the calculation formula is:

[0107] L=-log a p(x i )

[0108] In the formula, L represents the amount of information calculated, a represents the unit of selected information, and p(x i ) represents the probability that the information described in the high-frequency data is true;

[0109] In the above formula, L is calculated to be the amount of information in the high-frequency data. After the high-frequency data is compressed, the amount of information in the high-frequency data will also change. The probability p(x i ) indicates the complexity of the information in the high-frequency data. The more complex the information is, the more information it contains, and vice versa. For example, when the event in the high-frequency data is "the sun rises from the east", the probability of the event occurring is 100%, so the information contained is relatively simple and the amount of information is relatively small. If the probability of the event occurring in the high-frequency data is small, it proves that the uncertainty of the event is large, the complexity of the information is large, and the amount of information is large; and the "information is true" means the probability of the event occurring in the high-frequency data. According to the probability of the event occurring, the amount of information before and after the compression and transmission of the high-frequency data can be calculated. By comparing the amount of information before and after compression, the efficiency of the compression and transmission of the high-frequency data can be judged.

[0110] S433, according to the above formula, the information amounts contained in the data before and after high-frequency data compression and transmission are calculated to be {L_q1, L_q2, L_q3...L_q k} and {L_h1, L_h2, L_h3...L_h k}, L_q1, L_q2, L_q3...L_q k is the amount of information contained in the data before the 1st, 2nd, 3rd, ...kth high-frequency data compression, L_h1, L_h2, L_h3, ... L_h k is the amount of information contained in the data after the 1st, 2nd, 3rd...kth high-frequency data compression transmission, where k is a positive integer;

[0111] S434, calculating the information transmission efficiency of the high-frequency data at different compression ratios according to the information amount contained in the data before and after compression and transmission obtained by the above calculation, the formula is:

[0112]

[0113] In the formula, ρ represents the information transmission efficiency of high-frequency data at different compression ratios, L_q represents the amount of information contained in the data before high-frequency data compression, and L_h represents the amount of information contained in the data after high-frequency data compression transmission;

[0114] S435, assuming compression ratio as an independent variable and information transmission efficiency as a variable, a relationship curve between compression ratio and information transmission efficiency is plotted, and the functional relationship between compression ratio and information transmission efficiency is calculated using the segmentation method according to the curve:

[0115] S500, calculating a historical high-frequency data compression transmission quality function according to the calculated compression speed, transmission speed, and compression efficiency;

[0116] Furthermore, the specific steps for calculating the high-frequency data compression transmission quality are as follows:

[0117] S501, collect the three functional relationships calculated in S400, which are compression speed function V_y=f(θ), transmission speed function V_c=g(θ), and information transmission efficiency function

[0118] S502, calculating the high-frequency data compression transmission quality according to the above three functional relationships, the formula is:

[0119] M=V_y+V_c+V_ρ

[0120] In the formula, M is the high-frequency data compression transmission quality, V_y is the compression speed function value, V_c is the transmission speed function value, and V_ρ is the information transmission efficiency function value;

[0121] The formula obtained by the above calculation combines the compression speed, transmission speed and information transmission efficiency when high-frequency data is compressed to comprehensively judge the efficiency of high-frequency data in compression transmission. Only one value, the compression transmission quality, needs to be calculated to judge whether the compressed and transmitted data is valid.

[0122] S503. A functional relationship is obtained according to the above calculation, in which the compression ratio is an independent variable and the high-frequency data compression transmission quality is a variable. A relationship curve between the compression ratio and the high-frequency data compression transmission quality is drawn. The definition domain of the independent variable of the high-frequency data compression transmission quality function is restricted according to the value range of the compression ratio calculated in S300. The calculated high-frequency data compression transmission quality function M=V_y+V_c+V_ρ is set as the normal quality function of high-frequency data compression transmission, and the definition domain of the independent variable θ is θ∈(θ_min, θ_max).

[0123] S600. Build a high-frequency data compression transmission warning model by using the calculated high-frequency data compression transmission quality, and realize real-time warning for the high-frequency data compression transmission according to the warning model.

[0124] Further, the specific steps for real-time warning of the high-frequency data compression transmission are as follows:

[0125] S601. Transmit the historical high-frequency data compression transmission quality function M calculated by S500 to the convolutional neural network algorithm to build a warning model;

[0126] S602. When the system compresses and transmits high-frequency data, collect the compression ratio, compression time, and transmission time during the compression transmission in real time, and finally calculate the high-frequency data compression transmission quality M_x in real time;

[0127] S603. Input the high-frequency data compression transmission quality M_x calculated in real time into the warning model for judgment. When or θ ∈ (θ_min, θ_max), M_x < M, it is judged that the high-frequency data compression transmission quality is abnormal and there is a risk of data loss, and a warning is sent to the transmission end, and it is prompted to re-perform the high-frequency data compression transmission.

[0128] A high-frequency data compression management system based on artificial intelligence. The high-frequency data compression management system includes a data collection module, a data processing module, a function processing module, a quality function calculation module, a model construction module, and a real-time warning module;

[0129] The data collection module is used to collect historical high-frequency data compression transmission data, mark and record each high-frequency data, and build a data set;

[0130] The data processing module is used to calculate and process the historical high-frequency data compression data collected by the data collection module to obtain the compression ratio range during high-frequency data compression;

[0131] The function processing module is used to collect the compression ratio, compression time, transmission time, and the probability of events occurring in high-frequency data during historical high-frequency data compression transmission, calculate the compression speed, transmission speed, and information transmission efficiency for the collected historical data, and the data processing module includes a compression speed calculation unit, a transmission speed calculation unit, and an information transmission efficiency calculation unit;

[0132] The quality function calculation module is used to calculate the high-frequency data compression transmission quality function based on the calculated compression speed function, transmission speed function, and information transmission efficiency function;

[0133] The model construction module is used to transmit the calculated historical high-frequency data compression transmission quality function M to the convolutional neural network algorithm to build a warning model;

[0134] The real-time warning module is used to input the high-frequency data compression transmission quality M_x calculated in real time into the warning model for judgment, and to implement real-time warning for the compression transmission of high-frequency data according to the warning model.

[0135] Example 1

[0136] First, the compressed transmission of 5 high-frequency data in history is collected, and the data volume, compression time, transmission time, compression ratio, and event occurrence probability of the 5 high-frequency data compressed transmissions are obtained as follows:

[0137]

[0138] The compression speed and transmission speed of high-frequency data are calculated according to the formula:

[0139] V_y=(n*θ i ) / T i

[0140] V_c=(nn*θ i ) / T_c i

[0141] The compression time and transmission speed of 5 high-frequency data are calculated as follows:

[0142] first Second time The third time Fourth Fifth Compression speed 1.27 0.55 0.4 0.36 1.64 Transfer speed 16.5 8.68 5.71 6.54 11

[0143] The collected high-frequency data is analyzed to obtain the amount of information before and after compression in the high-frequency data, and the information transmission efficiency is calculated according to the formula, as follows:

[0144]

[0145]

[0146] A function curve is drawn based on the compression speed, transmission speed and information transmission efficiency calculated above, and the corresponding function relationships are calculated as y=-19.067x+13.881, y=-2.5517x+1.4054, y=0.6909x+0.02;

[0147] According to the three function expressions calculated above, the high-frequency function compression transmission quality is calculated as:

[0148] M=V_y+V_c+V_ρ

[0149] Substituting the three functional relationships into the equations, we obtain M = -20.9278x + 15.3064.

[0150] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0151] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high-frequency data compression management method based on artificial intelligence, characterized in that: The method comprises the following steps: S100, collecting high-frequency data that has been successfully compressed and transmitted in historical records, marking each high-frequency data and constructing a data set; The process of constructing the dataset is as follows: For all the compressed and transmitted data frequencies in the collection history, sort all the data frequencies from small to large, calculate the median, and define the data with a frequency greater than the median frequency as high-frequency data; All data in the high-frequency data are distinguished according to the tags, so that each data in the high-frequency data can be displayed separately, and the data set is constructed as {s1, s2, s3...s n }, s1, s2, s3...s n is the 1st, 2nd, 3rd...nth data in the high-frequency data, where n is a positive integer; S200, analyzing the constructed data set, extracting data features for labeling, and constructing a feature set of high-frequency data; S300, calculating the maximum compression ratio and the minimum compression ratio of each high-frequency data according to the data set and feature set obtained by analysis, and obtaining the compression ratio range of the high-frequency data; S400, by collecting the compression transmission time of historical high-frequency data, the compression speed, transmission speed and information transmission efficiency of the high-frequency data at different compression ratios are calculated; The specific steps for calculating the information transmission efficiency of high-frequency data at different compression ratios are as follows: S431, collecting the compression ratios of successful compression transmission of high-frequency data for k times in history, and collecting the probability that the information described in the high-frequency data is true; S432 calculates the amount of information contained in the high-frequency data before compression and after compression transmission, respectively, and the calculation formula is: L=-log a p(x i ) In the formula, L represents the amount of information calculated, a represents the unit of selected information, and p(x i ) represents the probability that the information described in the high-frequency data is true; S433, according to the above formula, the information amounts contained in the data before and after high-frequency data compression and transmission are calculated to be {L_q1, L_q2, L_q3...L_q k } and {L_h1, L_h2, L_h3...L_h k }, L_q1, L_q2, L_q3...L_q k is the amount of information contained in the data before the 1st, 2nd, 3rd, ...kth high-frequency data compression, L_h1, L_h2, L_h3, ... L_h k is the amount of information contained in the data after the 1st, 2nd, 3rd...kth high-frequency data compression transmission, where k is a positive integer; S434, calculating the information transmission efficiency of the high-frequency data at different compression ratios according to the information amount contained in the data before and after compression and transmission obtained by the above calculation, the formula is: In the formula, ρ represents the information transmission efficiency of high-frequency data at different compression ratios, L_q represents the amount of information contained in the data before high-frequency data compression, and L_h represents the amount of information contained in the data after high-frequency data compression transmission; S435, assuming compression ratio as an independent variable and information transmission efficiency as a variable, a relationship curve between compression ratio and information transmission efficiency is plotted, and the functional relationship between compression ratio and information transmission efficiency is calculated using the segmentation method according to the curve: S500, calculating a historical high-frequency data compression transmission quality function according to the calculated compression speed, transmission speed, and compression efficiency; The specific steps for calculating the high-frequency data compression transmission quality function are: S501, collect the three functional relationships calculated in S400, which are compression speed function V_y=f(θ), transmission speed function V_c=g(θ), and information transmission efficiency function S502, calculating the high-frequency data compression transmission quality according to the above three functional relationships, the formula is: M=V_y+V_c+V_ρ In the formula, M is the high-frequency data compression transmission quality, V_y is the compression speed function value, V_c is the transmission speed function value, and V_ρ is the information transmission efficiency function value; S503. A functional relationship is obtained according to the above calculation, wherein the compression ratio is an independent variable and the high-frequency data compression transmission quality is a variable. A relationship curve between the compression ratio and the high-frequency data compression transmission quality is plotted. The definition domain of the independent variable of the high-frequency data compression transmission quality function is restricted according to the value range of the compression ratio calculated in S300. The calculated high-frequency data compression transmission quality function M=V_y+V_c+V_ρ is set as a normal quality function of high-frequency data compression transmission, and the definition domain of the independent variable θ is θ∈(θ_min, θ_max). S600: construct a high-frequency data compression transmission warning model using the high-frequency data compression transmission quality obtained by calculation, and implement compression transmission of high-frequency data for real-time warning according to the warning model.

2. The high-frequency data compression management method based on artificial intelligence according to claim 1, characterized in that: The specific steps of constructing the feature set of data in S200 are: S201, classify the same data in the data set as the same type of data, and after classification, the type of data in the data set is obtained as x; S202, taking x types of data in the data set as the features of high-frequency data, and finally obtaining the feature set of high-frequency data as {t1, t2, t3...t m }, t1, t2, t3...t m is the 1st, 2nd, 3rd...mth feature in the high-frequency data, where m is a positive integer. <n。 3. The high-frequency data compression management method based on artificial intelligence according to claim 1, characterized in that: The specific steps of obtaining the compression ratio range of high-frequency data in S300 are: S301, collect the compression ratios of the k-times high-frequency data compression transmission success records, specifically {θ1, θ2, θ3...θ k }, θ1, θ2, θ3...θ k is the compression ratio when the 1st, 2nd, 3rd...kth high-frequency data collected are successfully compressed and transmitted, where k is a positive integer; S302, sorting the compression ratios of the k high-frequency data compression transmissions collected in ascending order, setting the minimum compression ratio θ_min at which the high-frequency data can be successfully compressed and transmitted, and setting the maximum compression ratio θ_max at which the high-frequency data can be successfully compressed and transmitted; S303. The compression ratio range of the high-frequency data for compressed transmission is obtained according to the sorting calculation: (θ_min, θ_max).

4. The high-frequency data compression management method based on artificial intelligence according to claim 1, characterized in that: The specific steps of calculating the compression speed of high-frequency data at different compression ratios in S400 are: S411, collect the compression ratio and compression time of the historical k high-frequency data compression transmission successes, the compression ratio is {θ1, θ2, θ3...θ k }, θ1, θ2, θ3...θ k is the compression ratio of the 1st, 2nd, 3rd...kth high-frequency data collected, k is a positive integer; the compression time is {T1, T2, T3...T k }, T1, T2, T3...T k is the compression time of the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer; S412, calculating the compression speed according to the collected compression time and compression ratio, the formula is: V_y=(n*θ i ) / T i In the formula, V_y represents the compression speed, θ i and T i represents the corresponding compression ratio and compression time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression; S413, assuming compression ratio as an independent variable and compression speed as a variable, a relationship curve graph between compression ratio and compression speed is drawn, and a functional relationship between compression ratio and compression speed is calculated using a segmentation method according to the curve graph as V_y=f(θ).

5. The high-frequency data compression management method based on artificial intelligence according to claim 4 is characterized in that: The specific steps of calculating the transmission speed of high-frequency data at different compression ratios in S400 are as follows: S421, using S411 to collect the compression ratios of k successful high-frequency data compression transmissions in history, and collect the transmission time of the high-frequency data after compression (T_c1, T_c2, T_c3...T_c k }, T_c1, T_c2, T_c3...T_c k is the transmission time after the 1st, 2nd, 3rd...kth high-frequency data compression, where k is a positive integer; S422. Calculate the transmission speed according to the collected compression ratio and transmission time, using the formula: V_c=(n-n*θ i ) / T_c i In the formula, V_c represents the transmission speed, θ i and T_c i represents the corresponding compression ratio and transmission time, i ranges from 1 to k, and n represents the total number of high-frequency data before compression; S423, assuming compression ratio as an independent variable and transmission speed as a variable, a relationship curve between compression ratio and transmission speed is plotted, and the functional relationship between compression ratio and transmission speed is calculated using the segmentation method according to the curve, which is V_c=g(θ).

6. The high-frequency data compression management method based on artificial intelligence according to claim 1, characterized in that: The specific steps of compressing and transmitting high-frequency data in S600 for real-time warning are as follows: S601, transmitting the historical high-frequency data compression transmission quality function M calculated in S500 to the convolutional neural network algorithm to build an early warning model; S602, when the system compresses and transmits high-frequency data, the compression ratio, compression time, and transmission time in the compression transmission are collected in real time, and finally the compression transmission quality M_x of the high-frequency data is calculated in real time; S603. Input the high-frequency data compression transmission quality \(M_x\) obtained by real-time calculation into the early warning model for judgment. When or \(\theta\in(\theta_{min},\theta_{max})\), \(M_x < M\), it is judged that the high-frequency data compression transmission quality is abnormal, there is a risk of data loss, an early warning is sent to the transmission end, and it is prompted to re-perform high-frequency data compression transmission.

7. An artificial intelligence-based high-frequency data compression management system applied to an artificial intelligence-based high-frequency data compression management method according to any one of claims 1 to 6, characterized in that: The high-frequency data compression management system includes a data collection module, a data processing module, a function processing module, a quality function calculation module, a model building module and a real-time warning module; The data collection module is used to collect historical high-frequency data, compress and transmit data, mark and record each high-frequency data and construct a data set; The data processing module is used to calculate and process the historical high-frequency data compression data collected by the data collection module to obtain the compression ratio range when the high-frequency data is compressed; The function processing module is used to collect the compression ratio, compression time, transmission time and probability of event occurrence in the historical high-frequency data during compression transmission, and calculate the compression speed, transmission speed and information transmission efficiency of the collected historical data, and the data processing module includes a compression speed calculation unit, a transmission speed calculation unit and an information transmission efficiency calculation unit; The quality function calculation module is used to calculate the high-frequency data compression transmission quality function according to the calculated compression speed function, transmission speed function and information transmission efficiency function; The model building module is used to transmit the calculated historical high-frequency data compression transmission quality function M to the convolutional neural network algorithm to build an early warning model; The real-time warning module is used to input the high-frequency data compression transmission quality M_x calculated in real time into the warning model for judgment, and to implement real-time warning for the compression transmission of high-frequency data according to the warning model.

Citation Information

Cited By

  • Information flow intelligent compression transmission method based on big data

    CN120856656A