Data packet processing method, apparatus, device, and medium
Patent Information
- Application Number
- CN202510551001.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2026-10-30
AI Technical Summary
[0050]本申请实施例提供的数据报文处理方法、装置、设备及介质，首先，获取第一数据报文；之后，若第一数据报文为正常报文，则确定第一数据报文中每个词向量出现频率；然后，对出现频率大于预设频率的高频词向量，根据预设字典将高频词向量替换为对应的字符串，得到第二数据报文，至此，通过利用预设字典，将出现频率较高的词向量（即高频词向量）替换为预设字典中更短的字符串，在不丢失原始语义的前提下，显著减小数据报文的大小，节省存储与传输成本，并且使后续的压缩算法可以在更小的空间上发挥更高效的压缩性能；最后，根据压缩算法，将所述第二数据报文进行压缩，得到第三数据报文。通过这种方法，提高了对数据报文的压缩比率以及降低了数据报文在传输过程中的数据带宽，提高了数据传输的效率。
Smart Images
Figure CN123020665A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data transmission, and in particular to a data message processing method, apparatus, device, and medium. Background Technology
[0002] In industrial data acquisition, data is typically collected in real time using devices such as industrial gateways, instruments, and sensors, and then transmitted to an industrial IoT platform for analysis and monitoring. However, because the volume of industrial data collected is usually large, significant bandwidth waste / high data consumption, high transmission latency, and data loss often occur during transmission. Therefore, to improve data transmission efficiency, appropriate data compression before transmission can effectively enhance data transfer efficiency.
[0003] In existing technologies, data packets are typically processed using single-packet compression. Specifically, after a packet is generated, the acquisition device usually compresses each packet independently using data compression algorithms such as GNU compression (GNU zip, Gzip) and Lempel-Ziv-Oberhumer compression (Lzo), and then transmits the compressed packet over the network.
[0004] However, existing data packet processing methods have low compression ratios, waste data transmission bandwidth, and are difficult to improve data transmission efficiency. Summary of the Invention
[0005] This application provides data packet processing methods, apparatus, devices, and media to address the problems in the prior art where data packet processing methods have low compression ratios, low resource utilization, and difficulty in improving data transmission efficiency.
[0006] In a first aspect, embodiments of this application provide a data packet processing method, including:
[0007] Obtain a first data packet, which contains multiple word vectors;
[0008] If the first data packet is a normal packet, then determine the frequency of each word vector in the first data packet;
[0009] For high-frequency word vectors that appear more frequently than a preset frequency, the high-frequency word vectors are replaced with corresponding strings according to a preset dictionary to obtain a second data message. The preset dictionary is constructed in advance based on different word vectors and their corresponding strings.
[0010] The second data packet is compressed according to the compression algorithm to obtain the third data packet.
[0011] In one possible implementation, the step of replacing the high-frequency word vector with the corresponding string according to a preset dictionary to obtain the second data message includes:
[0012] Determine whether a target string corresponding to the high-frequency word vector exists in the preset dictionary;
[0013] If it exists, the high-frequency word vector is replaced with the target string to obtain the second data message.
[0014] In one possible implementation, the method further includes:
[0015] If it does not exist, the high-frequency word vector is encoded according to the encoding rules to generate the corresponding target string;
[0016] The high-frequency word vectors and their corresponding encoded target strings are stored in the preset dictionary;
[0017] The target word vector is replaced with the target string to obtain the second data message.
[0018] In one possible implementation, the encoding rules include at least one of the following: a hash algorithm, a dictionary compression algorithm, and a bitmap encoding algorithm.
[0019] In one possible implementation, the method further includes:
[0020] The first data packet is input into a deep learning model for classification, and the classification result output by the deep learning model is obtained. The classification result is used to indicate whether the first data packet is a normal packet or an abnormal packet.
[0021] The deep learning model is a model that is pre-trained based on sample data messages and classification result labels to identify whether a message is normal or abnormal.
[0022] In one possible implementation, the method further includes:
[0023] If the classification result is an abnormal message, an alarm message is output, which is used to prompt the user to process the first data message.
[0024] In one possible implementation, after obtaining the third data packet, the method further includes:
[0025] The third data message and the preset dictionary are transmitted to other communication devices.
[0026] Secondly, embodiments of this application provide a data packet processing apparatus, including:
[0027] The acquisition module is used to acquire a first data packet, which contains multiple word vectors;
[0028] The determination module is used to determine the frequency of each word vector in the first data packet if the first data packet is a normal data packet.
[0029] The first processing module is used to replace the high-frequency word vectors that occur more frequently than a preset frequency with the corresponding strings according to a preset dictionary to obtain a second data message. The preset dictionary is constructed in advance based on different word vectors and their corresponding strings.
[0030] The second processing module is used to compress the second data packet according to the compression algorithm to obtain the third data packet.
[0031] In one possible implementation, the first processing device is specifically used for:
[0032] Determine whether a target string corresponding to the high-frequency word vector exists in the preset dictionary;
[0033] If it exists, the high-frequency word vector is replaced with the target string to obtain the second data message.
[0034] In one possible implementation, the first processing device is further configured to:
[0035] If it does not exist, the high-frequency word vector is encoded according to the encoding rules to generate the corresponding target string;
[0036] The high-frequency word vectors and their corresponding encoded target strings are stored in the preset dictionary;
[0037] The target word vector is replaced with the target string to obtain the second data message.
[0038] In one possible implementation, the encoding rules include at least one of the following: a hash algorithm, a dictionary compression algorithm, and a bitmap encoding algorithm.
[0039] In one possible implementation, the first processing device is further configured to:
[0040] The first data packet is input into a deep learning model for classification, and the classification result output by the deep learning model is obtained. The classification result is used to indicate whether the first data packet is a normal packet or an abnormal packet.
[0041] The deep learning model is a model that is pre-trained based on sample data messages and classification result labels to identify whether a message is normal or abnormal.
[0042] In one possible implementation, the first processing device is further configured to:
[0043] If the classification result is an abnormal message, an alarm message is output, which is used to prompt the user to process the first data message.
[0044] In one possible implementation, after obtaining the third data packet, the second processing device is further configured to:
[0045] The third data message and the preset dictionary are transmitted to other communication devices.
[0046] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0047] The memory stores computer-executed instructions;
[0048] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0050] The data packet processing method, apparatus, device, and medium provided in this application embodiment first acquire a first data packet; then, if the first data packet is a normal packet, determine the frequency of each word vector in the first data packet; then, for high-frequency word vectors with a frequency greater than a preset frequency, replace the high-frequency word vectors with corresponding strings according to a preset dictionary to obtain a second data packet. Thus, by utilizing the preset dictionary, the word vectors with higher frequencies (i.e., high-frequency word vectors) are replaced with shorter strings from the preset dictionary, significantly reducing the size of the data packet without losing the original semantics, saving storage and transmission costs, and enabling subsequent compression algorithms to achieve more efficient compression performance in a smaller space; finally, the second data packet is compressed according to the compression algorithm to obtain a third data packet. This method improves the compression ratio of the data packet and reduces the data bandwidth during transmission, thereby improving data transmission efficiency. Attached Figure Description
[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0052] Figure 1Flowchart of the data packet processing method provided in the embodiments of this application Figure 1 ;
[0053] Figure 2 Flowchart of the data packet processing method provided in the embodiments of this application Figure 2 ;
[0054] Figure 3 This is a schematic diagram of the structure of the data packet processing apparatus provided in the embodiments of this application;
[0055] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0058] In industrial data acquisition, data is typically collected in real time using devices such as industrial gateways, instruments, and sensors, and then transmitted to an industrial IoT platform for analysis and monitoring. However, because the volume of industrial data collected is usually large, significant bandwidth waste / high data consumption, high transmission latency, and data loss often occur during transmission. Therefore, to improve data transmission efficiency, appropriate data compression before transmission can effectively enhance data transfer efficiency.
[0059] In existing technologies, data packet processing typically employs single-packet compression. Specifically, after packet generation, the acquisition device usually compresses each packet independently using data compression algorithms such as Gzip or Lzo before transmitting it over the network.
[0060] However, while existing data packet processing methods reduce the amount of data transmitted to some extent, there is often a lot of duplicate information between packets in industrial equipment data acquisition. Single packet compression cannot achieve a high compression rate, resulting in a low compression ratio, low resource utilization, and difficulty in improving data transmission efficiency.
[0061] Based on this, this application proposes a data packet processing method. Since existing technologies directly use single-packet compression, resulting in low compression ratios, and data packets typically contain a large amount of repetitive word vector information—these high-frequency word vectors often occupy significant data space while exhibiting high redundancy in information representation—this method constructs a pre-defined dictionary to uniformly replace them with shorter, more compact strings or codes, effectively reducing the overall information entropy of the data. Subsequently, a conventional compression algorithm is applied to the replaced second data packet, maximizing the potential of the compression algorithm while maintaining lower data complexity, further reducing the data volume. This method not only significantly improves compression efficiency and reduces the bandwidth and space resources required for data transmission and storage, but also ensures the reversible decompression capability and the integrity of the original data structure.
[0062] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0063] Figure 1 Flowchart of the data packet processing method provided in the embodiments of this application Figure 1 ;
[0064] S101, Obtain the first data message.
[0065] The first data message contains multiple word vectors. Each word vector can be a keyword, a byte string, a fixed structure field, or a segment after word segmentation / segmentation from the original message. The specific composition of the word vectors will not be described in detail in this embodiment of the application.
[0066] It should be noted that since data messages are typically structured using JavaScript Object Notation (JSON), Extensible Markup Language (XML), and Hypertext Transfer Protocol (HTTP) request bodies, preprocessing is required before obtaining the first data message. This includes: message structure parsing, (extracting message fields in a structured manner), word segmentation / segmentation (segmenting words or cutting fields according to message format or semantics), noise removal and cleaning (removing irrelevant fields, special symbols, etc.), encoding mapping (converting the segmented words or fields into vectors), and normalization (unifying case, format, numerical precision, etc.). This process allows the extraction of a sequence of word vectors (the first data message) that can be used for model analysis.
[0067] It is understandable that by acquiring the first data message, a unified data foundation can be provided for the input of subsequent deep learning models, thereby improving the processing accuracy and generalization ability of deep learning models.
[0068] S102. If the first data packet is a normal packet, determine the frequency of each word vector in the first data packet.
[0069] It should be understood that for normal messages, in order to improve the compression rate of data messages, it is necessary to replace high-frequency words. Therefore, it is necessary to perform statistical analysis on the word vectors to determine the frequency of each word in the current message or sliding window.
[0070] In this way, on the one hand, the mishandling of abnormal messages can be avoided; on the other hand, high-frequency words can be accurately located, making them easy to replace, thereby maximizing the compression effect and improving the performance of subsequent compression and transmission.
[0071] S103. For high-frequency word vectors that appear more frequently than a preset frequency, replace the high-frequency word vectors with the corresponding strings according to a preset dictionary to obtain the second data message.
[0072] The preset dictionary is constructed in advance based on different word vectors and their corresponding strings.
[0073] For example, a preset dictionary can be represented as:
[0074]
[0075] As can be seen, by using a preset dictionary, the original 12 bytes in the data packet can be shortened to 1 byte. For high-frequency word vectors, this method can significantly reduce the length of the first data packet and effectively reduce the data space of the data packet.
[0076] Optionally, for low-frequency or mid-frequency word vectors that occur less than a preset frequency, replacement can be left undone or replaced with a relatively long string. The specific implementation process for this replacement method is described in this application. Figure 2 The embodiments are described in detail, and the embodiments of this application will not be described in detail here.
[0077] Understandably, by using a pre-defined dictionary to replace high-frequency word vectors with their corresponding strings, the length of the first data packet can be significantly reduced, and the replaced data format still retains its semantic structure, without affecting data parsing.
[0078] S104. According to the compression algorithm, the second data packet is compressed to obtain the third data packet.
[0079] The compression algorithm includes lossy compression algorithm and lossless compression algorithm. In practical applications, the corresponding compression algorithm can be selected according to the requirements. This application embodiment does not impose specific restrictions here.
[0080] Understandably, since the second data packet has replaced a large number of high-frequency words, it has stronger redundancy and a higher compression rate. By selecting a suitable compression algorithm, the third data packet can be made smaller, which can save network bandwidth, improve transmission efficiency and response speed, and reduce resource consumption during data transmission.
[0081] In one possible approach, after obtaining the third data packet, it is also necessary to transmit the third data packet and the preset dictionary to other communication devices.
[0082] It should be noted that the communication method for transmitting third data packets can be a wired network or a second-generation (2G) / 3G / 4G / 5G wireless network; other communication devices can be cloud servers equipped with industrial IoT platforms; furthermore, industrial IoT platforms can also be deployed on edge servers, which can be local computers, embedded devices, or mobile edge computing (MEC).
[0083] The data packet processing method provided in this application first obtains a first data packet; then, if the first data packet is a normal packet, the frequency of each word vector in the first data packet is determined; then, for high-frequency word vectors with a frequency greater than a preset frequency, the high-frequency word vectors are replaced with corresponding strings according to a preset dictionary to obtain a second data packet. Thus, by utilizing the preset dictionary, the word vectors with higher frequencies (i.e., high-frequency word vectors) are replaced with shorter strings from the preset dictionary, significantly reducing the size of the data packet without losing the original semantics, saving storage and transmission costs, and enabling subsequent compression algorithms to achieve more efficient compression performance in a smaller space; finally, the second data packet is compressed according to the compression algorithm to obtain a third data packet. This method improves the compression ratio of the data packet and reduces the data bandwidth during transmission, thereby improving data transmission efficiency.
[0084] It should be noted that before replacing the high-frequency word vectors in the data packet, it is necessary to first determine that the data packet is a normal data packet and can be transmitted. For abnormal data packets, an alarm message needs to be output to prompt the user to process the first data packet.
[0085] In one feasible approach, the specific steps for determining whether a data packet is a normal packet are as follows:
[0086] The first data packet is input into a deep learning model for classification, and the classification result output by the deep learning model is obtained.
[0087] The classification result is used to indicate whether the first data message is a normal message or an abnormal message; the deep learning model is a model that is pre-trained based on the sample data messages and the classification result labels to identify whether the message is normal or abnormal.
[0088] It should be noted that the deep learning model is a Text Convolutional Neural Network (CNN) for text classification. This TextCNN model consists of an input layer, a CNN layer, pooling layers, and an output layer. In the input layer, the first data packet is represented as a fixed-length sequence of word vectors, where each word vector is represented by its embedding dimension. The CNN layer includes multiple one-dimensional convolutional kernels used to extract local features from the first data packet. The size of each convolutional kernel is specified by the kernel size (kernel_size), and the output is specified by the number of output channels (num_channels). The pooling layer converts the output of the convolutional layer into a fixed-length feature vector through temporal max pooling. The output layer is a fully connected layer used to perform classification prediction on the feature vector. For a binary classification task, the output layer has a dimension of 2, representing the predicted probabilities of positive and negative sentiment, i.e., the probability that the first data packet is a normal or abnormal message.
[0089] Understandably, by using deep learning models to pre-classify the first data packets, abnormal packets can be effectively identified and filtered, avoiding incorrect compression of data with non-standard structures, thereby improving stability. At the same time, separating normal and abnormal packets can optimize the processing flow and improve resource utilization. In addition, the classification results can also assist in the optimization and updating of the preset dictionary, ensuring the applicability and accuracy of compression rules, and further enhancing the intelligence and robustness of the entire data transmission.
[0090] Figure 2 Flowchart of the data packet processing method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the embodiments, the process of replacing high-frequency word vectors with corresponding strings according to a preset dictionary is described in detail. The method includes:
[0091] S201. Determine whether there is a target string in the preset dictionary that corresponds to the high-frequency word vector.
[0092] Understandably, after determining the high-frequency word vectors in the first data packet through S103, a query is performed in the preset dictionary to check if there is a corresponding target string, so as to quickly replace it and improve processing efficiency.
[0093] S202. If it exists, replace the high-frequency word vector with the target string to obtain the second data message.
[0094] Understandably, if a string corresponding to a high-frequency word vector exists in the preset dictionary, there is no need to encode it repeatedly. The string can be directly identified as the target string, and the high-frequency word vector can be replaced with the target string, thus saving encoding resources and computational costs.
[0095] Optionally, if the target string does not exist, the high-frequency word vector is encoded according to the encoding rules to generate the corresponding target string; then, the high-frequency word vector and the corresponding target string generated by encoding are stored in a preset dictionary; finally, the target word vector is replaced with the target string and the second data message is obtained.
[0096] The encoding rules include at least one of the following: hash algorithm, dictionary compression algorithm, and bitmap encoding algorithm.
[0097] For example, if the word vector "temperature_warning" in the first data packet has a frequency of 20, which is greater than the preset frequency of 15, then the word vector is determined to be a high-frequency word vector. Then, according to the preset dictionary, the target string "#tw" corresponding to the high-frequency word vector is found, and the word vector is directly replaced. If the target string corresponding to the high-frequency word vector is not found according to the preset dictionary, the target string "@01" corresponding to the high-frequency word vector is generated according to the hash algorithm in the encoding rules, and the replacement is performed. Finally, the mapping relationship between "temperature_warning" and "@01" is stored in the preset dictionary for subsequent word vector replacement.
[0098] Optionally, for low-frequency word vectors, based on compression theories such as Huffman, a longer string can be used to replace them and the target string corresponding to the low-frequency word vector can be stored in a preset dictionary. Although this method does not save much space, it can maintain the consistency of the preset dictionary (i.e., all replaceable items have a set format); and it facilitates subsequent unified processing and reverse restoration using the preset dictionary, as well as allowing rare fields to quickly replace short codes when they are upgraded to high-frequency word vectors in subsequent use.
[0099] For example, if the word vector "temperature_warning" in the first data packet has a frequency of 10, which is greater than the preset frequency of 15, then the word vector is determined to be a low-frequency word vector. Then, a longer string, such as "#L0058", is assigned according to the encoding rules and replaced. Then, the mapping relationship between "temperature_warning" and "#L0058" is stored in the preset dictionary. Finally, if the frequency of this field increases in subsequent operations, its substitute code can be replaced with "#S04" (short code) to improve compression efficiency.
[0100] Understandably, this method of data packet transmission can effectively reduce the transmission burden and speed up communication.
[0101] Figure 3 This is a schematic diagram of the structure of the data packet processing apparatus provided in the embodiments of this application; as shown below. Figure 3As shown, the device includes:
[0102] The acquisition module 301 is used to acquire a first data packet, which contains multiple word vectors.
[0103] The determination module 302 is used to determine the frequency of each word vector in the first data packet if the first data packet is a normal data packet;
[0104] The first processing module 303 is used to replace the high-frequency word vectors that appear more frequently than a preset frequency with the corresponding strings according to a preset dictionary to obtain the second data message. The preset dictionary is constructed in advance based on different word vectors and their corresponding strings.
[0105] The second processing module 304 is used to compress the second data packet according to the compression algorithm to obtain the third data packet.
[0106] In one possible implementation, the first processing device 303 is specifically used for:
[0107] Determine whether a target string corresponding to a high-frequency word vector exists in the preset dictionary;
[0108] If it exists, the high-frequency word vectors are replaced with the target string to obtain the second data message.
[0109] In one possible implementation, the first processing device 303 is further configured to:
[0110] If it does not exist, the high-frequency word vectors are encoded according to the encoding rules to generate the corresponding target string;
[0111] Store the target strings generated from the high-frequency word vectors and their encodings into a preset dictionary;
[0112] Replace the target word vector with the target string and obtain the second data message.
[0113] In one possible implementation, the encoding rules include at least one of the following: hash algorithm, dictionary compression algorithm, and bitmap encoding algorithm.
[0114] In one possible implementation, the first processing device 303 is further configured to:
[0115] The first data packet is input into the deep learning model for classification, and the classification result output by the deep learning model is obtained. The classification result is used to indicate whether the first data packet is a normal packet or an abnormal packet.
[0116] Among them, the deep learning model is a model that is pre-trained based on sample data messages and classification result labels to identify whether a message is normal or abnormal.
[0117] In one possible implementation, the first processing device 303 is further configured to:
[0118] If the classification result is an abnormal message, an alarm message will be output to prompt the user to process the first data message.
[0119] In one possible implementation, after receiving the third data packet, the second processing device 304 is further configured to:
[0120] Transmit the third data message and the preset dictionary to other communication devices.
[0121] The data packet processing apparatus provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0122] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0123] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.
[0124] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0125] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0126] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0127] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0128] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0129] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0130] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0131] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0133] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0134] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0135] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0136] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A data packet processing method, characterized in that, include: Obtain a first data packet, which contains multiple word vectors; If the first data packet is a normal packet, then determine the frequency of each word vector in the first data packet; For high-frequency word vectors that appear more frequently than a preset frequency, the high-frequency word vectors are replaced with corresponding strings according to a preset dictionary to obtain a second data message. The preset dictionary is constructed in advance based on different word vectors and their corresponding strings. The second data packet is compressed according to the compression algorithm to obtain the third data packet.
2. The method according to claim 1, characterized in that, The step of replacing the high-frequency word vectors with corresponding strings according to a preset dictionary to obtain a second data message includes: Determine whether a target string corresponding to the high-frequency word vector exists in the preset dictionary; If it exists, the high-frequency word vector is replaced with the target string to obtain the second data message.
3. The method according to claim 2, characterized in that, The method further includes: If it does not exist, the high-frequency word vector is encoded according to the encoding rules to generate the corresponding target string; The high-frequency word vectors and their corresponding encoded target strings are stored in the preset dictionary; The target word vector is replaced with the target string to obtain the second data message.
4. The method according to claim 3, characterized in that, The encoding rules include at least one of the following: hash algorithm, dictionary compression algorithm, and bitmap encoding algorithm.
5. The method according to claims 1-3, characterized in that, The method further includes: The first data packet is input into a deep learning model for classification, and the classification result output by the deep learning model is obtained. The classification result is used to indicate whether the first data packet is a normal packet or an abnormal packet. The deep learning model is a model that is pre-trained based on sample data messages and classification result labels to identify whether a message is normal or abnormal.
6. The method according to claim 1, characterized in that, The method further includes: If the classification result is an abnormal message, an alarm message is output, which is used to prompt the user to process the first data message.
7. The method according to any one of claims 1-3, characterized in that, After obtaining the third data packet, the method further includes: The third data message and the preset dictionary are transmitted to other communication devices.
8. A data message processing apparatus, characterized in that, include: The acquisition module is used to acquire a first data packet, which contains multiple word vectors; The determination module is used to determine the frequency of each word vector in the first data packet if the first data packet is a normal data packet. The first processing module is used to replace the high-frequency word vectors that occur more frequently than a preset frequency with the corresponding strings according to a preset dictionary to obtain a second data message. The preset dictionary is constructed in advance based on different word vectors and their corresponding strings. The second processing module is used to compress the second data packet according to the compression algorithm to obtain the third data packet.
9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.