Word segmentation methods, network message generation methods, devices and network equipment

By segmenting the key-value pairs of network packets, a token sequence that is easy for the transformer model to understand is generated, which solves the problems of performance consumption and latency in the existing technology and achieves efficient packet generation.

CN123122850APending Publication Date: 2026-11-17CHINA MOBILE COMM LTD RES INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510618374.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-14
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, natural language segmentation methods are directly applied to the message sequence generation process based on the transformer model structure. This results in a large amount of parameters being needed to learn the relevant grammatical structure of the message and the relationships between information elements, leading to performance consumption and processing delay.

Method used

By segmenting the key-value pairs of network messages, the keys and values ​​are converted into key text information and value text information respectively, and then combined into a token sequence according to the order of the information elements, which is used in the generation process of the transformer model.

Benefits of technology

This avoids the transformer model spending a lot of parameters to learn the relevant syntax structure of the message and the relationship between information elements, thus improving the efficiency and performance of the generation process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123122850A_ABST
    Figure CN123122850A_ABST
Patent Text Reader

Abstract

This application provides a word segmentation processing method, a method for generating network packets, an apparatus, and a network device. The method includes: parsing each packet in the original code stream of a first packet to be processed to obtain key-value pairs of multiple information elements in the original code stream of the first packet; converting the keys in the multiple key-value pairs into key text information, and converting the values ​​in the multiple key-value pairs into value text information; determining a first token sequence corresponding to the key text information, and determining a second token sequence corresponding to the value text information; and combining the first token sequence and the second token sequence according to the order of the key-value pairs of multiple information elements to obtain a first synthetic token sequence of the original code stream of the first packet. Using this method, by segmenting the keys and values ​​in the key-value pairs separately, the generated token sequence forms a data structure that is easy for the model to understand.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a word segmentation processing method, a method for generating network messages, an apparatus, and a network device. Background Technology

[0002] In the fields of communications and networking, due to the similarity in format and content between packet messages and text data, applications such as autoregressive message sequence generation based on the deep learning transformer model structure can be implemented by referring to the large language model structure. These applications can be used in various scenarios such as message generation, security and anomaly detection, communication equipment testing, and network protocol simulation, greatly improving the production efficiency of related scenarios.

[0003] When generating messages using a deep learning transformer model, the original bytecode stream of the message needs to be segmented and encoded to obtain corresponding tokens. These tokens are then processed using a word embedding model to obtain word vectors. The transformer model is trained based on these word vectors to generate probability distributions for different tokens, thus generating a sequence of tokens for message transmission. Current technologies typically use natural language segmentation to segment the original bytecode stream of the message. However, due to differences between grouped messages and natural language in grammatical structure, value range, and the meaning of information element values, as well as the complex relationships between information elements in different levels of the message header, this segmentation method causes the transformer model to spend a large number of parameters learning the relevant grammatical structures and relationships between information elements in the original message, resulting in significant performance overhead and processing delays. Summary of the Invention

[0004] The purpose of this application is to provide a word segmentation processing method, a network message generation method, an apparatus, and a network device to solve the problem that existing natural language-based word segmentation methods, when directly applied to the message sequence generation process based on the transformer model structure, result in performance consumption due to the need for extensive parameter learning.

[0005] One embodiment of this application provides a method for word segmentation processing of network messages, comprising:

[0006] Each message in the original code stream of the first message to be processed is parsed to obtain key-value pairs of multiple information elements in the original code stream of the first message;

[0007] Convert the keys in the multiple key-value pairs into key text information, and convert the values ​​in the multiple key-value pairs into value text information;

[0008] Determine the first token sequence corresponding to the key text information, and determine the second token sequence corresponding to the value text information;

[0009] Based on the order of the key-value pairs of the multiple information elements, the first token sequence and the second token sequence are combined to obtain the first synthetic token sequence of the original code stream of the first message.

[0010] Optionally, in the word segmentation method, determining the first token sequence corresponding to the key text information includes:

[0011] Based on the pre-obtained first lexical relationship information, the first token sequence corresponding to the key text information is determined; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or,

[0012] Determining the second token sequence corresponding to the value text information includes:

[0013] Based on the pre-obtained second lexical relationship information, the second token sequence corresponding to the value text information is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0014] Optionally, the word segmentation method, wherein combining the first token sequence and the second token sequence according to the arrangement order of the key-value pairs of the plurality of information elements to obtain a first synthetic token sequence of the first message original code stream includes:

[0015] For each key-value pair, extract the third token corresponding to the key in the key-value pair in the first token sequence, and extract the fourth token corresponding to the value in the key-value pair in the second token sequence;

[0016] Based on the arrangement order of the key-value pairs of the multiple information elements, the third token and the fourth token corresponding to each key-value pair are arranged sequentially adjacent to each other to obtain the first synthetic token sequence.

[0017] Optionally, the word segmentation method further includes one or more of the following:

[0018] The data type of the value in the key-value pair is stored;

[0019] If the data type of the value in the key-value pair is address information, the value in the key-value pair is encrypted before being converted into value text information, and the encrypted information is saved.

[0020] Optionally, in the word segmentation processing method, in the first synthesized token sequence, the token sequences corresponding to two adjacent messages are separated by a message end identifier.

[0021] Optionally, the word segmentation method, wherein parsing each message to obtain multiple key-value pairs of information elements includes:

[0022] Each message is parsed to obtain the lowest-level information element among the multiple information elements of each message;

[0023] Obtain the key-value pairs of the lowest-level information element.

[0024] Optionally, the word segmentation method, wherein converting the keys of the multiple key-value pairs into key text information includes:

[0025] Extract the keys from multiple key-value pairs as extracted text;

[0026] The extracted text is segmented to obtain the key text information.

[0027] Optionally, the word segmentation method, wherein converting the values ​​in the plurality of key-value pairs into value text information includes:

[0028] When the value in the key-value pair is of text type, the word segmentation model is used to segment the value in the key-value pair to obtain the text information of the value;

[0029] If the value in the key-value pair is of type byte or integer, the value in the key-value pair is stored as the value text information in byte form.

[0030] Optionally, in the word segmentation processing method, after parsing each message to obtain multiple key-value pairs of information elements, the method further includes one or more of the following:

[0031] If the first information cell in a set of multiple information cells is in the tag-length-value TLV format, and the key-value pair of the L field in the first information cell has a fixed value, delete the key-value pair of the L field in the first information cell and store the numerical information of the L field of the first information cell.

[0032] If the first information element in a set of multiple information elements is in TLV format and the value corresponding to the V field of the first information element is of text type, delete the key-value pair of the information element corresponding to the L field in the first information element.

[0033] One embodiment of this application also provides a method for generating network packets, comprising:

[0034] Obtain the second synthesized token sequence output by the transformer model; the second synthesized token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information.

[0035] The corresponding key text information is determined based on the key indicator bit in the second synthesized token sequence, and the corresponding value text information is determined based on the value indicator bit in the second synthesized token sequence.

[0036] The key corresponding to the key text information and the value corresponding to the value text information are combined to generate key-value pairs of multiple information elements in the original code stream of the first message;

[0037] Based on the key-value pairs, the original code stream of the first message is obtained.

[0038] Optionally, the word segmentation method, wherein determining the corresponding key text information based on the token at the corresponding key indicator bit in the second synthesized token sequence includes:

[0039] Based on the pre-obtained first lexical relationship information, determine the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or

[0040] The corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0041] Based on the pre-obtained second lexical relationship information, the value text information corresponding to the token on the value indicator bit in the second synthesized token sequence is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0042] Optionally, in the word segmentation processing method, in the second synthesized token sequence, the key indicator bit and value indicator bit corresponding to one information element are adjacent, and the key indicator bit and value indicator bit corresponding to different information elements are spaced apart.

[0043] Optionally, the word segmentation method further includes:

[0044] Based on the message end identifier interval in the second synthesized token sequence, the second synthesized token sequence is split into multiple third token sequences, each of which corresponds to a message;

[0045] Specifically, determining the corresponding key text information based on the token at the key indicator bit in the second synthesized token sequence, and determining the corresponding value text information based on the token at the value indicator bit in the second synthesized token sequence, includes:

[0046] For each of the third token sequences, the corresponding key text information is determined based on the token on the corresponding key indicator bit; and for each of the third token sequences, the corresponding value text information is determined based on the token on the corresponding value indicator bit.

[0047] Optionally, the word segmentation method further includes:

[0048] Based on the storage information at the time of generation of the first synthesized token sequence in the original code stream of the first message, the data type of the value in the key-value pair corresponding to the key text information is obtained; wherein, the first synthesized token sequence is input to the transformer model and the second synthesized token sequence is output.

[0049] If the data type of the value in the key-value pair corresponding to the key text information is text, the value text information determined according to the token on the corresponding value indicator bit is decoded to determine the value corresponding to the value text information.

[0050] Optionally, the word segmentation method further includes:

[0051] If the data type of the value in the key-value pair corresponding to the key text information is address information, the token on the corresponding value indicator bit in the second synthesized token sequence is decrypted according to the encryption information stored in the storage information.

[0052] Specifically, the corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0053] The corresponding value text information is determined based on the decrypted token.

[0054] One embodiment of this application also provides a network message segmentation processing apparatus, comprising:

[0055] The first parsing module is used to parse each message in the original code stream of the first message to be processed, and obtain key-value pairs of multiple information elements in the original code stream of the first message.

[0056] A conversion module is used to convert the keys in the multiple key-value pairs into key text information, and to convert the values ​​in the multiple key-value pairs into value text information;

[0057] The determining module is used to determine the first token sequence corresponding to the key text information, and to determine the second token sequence corresponding to the value text information;

[0058] The combination module is used to combine the first token sequence and the second token sequence according to the arrangement order of the key-value pairs of the multiple information elements to obtain a first synthetic token sequence of the original code stream of the first message.

[0059] One embodiment of this application also provides a network packet generation apparatus, comprising:

[0060] The first acquisition module is used to acquire the second synthetic token sequence output by the transformer model; the second synthetic token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information.

[0061] The determining module is used to determine the corresponding key text information based on the token on the key indicator bit in the second synthesized token sequence, and to determine the corresponding value text information based on the token on the value indicator bit in the second synthesized token sequence;

[0062] The synthesis module is used to synthesize the key corresponding to the key text information and the value corresponding to the value text information to generate key-value pairs of multiple information elements in the original code stream of the first message;

[0063] The second acquisition module is used to obtain the original code stream of the first message based on the key-value pair.

[0064] One embodiment of this application also provides a network device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the word segmentation processing method as described in any of the preceding claims, or implements the network packet generation method as described in any of the preceding claims.

[0065] One embodiment of this application also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the steps of the word segmentation processing method as described in any of the preceding claims, or implements the steps of the network packet generation method as described in any of the preceding claims.

[0066] One embodiment of this application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the word segmentation processing method as described in any of the preceding claims, or implement the steps of the network packet generation method as described in any of the preceding claims.

[0067] At least one of the above-mentioned technical solutions of this application has the following beneficial effects:

[0068] The network message segmentation method described in this application segmentes the key and value in key-value pairs into words, so that the generated token sequence forms a data structure that is easy for the model to understand. It can be directly applied to the message sequence generation process based on the transformer model structure, avoiding the performance consumption problem caused by the model spending a lot of parameters to learn the relevant grammatical structure and the relationship between information elements of the message. Attached Figure Description

[0069] Figure 1 This is a schematic diagram illustrating the message sequence generation process based on the transformer model structure.

[0070] Figure 2 This is a flowchart illustrating the word segmentation method according to one embodiment of this application;

[0071] Figure 3 This is a schematic diagram of the message processing flow using the word segmentation processing method described in the embodiments of this application;

[0072] Figure 4This is a schematic diagram illustrating the training process using the method described in the embodiments of this application;

[0073] Figure 5 A diagram illustrating one set of key-value pairs;

[0074] Figure 6 A schematic diagram representing the first vocabulary corresponding to the key text information;

[0075] Figure 7 This is one of the schematic diagrams illustrating word segmentation using the BPE algorithm;

[0076] Figure 8 The second illustration shows the use of the BPE algorithm for word segmentation.

[0077] Figure 9 This is the third illustration showing the use of the BPE algorithm for word segmentation;

[0078] Figure 10 A schematic diagram representing the second vocabulary corresponding to the value text information;

[0079] Figure 11 This is a schematic diagram illustrating the coding process using the method described in the embodiments of this application;

[0080] Figure 12 A schematic diagram illustrating key-value combinations in the encoding process;

[0081] Figure 13 This is a schematic diagram illustrating the decoding process using the method described in the embodiments of this application;

[0082] Figure 14 This is a schematic diagram illustrating message synthesis during the decoding process;

[0083] Figure 15 This is a flowchart illustrating a method for generating network packets according to one embodiment of this application;

[0084] Figure 16 This is a schematic diagram illustrating the structure of the word segmentation processing device described in the embodiments of this application;

[0085] Figure 17 This is a schematic diagram illustrating the structure of the generating apparatus described in the embodiments of this application. Detailed Implementation

[0086] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0087] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0088] Currently, the process of generating autoregressive message sequences based on a transformer model structure using a large language model structure can be found in [reference needed]. Figure 1 As shown, the specific process includes the following:

[0089] The original bytecode stream of the message is transmitted to the tokenizer for encoding to obtain the token corresponding to the message;

[0090] The acquired lexical units are transmitted to the word embedding model to obtain the word vector corresponding to the lexical unit;

[0091] The obtained word vectors are transmitted to a multi-layer transformer model to obtain a probability distribution corresponding to the generation of different tokens. The top p or top k are taken to form the sampling distribution of the tokens.

[0092] The token sequence generated by the large model is obtained by sampling from the sampling distribution using a sampler;

[0093] The generated token sequence is transmitted to the decoder of the word segmenter, which outputs the original bytecode stream corresponding to the token sequence.

[0094] In addition, the current implementation of word segmentation is mainly applied to text segmentation. Based on large language models, the Byte-Pair Encoding (BPE) algorithm is mostly used to convert the original text into bytes corresponding to UTF-8. Then, the byte sequence is mapped into the corresponding token sequence according to the trained vocabulary and fed into the corresponding model for training and inference.

[0095] However, packet messages and natural language differ in grammatical structure, value range, and the meaning of information element values, and the relationships between information elements in different levels of message headers are complex. If conventional word segmentation methods are used, the following problems will arise:

[0096] In natural language, the same token has the same objective semantics. However, in a data packet, the token mapped to the same segment of bytes depends on its position in the packet. For example, two 00 00 bytes in a packet may have different meanings: the first is the IP layer header checksum, while the second is the TCP layer urgent pointer.

[0097] The message contains a format that is inconsistent with the syntax structure of natural language. Message encoding formats typically include fixed-length fixed-position encoding, type-value format encoding, type-length-value format encoding, key-value pair format encoding, and delimiter format encoding, and these are prevalent in various protocol stack layers of the message. If parsing is based on natural language formats, it will inevitably consume a large number of model parameters to associate the key-value pairs.

[0098] The values ​​of the same information element at the same location are inconsistent and have a wide range. Natural language segmenters suffer from insufficient vocabulary capacity for representation; with a fixed vocabulary size, the segmentation granularity may be too fine, with hundreds of tokens corresponding to a single message, leading to significant performance overhead during model training and inference. For example, if a single-byte (value range: 0-255) vocabulary is used for segmentation, and the message is segmented byte by byte, a 1400-byte message would require 1400 tokens. A large model with 8K contexts can only input and output no more than 5 messages, resulting in low practical application value.

[0099] Therefore, based on the above, due to the information-dense, semantically rich, and inconsistent values ​​of the same information element at the same location in network packets, network packets cannot be represented by a token corresponding to one or more letters or words, as is the case with natural language encoding methods. Typically, a network packet requires multiple tokens. Using typical natural language segmentation methods would require the model to spend a large number of parameters learning the relevant syntactic structure and relationships between information elements, and the accuracy of these relationships might not meet requirements. Therefore, existing natural language-based segmentation methods cannot be directly applied to the packet sequence generation process based on the transformer model structure.

[0100] To address the aforementioned technical problems, this application provides a method for word segmentation of network packets. Based on the characteristics of network packets and word segmentation requirements, the method parses key-value pairs in the packet, segments the key and value in each key-value pair into words, and obtains a first token sequence corresponding to the key and a second token sequence corresponding to the value. Then, the first token sequence and the second token sequence are combined to obtain a first composite token sequence of the original packet code stream. This first composite token sequence can be used by a transformer model to perform token sampling distribution analysis to generate the packet sequence to be transmitted.

[0101] This approach segments the keys and values ​​in key-value pairs separately, creating a data structure that is easy for the model to understand. This structure can be directly applied to the message sequence generation process based on the transformer model, avoiding the problem of the model spending a lot of parameters to learn the relevant syntactic structure of the message and the relationship between information elements.

[0102] like Figure 2 As shown, one embodiment of this application provides a network message segmentation processing method, including:

[0103] S210, each message in the original code stream of the first message to be processed is parsed to obtain key-value pairs of multiple information elements in the original code stream of the first message;

[0104] S220, converting the keys in the plurality of key-value pairs into key text information, and converting the values ​​in the plurality of key-value pairs into value text information;

[0105] S230, determine the first token sequence corresponding to the key text information, and determine the second token sequence corresponding to the value text information;

[0106] S240, based on the arrangement order of the key-value pairs of the multiple information elements, the first token sequence and the second token sequence are combined to obtain the first synthetic token sequence of the original code stream of the first message.

[0107] The word segmentation method for network packets described in this application involves parsing the original code stream of a first packet to obtain multiple information elements and their key-value pairs. Then, by segmenting and encoding the keys and values ​​in the key-value pairs of the multiple information elements in the original code stream, a first token sequence and a second token sequence are obtained. After combining the first token sequence and the second token sequence, a first composite token sequence is obtained, which is output as the token sequence corresponding to the original code stream of the first packet. This sequence is used by the transformer model for token sampling distribution analysis. By segmenting the keys and values ​​in the key-value pairs of information elements, the generated token sequence forms a data structure that is easy for the model to understand. This avoids the problems of traditional language model word segmentation methods, where key-value pairs may lose or miss explicit association information between information element keys and values, and may also lead to insufficient context length.

[0108] Optionally, by using the first lexical relationship information of the corresponding information element key and the second lexical relationship information of the corresponding information element value obtained through pre-training, the key and value of multiple information elements in the original code stream of the first message are respectively segmented and encoded to obtain the first word element token sequence and the second word element token sequence.

[0109] In some embodiments, optionally, in step S230, determining the first token sequence corresponding to the key text information includes:

[0110] Based on the pre-obtained first lexical relationship information, the first token sequence corresponding to the key text information is determined; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or,

[0111] In step S230, determining the second token sequence corresponding to the value text information includes:

[0112] Based on the pre-obtained second lexical relationship information, the second token sequence corresponding to the value text information is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0113] By employing this implementation method, key-value pair training is performed on multiple training message raw code streams to obtain corresponding first and second word relation information, which are used to determine the first token sequence corresponding to the key text information and the second token sequence corresponding to the value text information. This makes the determination process of the first and second token sequences more accurate and easier to implement.

[0114] like Figure 3 This is a schematic diagram of the message processing flow using the word segmentation processing method described in the embodiments of this application. This word segmentation processing method can be applied to a word segmenter, or word segmentation device. The specific message processing flow includes:

[0115] S301, Obtain the original bitstream of the message (e.g., the original bitstream of the first message);

[0116] S302, the acquired original message code stream is input into the word segmenter, which executes the word segmentation processing method of this application embodiment, uses the first word relation information and the second word relation information to parse and encode the original message code stream, and outputs the first synthesized token sequence of the original message code stream;

[0117] S303, the first synthetic token sequence is input into the model deployment device. Optionally, the model deployment device is equipped with a transformer model. After performing token sampling distribution analysis on the first synthetic token sequence using the transformer model, a second synthetic token sequence is obtained.

[0118] S304, the second synthesized token sequence is input into the word segmenter. In this embodiment of the application, the word segmenter implementing the word segmentation processing method described in this embodiment of the application is further used to convert the token in the second synthesized token sequence into a text bytecode stream according to the first lexical relationship information and the second lexical relationship information, so as to generate the original code stream of the message to be transmitted (such as the original code stream of the second message).

[0119] S305 outputs the raw bitstream of the message to be transmitted.

[0120] In one embodiment of the word segmentation processing method described in this application, the word segmenter is optionally further used to collect the original message code stream for model training. By statistically analyzing the collected original message code stream, the key and value of multiple information elements in the original message code stream are used as training data respectively. Based on the basic lexical relationship information, the first lexical relationship information of the corresponding information element key and the second lexical relationship information of the corresponding information element value are obtained.

[0121] Therefore, in this embodiment of the application, the word segmenter using the word segmentation processing method described in this embodiment of the application has the following functions:

[0122] The vocabulary training function is used to obtain the first vocabulary relationship information of the corresponding information element key and the second vocabulary relationship information of the corresponding information element value based on the basic vocabulary relationship information, using the keys and values ​​of multiple information elements in the original code stream of the message as training data respectively.

[0123] The word segmenter encoding function is used to generate a token sequence of the original message code stream based on the first and second word relationship information obtained through training.

[0124] The word segmenter decoding function is used to convert the token sequence into the corresponding text bytecode stream based on the first and second word relationship information obtained through training, and to generate the original code stream of the message to be transmitted based on the converted text bytecode stream.

[0125] Based on the above, the word segmentation processing method described in the embodiments of this application further includes a word list training process. Therefore, in one embodiment, optionally, the method further includes:

[0126] Obtain the third message raw code stream for training; parse each message in the third message raw code stream to obtain key-value pairs of multiple information elements in the third message raw code stream;

[0127] The keys of multiple key-value pairs in the original code stream of the third message are converted into key text information, and the values ​​of multiple key-value pairs in the original code stream of the third message are converted into value text information; a corresponding token is assigned based on the key text information converted from the original code stream of the third message to obtain the first lexical relationship information; and a corresponding token is assigned based on the value text information converted from the original code stream of the third message to obtain the second lexical relationship information.

[0128] Combination Figure 4 As shown, using the word segmentation processing method described in the embodiments of this application, during the vocabulary training process, after obtaining the original code stream of the third message used for training, the individual messages in the original code stream of the third message are parsed sequentially to obtain key-value pairs of multiple information elements in the original code stream of the third message.

[0129] In one implementation, optionally, a message parsing tool can be used to parse the original code stream of the third message. The message parsing tool can parse the communication protocol stack and output the information elements of each layer of the protocol in the original code stream of the third message in the format of key-value pairs to form a sequence of multiple key-value pairs.

[0130] In one implementation, optionally, each message is parsed to obtain multiple key-value pairs of information elements, including:

[0131] Each message is parsed to obtain the lowest-level information element in each message;

[0132] Obtain the key-value pairs of the lowest-level information element.

[0133] In this implementation method, for each message containing multiple information elements, it is necessary to expand and subdivide them to the lowest level to obtain the lowest level information elements, and output the key-value pairs corresponding to the lowest level information elements to ensure the accuracy of key-value pair parsing.

[0134] Optionally, it is not necessary to parse the key-value pairs of intermediate layer cells. For example, if the lowest layer cell is tls.handshake.sig_hash_alg, then it is not necessary to parse and output the key-value pairs of tls.handshake.

[0135] In one embodiment, after parsing each message to obtain key-value pairs of multiple information cells, the method further includes one or more of the following:

[0136] If the first information cell in a set of multiple information cells is in the tag-length-value TLV format, and the key-value pair of the L field in the first information cell has a fixed value, delete the key-value pair of the L field in the first information cell and store the numerical information of the L field of the first information cell.

[0137] If the first information element in a plurality of information elements is in TLV format and the value corresponding to the V field of the first information element is of text type, delete the key-value pair of the information element corresponding to the L field in the first information element; for multiple information elements belonging to the same level and having the same type, add a number signal to the key in the key-value pair corresponding to the multiple information elements respectively.

[0138] In one implementation, optionally, during the parsing of the message, if the parsed information element is in TLV format (first information element) and the value of the key-value pair corresponding to the information element is a fixed value, the key-value pair corresponding to the L field in the information element can be deleted, and only the numerical information of the L field corresponding to the information element is saved in the generated first word relation information, that is, the length information of the information element is saved, so as to reduce the length of the message converted into a token sequence.

[0139] In one implementation, optionally, for information elements belonging to the same level and having the same type, numbering information is added to the keys of the key-value pairs corresponding to the multiple information elements to distinguish them. For example, if the TLS protocol contains multiple tls.handshake.sig_hash_alg, these multiple tls.handshake.sig_hash_alg can be numbered and identified as tls.handshake.sig_hash_alg1 and tls.handshake.sig_hash_alg2.

[0140] Alternatively, in another implementation, during the parsing of the message, for TLV information elements in text format, it is not necessary to save the L information element. When the word segmenter decodes, the information element length can be obtained from the text value.

[0141] Based on the above process, each message can be parsed to obtain key-value pairs of multiple information elements. For example, taking an IP layer protocol message as an example, in one implementation method, the key-value pairs of multiple information elements obtained through parsing can be as follows: Figure 5 As shown.

[0142] Optionally, in this embodiment of the application, converting the keys of the multiple key-value pairs into key text information includes:

[0143] Extract the keys from multiple key-value pairs as extracted text;

[0144] The extracted text is segmented to obtain the key text information.

[0145] After extracting the keys from multiple key-value pairs as extracted text, the method further includes one or more of the following:

[0146] Store the type information corresponding to the value in each of the key-value pairs;

[0147] When the information cell has a fixed length, the value length corresponding to the information cell is stored.

[0148] In this implementation, after parsing and obtaining key-value pairs of multiple information cells, only the keys of the information cell sequence are extracted or converted into text, while the type information of the value corresponding to the key is saved. Optionally, in one implementation, this type information may include, but is not limited to, only IP addresses, bytes, integers, and text. Optionally, in another implementation, if the information cell has a fixed length, or if the value length is fixed across all message statistics, then the value length of the corresponding information cell is saved.

[0149] In one embodiment, optionally, it is combined with Figure 4 As shown, the process of extracting the keys from multiple key-value pairs as extracted text can be implemented using a key converter.

[0150] Optionally, after obtaining the key text information corresponding to each key-value pair in the extracted text, the method further includes:

[0151] Assign a corresponding token to the obtained key text information to obtain the first word relationship information.

[0152] Alternatively, in another implementation, the process of segmenting the extracted text and assigning corresponding tokens to the obtained key text information can be implemented using a key segmenter.

[0153] Optionally, when assigning corresponding tokens to the obtained key text information, the tokenizer assigns an index to each unique key text information, i.e., a corresponding token; multiple key text information do not need to be merged. Then, based on the index assigned to each key text information, first lexical relationship information corresponding to the information element key is generated. Thus, after the information element statistics of all messages are completed, the training of the key tokenizer is completed, and the first lexical relationship information is obtained, with each key text information corresponding to a token. For example, the vocabulary (first lexical relationship information) of the key tokenizer generated based on multiple key text information can be as follows: Figure 6 As shown.

[0154] In another implementation, during the vocabulary training process, while converting the keys of multiple key-value pairs in the original code stream of the third message into key text information and allocating corresponding tokens according to the key text information converted from the original code stream of the third message to obtain the first vocabulary relationship information, the process of converting the values ​​of multiple key-value pairs in the original code stream of the third message into value text information and allocating corresponding tokens according to the value text information converted from the original code stream of the third message to obtain the second vocabulary relationship information is also performed.

[0155] Optionally, the method further includes:

[0156] The BPE algorithm is used to train the text-type values ​​in multiple key-value pairs until basic lexical relationship information including a preset number of words is obtained, thus obtaining a word segmentation model.

[0157] Converting the values ​​in the multiple key-value pairs into value text information includes:

[0158] If the value in the key-value pair is of text type, the word segmentation model is used to segment the value in the key-value pair to obtain the text information of the value.

[0159] In this implementation, during vocabulary training, if the value of a key-value pair is of text type, the BPE algorithm is used to train the text-type values ​​among multiple key-value pairs to obtain a word segmentation model. Specifically, the Byte-Pair Encoding (BPE) algorithm can be used, with text-type values ​​among multiple key-value pairs as training data, to train the word segmentation model of the value segmenter, obtaining the trained word segmentation model. Optionally, during the training of the word segmentation model, based on a basic vocabulary relationship information, paired bytes are recursively merged from high to low until a pre-set vocabulary size is reached, thus completing the training of the word segmentation model of the value segmenter.

[0160] Combination Figure 7 , Figure 8 and Figure 9 As shown, the process of training the word segmentation model using the BPE algorithm is as follows:

[0161] Phase 1: BPE learning phase, based on a large amount of corpus, the characters are split into multiple byte pairs, the frequency of each byte pair is determined, and the multiple byte pairs are ranked according to the frequency;

[0162] The second stage, the BPE application stage, is based on a basic vocabulary relationship information. According to the frequency of each byte group analyzed and statistically analyzed in the BPE learning stage, the paired bytes in the basic vocabulary relationship information are recursively merged in descending order of frequency. That is, the two symbols that appear most frequently together are selected from the basic vocabulary relationship information and combined into a new symbol, which is then added to the basic vocabulary relationship information until the pre-set vocabulary size is reached.

[0163] In this embodiment of the application, based on the principle of the BPE algorithm described above, by training the values ​​of the text type in multiple key-value pairs until basic vocabulary relationship information including a preset number of words is obtained, a word segmentation model can be obtained.

[0164] Using this word segmentation model, after converting the values ​​of multiple key-value pairs in the original code stream of the third message into value text information, a corresponding index (i.e., a token) can be assigned to the value text information to obtain the second lexical relationship information (also known as the value segmentation table) corresponding to the word values, such as... Figure 10 As shown.

[0165] In this embodiment of the application, optionally, during the process of converting the values ​​of multiple key-value pairs in the original code stream of the third message into value text information, if the value in the key-value pair is of byte type or integer type, the value in the key-value pair is stored in byte form to the word embedding module. Using this method, when the value of the information element is a byte or integer, there is no need to segment the information element value; the information element value can be directly transmitted to the word embedding module in byte form. The word embedding module is used for encoding when generating the original code stream of the message to be sent based on the token sequence.

[0166] In this implementation, different methods are used to process the values ​​of different types in key-value pairs into value text information, so that values ​​of different types are converted into value text information.

[0167] Alternatively, in another embodiment, the method may further include one or more of the following:

[0168] The data type of the value in the key-value pair is stored;

[0169] If the data type of the value in the key-value pair is address information, the value in the key-value pair is encrypted before being converted into value text information, and the encrypted information is saved.

[0170] Optionally, encrypting the values ​​in the key-value pairs includes performing hash-based de-identification processing on the values. In this embodiment, the stored encrypted information includes the correspondence between the hash value after hash-based de-identification processing and the initial value of the key-value pair.

[0171] This implementation method encrypts the data type of the key-value pair corresponding to the address information to ensure the security of the information element corresponding to the address information during word segmentation. By saving the encrypted information, the corresponding address information can be obtained by reading the encrypted information when generating the transmitted network packet.

[0172] Optionally, this address information includes sensitive information such as IP address and / or port information, combined with... Figure 10 As shown, when the value type corresponding to this information element is sensitive information such as IP address or port, the value needs to be hashed and desensitized. The correspondence between the hash value and the original value is saved to the mapping module for index generation when the token is recovered to the message. Based on the above training process, after training is completed, the first lexical relationship information corresponding to the lexical key and the second lexical relationship information corresponding to the lexical value are obtained. Using the obtained first and second lexical relationship information, the encoding and decoding process of multiple information elements in the original message code stream can be completed.

[0173] Combination Figure 2 and Figure 11As shown, the process of encoding multiple information elements in the original message bitstream using a word segmenter can include:

[0174] Obtain the original bitstream of the first message to be processed; parse each message in the original bitstream of the first message in turn to obtain key-value pairs of multiple information elements in the original bitstream of the first message. In one embodiment, optionally, the original bitstream of the first message can be parsed using a message parsing tool. The parsing method is the same as the method used to parse the information elements in the original bitstream of the third message during the training process, and will not be described in detail here.

[0175] After parsing, the message parsing tool outputs the information elements of multiple messages in the original code stream of the first message in a key-value pair format, forming a sequence of multiple key-value pair information elements.

[0176] After obtaining key-value pairs of multiple information elements in the original code stream of the first message, one processing step converts the keys in the multiple key-value pairs into key text information and determines the first token sequence corresponding to the key text information based on the first lexical relationship information obtained in advance; another processing step converts the values ​​in the multiple key-value pairs into value text information and determines the second token sequence corresponding to the value text information based on the second lexical relationship information obtained in advance.

[0177] See Figure 11 As shown, optionally, the conversion of keys in multiple key-value pairs into key text information is performed by a key converter. That is, the key converter extracts the keys of the information element sequence into text and obtains the key text information corresponding to the key of each information element. Based on the pre-obtained first lexical relationship information, the first token sequence corresponding to the key text information is determined by a key segmenter. That is, the key segmenter outputs the first token sequence corresponding to the keys of multiple information elements based on the first lexical relationship information obtained during the training process.

[0178] In another processing flow, optionally, during the process of converting the values ​​in multiple key-value pairs into value text information, after desensitizing the sensitive information (optionally address information) in the information cell, the values ​​in the key-value pairs of the information cell are divided into text and integer types, and processed in different ways respectively.

[0179] Optionally, if the value in the key-value pair of the information element is a byte or an integer, there is no need to segment the value in the key-value pair. Instead, the value in the key-value pair is stored as the value text information in byte form, that is, the value in the key-value pair is directly output to the corresponding word embedding module.

[0180] Optionally, if the value in the key-value pair of an information element is of text type, then the value in the key-value pair of the information element is extracted, a word segmentation model is used to segment the value in the key-value pair to obtain the value text information, and the second token sequence corresponding to the value text information is determined according to the pre-obtained second lexical relationship information.

[0181] Optionally, the above-mentioned type analysis of the values ​​in key-value pairs and conversion of the values ​​in multiple key-value pairs into value text information can be performed by a value converter; the above-mentioned word segmentation of the values ​​in key-value pairs to obtain value text information and determination of the second token sequence corresponding to the value text information based on the pre-obtained second lexical relationship information can be performed by a value segmenter.

[0182] Optionally, in the embodiments of this application, combined with Figure 2 and Figure 12 As shown, in step S240, the first token sequence and the second token sequence are combined according to the arrangement order of the key-value pairs of the multiple information cells to obtain a first synthetic token sequence of the first message original code stream, including:

[0183] For each key-value pair, extract the third token corresponding to the key in the key-value pair in the first token sequence, and extract the fourth token corresponding to the value in the key-value pair in the second token sequence;

[0184] Based on the arrangement order of the key-value pairs of the multiple information elements, the third token and the fourth token corresponding to each key-value pair are arranged sequentially adjacent to each other to obtain the first synthetic token sequence.

[0185] In this implementation method, the third token corresponding to the key and the fourth token corresponding to the value in the key-value pairs of multiple information cells in the obtained first synthetic token sequence are arranged in a sequential structure, so as to facilitate the parsing of the token sequence and the generation of the original code stream of the message during the generation process of the network message.

[0186] Optionally, the method further includes:

[0187] Add a message end identifier to the end of the token sequence corresponding to each message in the first synthesized token sequence.

[0188] Specifically, when combining the first token sequence and the second token sequence, the process is as follows:

[0189] Based on the order of key-value pairs of multiple information cells, the third token corresponding to the key and the fourth token corresponding to the value are arranged sequentially, with the key in the odd position of the sequence and the value in the even position of the sequence, and the corresponding key and value are adjacent to each other.

[0190] Optionally, the length of the third token of the key is always 1, while the length of the fourth token of the value is determined according to the text segmentation, and can be 1 or more. If the two lengths are different, the key token length is copied n positions, where n is the value token length.

[0191] Furthermore, during the process of combining the first token sequence and the second token sequence, after the key-value pairs of all cells in one of the messages are output to the corresponding synthesized token sequence, a message end marker is added to the end of the synthesized token sequence of the corresponding message, such as... Figure 12 As shown.

[0192] For example, the message end identifier can be "00 00", which is used to indicate the end of the corresponding message token sequence.

[0193] Using the above implementation method, after generating the first synthetic token sequence, the generated first synthetic token sequence can be transmitted to the transformer model for training or inference to perform token sampling distribution analysis and obtain the second synthetic token sequence. The obtained second synthetic token sequence can be further decoded to realize the process of generating a message from a token sequence.

[0194] Optionally, in the embodiments of this application, the token sequences corresponding to two adjacent messages in the first synthesized token sequence are separated by a message end identifier.

[0195] In this implementation, the token sequences corresponding to two adjacent messages are separated by a message end identifier, so that the token sequences corresponding to different messages can be distinguished when generating network messages.

[0196] Optionally, in this embodiment of the application, the method further includes:

[0197] Obtain the second synthesized token sequence output by the transformer model; the second synthesized token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information.

[0198] The corresponding key text information is determined based on the key indicator bit in the second synthesized token sequence, and the corresponding value text information is determined based on the value indicator bit in the second synthesized token sequence.

[0199] The key corresponding to the key text information and the value corresponding to the value text information are combined to generate key-value pairs of multiple information elements in the original code stream of the first message;

[0200] Based on the key-value pairs, the original code stream of the first message is obtained.

[0201] The corresponding key text information is determined based on the key indicator bit in the second synthesized token sequence, including:

[0202] Based on the pre-obtained first lexical relationship information, determine the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or

[0203] The corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0204] Based on the pre-obtained second lexical relationship information, the value text information corresponding to the token on the value indicator bit in the second synthesized token sequence is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0205] In this embodiment of the application, when a message end marker is added to the end of the token sequence of each message during the generation of the first synthetic token sequence, the method further includes:

[0206] Based on the message end identifier interval in the second synthesized token sequence, the second synthesized token sequence is split into multiple third token sequences, each of which corresponds to a message;

[0207] Specifically, determining the corresponding key text information based on the token at the key indicator bit in the second synthesized token sequence, and determining the corresponding value text information based on the token at the value indicator bit in the second synthesized token sequence, includes:

[0208] For each of the third token sequences, the corresponding key text information is determined based on the token on the corresponding key indicator bit; and for each of the third token sequences, the corresponding value text information is determined based on the token on the corresponding value indicator bit.

[0209] In the second synthesized token sequence, the key indicator bit and value indicator bit corresponding to a single information element are adjacent, while the key indicator bit and value indicator bit corresponding to different information elements are spaced apart.

[0210] Combination Figure 13 and Figure 14 As shown, after obtaining the second synthesized token sequence output by the transformer model, the second synthesized token sequence is split into multiple third token sequences according to the message end identifier in the second synthesized token sequence.

[0211] Subsequently, multiple third token sequences are parsed sequentially to determine the token corresponding to the key after converting the key in the key-value pair into key text information, and the token corresponding to the value in the key-value pair after converting it into value text information. Based on this, the corresponding key text information is determined according to the first lexical relationship information and the token on the corresponding key indicator bit in the second synthesized token sequence, and the corresponding value text information is obtained according to the second lexical relationship information and the token on the corresponding value indicator bit in the second synthesized token sequence. The key corresponding to the key text information and the value corresponding to the value text information are synthesized to generate the key-value pair of the first message raw code stream to be sent.

[0212] Specifically, after obtaining the second synthesized token sequence, if the token corresponding to the key text information is arranged in odd positions and the token corresponding to the value text information is arranged in even positions, when parsing each third token sequence, the tokens in the odd positions are extracted as the tokens corresponding to the key text information, and the tokens in the even positions are extracted as the tokens corresponding to the value text information.

[0213] In addition, the following processing is performed when parsing the third token sequence:

[0214] Multiple consecutive identical keys are deduplicated into a single key;

[0215] The values ​​corresponding to consecutive identical keys are merged into one value;

[0216] Discard the token corresponding to the end-of-message identifier.

[0217] Optionally, the process of splitting the second synthetic token sequence described above can be performed by the key-value scattering module.

[0218] Optionally, determining the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence based on the first lexical relationship information can be performed by a key segmenter; determining the value text information corresponding to the token at the value indicator bit in the second synthesized token sequence based on the second lexical relationship information can be performed by a value parser.

[0219] In one embodiment, optionally, after determining the token corresponding to the key in the key-value pair after converting the key into key text information, when determining the key text information corresponding to the token on the key indicator bit in the second synthesized token sequence according to the first lexical relationship information, the key text information corresponding to the token is obtained in the first lexical relationship information, and the data type and data length of the corresponding value are determined according to the data type and data length of the key-corresponding value stored during encoding.

[0220] In one embodiment, optionally, the method further includes:

[0221] Based on the storage information at the time of generation of the first synthesized token sequence in the original code stream of the first message, the data type of the value in the key-value pair corresponding to the key text information is obtained; wherein, the first synthesized token sequence is input to the transformer model and the second synthesized token sequence is output.

[0222] If the data type of the value in the key-value pair corresponding to the key text information is text, the value text information determined according to the token on the corresponding value indicator bit is decoded to determine the value corresponding to the value text information.

[0223] Using this implementation method, the data type of the value corresponding to the stored key text information can be obtained based on the stored information during the generation of the first synthetic token sequence; based on the obtained data type, the token on the value text information corresponding to the key text information is parsed to determine the value corresponding to the value text information.

[0224] Optionally, the data type of the value corresponding to the key text information can be obtained by the key tokenizer.

[0225] In one embodiment, if the data length corresponding to the key text information is also stored during the generation of the first synthetic token sequence, the data length can be obtained from the key segmenter and verified.

[0226] Optionally, if the token corresponding to the value text information is a sensitive field (such as address information) that has been hashed, the original value text information corresponding to the token is obtained from the mapping module.

[0227] In this embodiment, the method further includes:

[0228] If the data type of the value in the key-value pair corresponding to the key text information is address information, the token on the corresponding value indicator bit in the second synthesized token sequence is decrypted according to the encryption information stored in the storage information.

[0229] Specifically, the corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0230] The corresponding value text information is determined based on the decrypted token.

[0231] In this embodiment of the application, when the data type of the value corresponding to the key text information is text, the value corresponding to the token on the value indicator bit is obtained according to the second lexical relationship information. That is, the original information content of the value text information corresponding to the token is obtained by decoding and restoring it using a value segmenter.

[0232] Based on this, after obtaining the key corresponding to the token on the key indicator bit according to the first vocabulary relationship information, and obtaining the value corresponding to the token on the value indicator bit according to the second vocabulary relationship information, the obtained key and value are merged to generate the key-value pair of the first message raw code stream to be sent.

[0233] Optionally, the message synthesis module can synthesize the obtained keys and values ​​to form key-value pairs.

[0234] In one embodiment, optionally, the method further includes:

[0235] Based on the key-value pairs, cell filling is performed in the message template of the first message original code stream to obtain the first message original code stream.

[0236] Specifically, after key-value synthesis, the generated message template is used to fill information cells. Once all key-value pairs corresponding to information cells are filled, the byte code stream to be sent is obtained, which is the original code stream of the first message.

[0237] The word segmentation processing method described in this application is optimized and adjusted according to the characteristics of network packets and analysis requirements, which can improve the accuracy and efficiency of word segmentation. Specifically:

[0238] This method addresses the issue that tokens generated by traditional large-scale language models cannot handle scenarios where the same code stream at different locations in different layers of network packets has different representational meanings. The identified key-value pair structures, including but not limited to TLV (Tag-Length-Value) format and other custom protocol formats, reduce the difficulty for the model to understand the semantics of network packets, significantly reducing the demands on model parameters and performance. Furthermore, by segmenting keys and values ​​separately, a data structure easily understood by the model is formed. Compared to direct segmentation of the packet code stream, the large-scale language model reduces the number of tokens occupied by the code stream, significantly reducing the demand on model context length. This approach also solves the parsing problem of complex information cell structures containing multiple identical information cells in nested packet protocol stacks, achieving more efficient and accurate semantic association.

[0239] One embodiment of this application also provides a method for generating network packets, such as... Figure 15 As shown, it includes:

[0240] S1501, Obtain the second synthesized token sequence output by the transformer model; The second synthesized token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and includes the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information.

[0241] S1502, determine the corresponding key text information based on the token on the key indicator bit in the second synthesized token sequence, and determine the corresponding value text information based on the token on the value indicator bit in the second synthesized token sequence;

[0242] S1503, combine the key corresponding to the key text information and the value corresponding to the value text information to generate key-value pairs of multiple information elements in the original code stream of the first message;

[0243] S1504, Obtain the original code stream of the first message based on the key-value pair.

[0244] The network packet generation method described in this application, during word segmentation, parses key-value pairs in the packet, segments the keys and values ​​in the key-value pairs separately, and obtains a first token sequence corresponding to the keys and a second token sequence corresponding to the values. Then, the first token sequence and the second token sequence are combined to obtain a first synthetic token sequence of the original packet code stream. This first synthetic token sequence can be used for token sampling distribution analysis by the transformer model. The second synthetic token sequence output by the transformer model is obtained by extracting the tokens corresponding to the keys and the tokens corresponding to the values ​​in the key-value pairs in the second synthetic token sequence and synthesizing them. Using this method, the token sequence processing method in the packet segmentation and synthesis process forms a data structure that is easy for the model to understand. It can be directly applied to the packet sequence generation process based on the transformer model structure, avoiding the problem of the model spending a lot of parameters to learn the relevant grammatical structure and the relationship between information elements of the packet.

[0245] Optionally, the generation method, wherein determining the corresponding key text information based on the token at the corresponding key indicator bit in the second synthesized token sequence, includes:

[0246] Based on the pre-obtained first lexical relationship information, determine the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or

[0247] The corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0248] Based on the pre-obtained second lexical relationship information, the value text information corresponding to the token on the value indicator bit in the second synthesized token sequence is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0249] Optionally, in the generation method, in the second synthesized token sequence, the key indicator bit and value indicator bit corresponding to a single information element are adjacent, while the key indicator bit and value indicator bit corresponding to different information elements are spaced apart.

[0250] Optionally, the generation method further includes:

[0251] Based on the message end identifier interval in the second synthesized token sequence, the second synthesized token sequence is split into multiple third token sequences, each of which corresponds to a message;

[0252] Specifically, determining the corresponding key text information based on the token at the key indicator bit in the second synthesized token sequence, and determining the corresponding value text information based on the token at the value indicator bit in the second synthesized token sequence, includes:

[0253] For each of the third token sequences, the corresponding key text information is determined based on the token on the corresponding key indicator bit; and for each of the third token sequences, the corresponding value text information is determined based on the token on the corresponding value indicator bit.

[0254] Optionally, the generation method further includes:

[0255] Based on the storage information at the time of generation of the first synthesized token sequence in the original code stream of the first message, the data type of the value in the key-value pair corresponding to the key text information is obtained; wherein, the first synthesized token sequence is input to the transformer model and the second synthesized token sequence is output.

[0256] If the data type of the value in the key-value pair corresponding to the key text information is text, the value text information determined according to the token on the corresponding value indicator bit is decoded to determine the value corresponding to the value text information.

[0257] Optionally, the generation method further includes:

[0258] If the data type of the value in the key-value pair corresponding to the key text information is address information, the token on the corresponding value indicator bit in the second synthesized token sequence is decrypted according to the encryption information stored in the storage information.

[0259] Specifically, the corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0260] The corresponding value text information is determined based on the decrypted token.

[0261] For a detailed description of the network packet generation method described in this application, please refer to the detailed description of the network packet word segmentation processing method described above, which will not be repeated here.

[0262] One embodiment of this application also provides a network message segmentation processing device, such as... Figure 16 As shown, it includes:

[0263] The first parsing module 1601 is used to parse each message in the original code stream of the first message to be processed, and obtain key-value pairs of multiple information elements in the original code stream of the first message.

[0264] The conversion module 1602 is used to convert the keys in the multiple key-value pairs into key text information, and to convert the values ​​in the multiple key-value pairs into value text information;

[0265] The determining module 1603 is used to determine the first token sequence corresponding to the key text information, and to determine the second token sequence corresponding to the value text information;

[0266] The combination module 1604 is used to combine the first token sequence and the second token sequence according to the arrangement order of the key-value pairs of the multiple information cells to obtain a first synthetic token sequence of the original code stream of the first message.

[0267] Optionally, in the word segmentation processing device, the determining module 1603 determines the first token sequence corresponding to the key text information, including:

[0268] Based on the pre-obtained first lexical relationship information, the first token sequence corresponding to the key text information is determined; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or,

[0269] The determining module 1603 determines the second token sequence corresponding to the value text information, including:

[0270] Based on the pre-obtained second lexical relationship information, the second token sequence corresponding to the value text information is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0271] Optionally, in the word segmentation processing device, the combination module 1604 combines the first token sequence and the second token sequence according to the arrangement order of the key-value pairs of the plurality of information elements to obtain a first synthesized token sequence of the first message original code stream, including:

[0272] For each key-value pair, extract the third token corresponding to the key in the key-value pair in the first token sequence, and extract the fourth token corresponding to the value in the key-value pair in the second token sequence;

[0273] Based on the arrangement order of the key-value pairs of the multiple information elements, the third token and the fourth token corresponding to each key-value pair are arranged sequentially adjacent to each other to obtain the first synthetic token sequence.

[0274] Optionally, in the word segmentation processing apparatus, the determining module 1603 is further configured to perform one or more of the following:

[0275] The data type of the value in the key-value pair is stored;

[0276] If the data type of the value in the key-value pair is address information, the value in the key-value pair is encrypted before being converted into value text information, and the encrypted information is saved.

[0277] Optionally, in the word segmentation processing device, in the first synthesized token sequence, the token sequences corresponding to two adjacent messages are separated by a message end identifier.

[0278] Optionally, in the word segmentation processing device, the first parsing module 1601 parses each message to obtain multiple key-value pairs of information elements, including:

[0279] Each message is parsed to obtain the lowest-level information element among the multiple information elements of each message;

[0280] Obtain the key-value pairs of the lowest-level information element.

[0281] Optionally, in the word segmentation processing apparatus, the conversion module 1602 converts the keys of the multiple key-value pairs into key text information, including:

[0282] Extract the keys from multiple key-value pairs as extracted text;

[0283] The extracted text is segmented to obtain the key text information.

[0284] Optionally, in the word segmentation processing apparatus, the conversion module 1602 converts the values ​​in the plurality of key-value pairs into value text information, including:

[0285] When the value in the key-value pair is of text type, the word segmentation model is used to segment the value in the key-value pair to obtain the text information of the value;

[0286] If the value in the key-value pair is of type byte or integer, the value in the key-value pair is stored as the value text information in byte form.

[0287] Optionally, in the aforementioned word segmentation processing apparatus, after the first parsing module 1601 parses each message and obtains key-value pairs of multiple information elements, it is further configured to perform one or more of the following:

[0288] If the first information cell in a set of multiple information cells is in the tag-length-value TLV format, and the key-value pair of the L field in the first information cell has a fixed value, delete the key-value pair of the L field in the first information cell and store the numerical information of the L field of the first information cell.

[0289] If the first information element in a set of multiple information elements is in TLV format and the value corresponding to the V field of the first information element is of text type, delete the key-value pair of the information element corresponding to the L field in the first information element.

[0290] One embodiment of this application also provides a network packet generation apparatus, such as... Figure 17 As shown, it includes:

[0291] The first acquisition module 1701 is used to acquire the second synthetic token sequence output by the transformer model; the second synthetic token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information.

[0292] The determining module 1702 is used to determine the corresponding key text information based on the token on the key indicator bit in the second synthesized token sequence, and to determine the corresponding value text information based on the token on the value indicator bit in the second synthesized token sequence.

[0293] The synthesis module 1703 is used to synthesize the key corresponding to the key text information and the value corresponding to the value text information to generate key-value pairs of multiple information elements in the original code stream of the first message;

[0294] The second acquisition module 1704 is used to obtain the original code stream of the first message based on the key-value pair.

[0295] Optionally, in the generation apparatus, the determining module 1702 determines the corresponding key text information based on the token at the corresponding key indicator bit in the second synthesized token sequence, including:

[0296] Based on the pre-obtained first lexical relationship information, determine the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or

[0297] The determining module 1702 determines the corresponding value text information based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0298] Based on the pre-obtained second lexical relationship information, the value text information corresponding to the token on the value indicator bit in the second synthesized token sequence is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

[0299] Optionally, in the generation apparatus, in the second synthesized token sequence, the key indicator bit and value indicator bit corresponding to one information element are adjacent, and the key indicator bit and value indicator bit corresponding to different information elements are spaced apart.

[0300] Optionally, in the generating apparatus, the determining module 1702 is further configured to:

[0301] Based on the message end identifier interval in the second synthesized token sequence, the second synthesized token sequence is split into multiple third token sequences, each of which corresponds to a message;

[0302] The determining module 1702 determines the corresponding key text information based on the token at the key indicator bit in the second synthesized token sequence, and determines the corresponding value text information based on the token at the value indicator bit in the second synthesized token sequence, including:

[0303] For each of the third token sequences, the corresponding key text information is determined based on the token on the corresponding key indicator bit; and for each of the third token sequences, the corresponding value text information is determined based on the token on the corresponding value indicator bit.

[0304] Optionally, in the generating apparatus, the determining module 1702 is further configured to:

[0305] Based on the storage information at the time of generation of the first synthesized token sequence in the original code stream of the first message, the data type of the value in the key-value pair corresponding to the key text information is obtained; wherein, the first synthesized token sequence is input to the transformer model and the second synthesized token sequence is output.

[0306] If the data type of the value in the key-value pair corresponding to the key text information is text, the value text information determined according to the token on the corresponding value indicator bit is decoded to determine the value corresponding to the value text information.

[0307] Optionally, in the generating apparatus, the determining module 1702 is further configured to:

[0308] If the data type of the value in the key-value pair corresponding to the key text information is address information, the token on the corresponding value indicator bit in the second synthesized token sequence is decrypted according to the encryption information stored in the storage information.

[0309] Specifically, the determining module 1702 determines the corresponding value text information based on the token at the corresponding value indicator bit in the second synthesized token sequence, including:

[0310] The corresponding value text information is determined based on the decrypted token.

[0311] One embodiment of this application also provides a network device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the network packet segmentation processing method as described in any of the preceding claims or the network packet generation method as described in any of the preceding claims.

[0312] The specific implementation of the word segmentation processing method or generation method executed by the program running on the processor of the network device can be found in the detailed description of the above specific implementation, and will not be repeated here.

[0313] In addition, specific embodiments of this application also provide a readable storage medium storing a computer program thereon, wherein when the program is executed by a processor, it implements the steps in the network packet segmentation processing method as described in any of the above claims or the steps in the network packet generation method as described in any of the above claims.

[0314] Specifically, the readable storage medium is applied to the aforementioned network device. When applied to the network device, the execution steps in the corresponding word segmentation processing method or generation method are described in detail above, and will not be repeated here.

[0315] Another embodiment of this application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the steps in the word segmentation processing method as described above, or implement the steps in the generation method as described above.

[0316] Optionally, embodiments of this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0317] The computer program product described in this application includes computer instructions that, when executed by a processor, implement the various processes of the method embodiments shown above and achieve the same technical effects. To avoid repetition, these will not be repeated here.

[0318] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0319] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0320] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0321] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing a network message, characterized in that, include: Each message in the original code stream of the first message to be processed is parsed to obtain key-value pairs of multiple information elements in the original code stream of the first message; Convert the keys in the multiple key-value pairs into key text information, and convert the values ​​in the multiple key-value pairs into value text information; Determine the first token sequence corresponding to the key text information, and determine the second token sequence corresponding to the value text information; Based on the order of the key-value pairs of the multiple information elements, the first token sequence and the second token sequence are combined to obtain the first synthetic token sequence of the original code stream of the first message.

2. The word segmentation processing method of claim 1, wherein, Determining the first token sequence corresponding to the key text information includes: Based on the pre-obtained first lexical relationship information, the first token sequence corresponding to the key text information is determined; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or, Determining the second token sequence corresponding to the value text information includes: Based on the pre-obtained second lexical relationship information, the second token sequence corresponding to the value text information is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

3. The word segmentation processing method according to claim 1, characterized in that, Based on the arrangement order of the key-value pairs of the multiple information cells, the first token sequence and the second token sequence are combined to obtain a first synthetic token sequence of the original code stream of the first message, including: For each key-value pair, extract the third token corresponding to the key in the key-value pair in the first token sequence, and extract the fourth token corresponding to the value in the key-value pair in the second token sequence; Based on the arrangement order of the key-value pairs of the multiple information elements, the third token and the fourth token corresponding to each key-value pair are arranged sequentially adjacent to each other to obtain the first synthetic token sequence.

4. The word segmentation processing method according to claim 1, characterized in that, The method further includes one or more of the following: The data type of the value in the key-value pair is stored; If the data type of the value in the key-value pair is address information, the value in the key-value pair is encrypted before being converted into value text information, and the encrypted information is saved.

5. The word segmentation processing method according to claim 1, characterized in that, In the first synthesized token sequence, the token sequences corresponding to two adjacent messages are separated by a message end identifier.

6. The word segmentation processing method according to claim 1, characterized in that, Each message is parsed to obtain key-value pairs of multiple information elements, including: Each message is parsed to obtain the lowest-level information element among the multiple information elements of each message; Obtain the key-value pairs of the lowest-level information element.

7. The word segmentation processing method according to claim 1, characterized in that, Converting the keys of multiple key-value pairs into key text information includes: Extract the keys from multiple key-value pairs as extracted text; The extracted text is segmented to obtain the key text information.

8. The word segmentation processing method according to claim 1, characterized in that, Converting the values ​​in the multiple key-value pairs into value text information includes: When the value in the key-value pair is of text type, the word segmentation model is used to segment the value in the key-value pair to obtain the text information of the value; If the value in the key-value pair is of type byte or integer, the value in the key-value pair is stored as the value text information in byte form.

9. The word segmentation processing method according to claim 1, characterized in that, After parsing each message to obtain key-value pairs of multiple information cells, the method further includes one or more of the following: If the first information cell in a set of multiple information cells is in the tag-length-value TLV format, and the key-value pair of the L field in the first information cell has a fixed value, delete the key-value pair of the L field in the first information cell and store the numerical information of the L field of the first information cell. If the first information element in a set of multiple information elements is in TLV format and the value corresponding to the V field of the first information element is of text type, delete the key-value pair of the information element corresponding to the L field in the first information element.

10. A method for generating network packets, characterized in that, include: Obtain the second synthesized token sequence output by the transformer model; the second synthesized token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information. The corresponding key text information is determined based on the key indicator bit in the second synthesized token sequence, and the corresponding value text information is determined based on the value indicator bit in the second synthesized token sequence. The key corresponding to the key text information and the value corresponding to the value text information are combined to generate key-value pairs of multiple information elements in the original code stream of the first message; Based on the key-value pairs, the original code stream of the first message is obtained.

11. The generation method according to claim 10, characterized in that, The corresponding key text information is determined based on the key indicator bit in the second synthesized token sequence, including: Based on the pre-obtained first lexical relationship information, determine the key text information corresponding to the token at the key indicator bit in the second synthesized token sequence; wherein, the first lexical relationship information includes tokens corresponding to multiple different key text information; the first lexical relationship information is obtained by converting the keys in the key-value pairs of information elements in multiple training message original code streams into key text information, and then assigning corresponding tokens to the corresponding key text information; and / or The corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including: Based on the pre-obtained second lexical relationship information, the value text information corresponding to the token on the value indicator bit in the second synthesized token sequence is determined; wherein, the second lexical relationship information includes tokens corresponding to multiple different value text information respectively, and the second lexical relationship information is obtained by converting the values ​​in the key-value pairs of multiple training message original code streams into value text information, and then assigning corresponding tokens to the corresponding value text information.

12. The generation method according to claim 10, characterized in that, In the second synthesized token sequence, the key indicator bit and value indicator bit corresponding to a single information element are adjacent, while the key indicator bit and value indicator bit corresponding to different information elements are spaced apart.

13. The generation method according to claim 10, characterized in that, The method further includes: Based on the message end identifier interval in the second synthesized token sequence, the second synthesized token sequence is split into multiple third token sequences, each of which corresponds to a message; Specifically, determining the corresponding key text information based on the token at the key indicator bit in the second synthesized token sequence, and determining the corresponding value text information based on the token at the value indicator bit in the second synthesized token sequence, includes: For each of the third token sequences, the corresponding key text information is determined based on the token on the corresponding key indicator bit; and for each of the third token sequences, the corresponding value text information is determined based on the token on the corresponding value indicator bit.

14. The generation method according to claim 11, characterized in that, The method further includes: Based on the storage information at the time of generation of the first synthesized token sequence in the original code stream of the first message, the data type of the value in the key-value pair corresponding to the key text information is obtained; wherein, the first synthesized token sequence is input to the transformer model and the second synthesized token sequence is output. If the data type of the value in the key-value pair corresponding to the key text information is text, the value text information determined according to the token on the corresponding value indicator bit is decoded to determine the value corresponding to the value text information.

15. The generation method according to claim 14, characterized in that, The method further includes: If the data type of the value in the key-value pair corresponding to the key text information is address information, the token on the corresponding value indicator bit in the second synthesized token sequence is decrypted according to the encryption information stored in the storage information. Specifically, the corresponding value text information is determined based on the token at the corresponding value indicator bit in the second synthesized token sequence, including: The corresponding value text information is determined based on the decrypted token.

16. A word segmentation processing device for network messages, characterized in that, include: The first parsing module is used to parse each message in the original code stream of the first message to be processed, and obtain key-value pairs of multiple information elements in the original code stream of the first message. A conversion module is used to convert the keys in the multiple key-value pairs into key text information, and to convert the values ​​in the multiple key-value pairs into value text information; The determining module is used to determine the first token sequence corresponding to the key text information, and to determine the second token sequence corresponding to the value text information; The combination module is used to combine the first token sequence and the second token sequence according to the arrangement order of the key-value pairs of the multiple information elements to obtain a first synthetic token sequence of the original code stream of the first message.

17. A network packet generation apparatus, characterized in that, include: The first acquisition module is used to acquire the second synthetic token sequence output by the transformer model; the second synthetic token sequence includes the token corresponding to the key in the key-value pair of multiple information elements in the original code stream of the first message to be sent after converting the key into key text information, and the token corresponding to the value in the key-value pair of the multiple information elements after converting it into value text information. The determining module is used to determine the corresponding key text information based on the token on the key indicator bit in the second synthesized token sequence, and to determine the corresponding value text information based on the token on the value indicator bit in the second synthesized token sequence; The synthesis module is used to synthesize the key corresponding to the key text information and the value corresponding to the value text information to generate key-value pairs of multiple information elements in the original code stream of the first message; The second acquisition module is used to obtain the original code stream of the first message based on the key-value pair.

18. A network device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the word segmentation processing method as described in any one of claims 1 to 9, or implements the network packet generation method as described in any one of claims 10 to 15.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the word segmentation processing method as described in any one of claims 1 to 9, or the steps of the network packet generation method as described in any one of claims 10 to 15.

20. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the word segmentation processing method as described in any one of claims 1 to 9, or the steps of the network packet generation method as described in any one of claims 10 to 15.