Protocol analysis rule generation method, system and equipment based on machine learning

Abstract

The invention provides a protocol analysis rule generation method, system and device based on machine learning. The method comprises the steps of obtaining an original communication message of a device to be accessed; inputting the original communication message into a pre-trained protocol generation model, wherein the protocol generation model is a sequence-to-sequence language model based on an Encoder-Decoder architecture; receiving a machine-readable standardized protocol description format directly output by the protocol generation model; and calling a protocol analysis engine according to the standardized description format, the protocol analysis engine being used for analyzing subsequent similar communication messages from the to-be-accessed device according to the standardized description format, and converting the similar communication messages into a data set with clear field names and data types.

Description

Technical Field

[0001] This application relates to the field of protocol parsing and processing technology, and in particular to a method, system and electronic device for generating protocol parsing rules based on machine learning. Background Technology

[0002] With the rapid expansion of IoT device types and application scenarios, the message formats generated between devices and cloud platforms are becoming increasingly diverse and complex. The traditional model of manually defining parsing rules is no longer sufficient to meet the efficiency and accuracy requirements of large-scale, multi-vendor device access. For massive heterogeneous devices, there is an urgent need for a method that can automatically convert raw messages into machine-readable, scalable protocol rules to reduce manual costs, minimize configuration errors, and accelerate device deployment.

[0003] In related technologies, there are already practices that use compact string tags to describe protocol structures. However, this type of method has obvious shortcomings such as ambiguous structure, missing information, and poor scalability. It cannot directly provide downstream businesses with meaningful field names and data types, thus limiting its practicality in complex industrial, power, meteorological, and other scenarios. Summary of the Invention

[0004] In view of this, it is necessary to provide a method, system and electronic device for generating protocol parsing rules based on machine learning, which can at least overcome one of the above defects.

[0005] In a first aspect, embodiments of this application provide a protocol parsing rule generation method based on machine learning, applied to an IoT device access platform, the method comprising: Obtain the original communication messages of the device to be connected; The original communication message is input into a pre-trained protocol generation model, which is a sequence-to-sequence language model based on an Encoder-Decoder architecture. Receive the standardized, machine-readable protocol description format directly output by the protocol generation model; The protocol parsing engine is invoked according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and convert the similar communication messages into a data set with explicit field names and data types.

[0006] In one embodiment, the method further includes a model training step: Obtain a dataset for model training, which contains multiple sets of historical device communication message samples and the protocol parsing rules corresponding to each sample; Preprocessing operations are performed on historical device communication message samples in the dataset to construct structured training samples for model fine-tuning; wherein, the preprocessing operations include structural parsing and standardization of the message samples according to preset structural analysis rules; Based on the structured training samples and their corresponding protocol parsing rules, a pre-trained sequence-to-sequence language model is adjusted so that the protocol generation model learns to output the standardized protocol description format based on the original communication message.

[0007] In one embodiment, the preprocessing operation further includes: The message samples are identified by communication encoding, and the identified message samples with different encoding forms are converted into string sequences with a unified representation. Based on the preset structural analysis rules, the string sequence is segmented to identify and separate the structural and data parts of the message; The identified structural and data elements are standardized and labeled, and the length or position information of each segment in the original message is marked.

[0008] In one embodiment, adjusting a pre-trained sequence-to-sequence language model based on the structured training samples and their corresponding protocol parsing rules includes: Predefined task prompt information is concatenated before the message sample text in each of the structured training samples to form a complete input sequence for the model training phase. The task prompt information is used to instruct the model to learn the task of generating device messages into protocol rules during training, and to constrain the format specifications that the model output must follow.

[0009] In one embodiment, the segmentation rule based on character position is used to identify protocol header or end fields at fixed positions in the string sequence; The byte-length-based segmentation rules are used to identify data fields with fixed or variable lengths in the string sequence; The segmentation rule based on delimiter features is used to identify delimiters in the string sequence and segment the string sequence into multiple fields; The segmentation rule based on repetition pattern is used to identify groups of fields with repetitive structures in the string sequence.

[0010] In one embodiment, bytes or characters of corresponding length are extracted sequentially from the same type of communication messages according to the protocol unit order defined in the standardized description format; Based on the type and attributes of each protocol unit, the extracted bytes or characters are parsed and converted into the corresponding data type; The parsed data is associated with the field names defined in the protocol unit to form a data set with explicit field names and data types.

[0011] In one embodiment, the standardized description format is a JSON-based structured protocol definition array, where each element in the structured protocol definition array is described as a JSON object of a corresponding protocol unit, and the type of each protocol unit includes at least one of a header, data field, delimiter, terminator, and checksum.

[0012] In one embodiment, when the protocol unit is a data field, the JSON object corresponding to the protocol unit includes at least: name, data type, original message encoding, length, and byte order.

[0013] Secondly, embodiments of this application provide a protocol parsing rule generation system, applied to implement the protocol parsing rule generation method as described in the first aspect, the system comprising: The message acquisition module is used to acquire the original communication messages of the device to be connected. A protocol generation model is used to receive the original communication message. The protocol generation model is a sequence-to-sequence language model based on the Encoder-Decoder architecture. The protocol engine calling module is used to receive the machine-readable standardized protocol description format output by the protocol generation model; and to call the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and to convert the similar communication messages into a data set with explicit field names and data types.

[0014] Secondly, embodiments of this application provide an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to implement the machine learning-based protocol parsing rule generation method described in the first aspect when executing the instructions.

[0015] The protocol parsing rule generation method, system, and electronic device provided in this application, based on machine learning, automatically "translates" raw device messages into well-structured, machine-readable structured protocol definitions (such as JSON format) by training a sequence-to-sequence language model based on an Encoder-Decoder architecture. This achieves automated generation of protocol parsing rules, resulting in an order-of-magnitude improvement in the access efficiency of large-scale, multi-category devices and significantly reducing the technical threshold for protocol configuration. Even non-professionals can quickly complete device access by providing message samples. Furthermore, the protocol rules generated by AI are superior to those written manually in terms of rigor and completeness, reducing data parsing errors caused by rule ambiguity or omissions and enhancing the system's robustness. In addition, the structured protocol definitions decouple the parsing engine from specific protocols, significantly enhancing system scalability and facilitating support for new data types and complex protocol structures. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a machine learning-based protocol parsing rule generation method provided in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of a protocol parsing rule generation system module provided in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Explanation of main component symbols Protocol parsing rule generation system 10 Message Acquisition Module 11 Protocol Generation Model 12 Protocol engine calling module 13 Electronic devices 20 Processor 21 Memory 22 Method steps S100-400 Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0020] It should be noted that, in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0021] It should be noted that in the embodiments of this application, the terms "first," "second," etc., are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor as indicating or implying order. Features specified as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0022] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] With the rapid expansion of IoT device types and application scenarios, the message formats generated between devices and cloud platforms are becoming increasingly diverse and complex. The traditional model of manually defining parsing rules is no longer sufficient to meet the efficiency and accuracy requirements of large-scale, multi-vendor device access. For massive heterogeneous devices, there is an urgent need for a method that can automatically convert raw messages into machine-readable, scalable protocol rules to reduce manual costs, minimize configuration errors, and accelerate device deployment.

[0024] In related technologies, there are already practices that use compact string tags to describe protocol structures. However, this type of method has obvious shortcomings such as ambiguous structure, missing information, and poor scalability. It cannot directly provide downstream businesses with meaningful field names and data types, thus limiting its practicality in complex industrial, power, meteorological, and other scenarios.

[0025] This application provides a machine learning-based protocol parsing rule generation method, system, and electronic device. By training a sequence-to-sequence language model based on an Encoder-Decoder architecture, the system can automatically "translate" raw device messages into well-structured, machine-readable structured protocol definitions (such as JSON format). This achieves automated generation of protocol parsing rules, resulting in an order-of-magnitude improvement in the access efficiency of large-scale, multi-category devices and significantly reducing the technical threshold for protocol configuration. Even non-professionals can quickly complete device access by providing message samples. Furthermore, AI-generated protocol rules are superior to manually written rules in terms of rigor and completeness, reducing data parsing errors caused by rule ambiguity or omissions, and enhancing the system's robustness. In addition, structured protocol definitions decouple the parsing engine from specific protocols, significantly enhancing system scalability and facilitating support for new data types and complex protocol structures.

[0026] Figure 1 This is a flowchart illustrating a machine learning-based protocol parsing rule generation method according to an embodiment of this application. Figure 1 The machine learning-based protocol parsing rule generation method shown includes at least the following steps: S100: Obtain the original communication message from the device to be accessed; S200: Input the original communication message into a pre-trained protocol generation model; S300: Receive the machine-readable standardized protocol description format output by the protocol generation model; S400: Call the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and convert similar communication messages into a data set with explicit field names and data types.

[0027] S100: Obtain the original communication messages of the device to be connected.

[0028] In this embodiment of the application, the protocol parsing rule generation method based on machine learning includes obtaining the original communication messages of the device to be accessed in step S100.

[0029] Specifically, the acquisition steps include collecting real-time uplink packets from the device to be connected via an IoT access gateway or edge device, importing packet samples in batches from device logs or historical databases, or generating representative communication packets via a simulator / test bench. During the acquisition of the original communication packets, metadata associated with the packets is recorded simultaneously. The metadata includes device type identifier, device unique identifier, acquisition timestamp, channel / port information, and original encoding format, etc., for subsequent preprocessing and model context splicing.

[0030] Understandably, the acquisition can be performed in real-time streaming or offline batch processing. It can collect a small number of samples from a single device for quick analysis or collect samples from similar devices in batches to form a training set. In addition, the acquisition process can be configured with filtering rules to remove noise or abnormal messages, or retain abnormal messages for model robustness training.

[0031] S200: Input the original communication message into the pre-trained protocol generation model.

[0032] In this embodiment of the application, the protocol parsing rule generation method based on machine learning includes, in step S300, inputting the original communication message into a pre-trained protocol generation model, wherein the protocol generation model is a sequence-to-sequence language model based on an Encoder-Decoder architecture.

[0033] Specifically, the input steps include: concatenating the acquired raw communication messages and their associated metadata (such as device type identifier, acquisition timestamp, port / channel information, original encoding format, etc.) into the model input according to a pre-defined template; performing lightweight preprocessing on the raw communication messages when necessary (such as unified encoding conversion, noise / control character removal, or minimal placeholder padding) to meet the model input constraints; concatenating the task prompt information and several example pairs into the input header or side notes before model input to limit the model's parsing target and output structure; then, tokenizing the concatenated input and sending it to the trained or fine-tuned Encoder-Decoder sequence via a local or remote inference interface (such as REST / gRPC) for inference in the sequence model; the model inference process can employ decoding strategies such as bundle search, temperature adjustment, or top-k / top-p, and returns the original text of the rule label sequence and the corresponding confidence distribution or probability information.

[0034] Understandably, model invocation can employ either single-line real-time inference or batch / streaming inference to improve throughput; models can be deployed on cloud or edge devices to balance latency and privacy, and input concatenation hint strategies can include static templates, dynamically generated contextual hints based on the device, or a mix of few-shot examples; in high-concurrency scenarios, request queuing, caching of the most recently successful protocol descriptions, and circuit breaking strategies should be combined to ensure service stability.

[0035] S300: A machine-readable standardized protocol description format output by the protocol generation model.

[0036] In this embodiment of the application, the protocol parsing rule generation method based on machine learning includes receiving a standardized protocol description format that is machine-readable and output by the protocol generation model in step S300.

[0037] Specifically, the receiving and processing steps include: parsing the rule label sequence returned by the model and mapping the label items to standardized description fields according to predefined mapping rules; constructing a machine-readable structured protocol definition array (each element is a JSON object describing a single protocol unit) according to a predefined JSONSchema or protocol description template, and filling each field item with attributes such as name, data type, offset / length, byte order, original encoding, and necessary mapping / validation rules; performing syntax validity checks (e.g., JSONParse and Schema validation), semantic consistency checks (e.g., whether field length and offset exceed the limit, whether checksum position matches the algorithm), and confidence threshold judgment on the constructed standardized description; marking entries with low confidence or failed validation as requiring manual review or rollback and re-inference; persisting the validated description in draft form and recording metadata (model version, generation time, confidence statistics), while generating visual previews and difference comparison information for easy manual viewing.

[0038] Understandably, the generated standardized descriptions can be automatically accepted and distributed for operation, or human-computer interaction can be triggered based on confidence thresholds; the platform can be configured to automatically merge historical templates or similar protocols to improve stability, or adopt a rollback strategy (e.g., prioritizing historically verified protocols) when structural conflicts occur; for ease of auditing and backtracking, all model outputs and verification logs should be versioned and stored.

[0039] S400: Calls the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and converts the similar communication messages into a data set with clear field names and data types.

[0040] In this embodiment of the application, the protocol parsing rule generation method based on machine learning includes, in step S400, calling the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse the similar communication messages from the device to be accessed according to the standardized description format, and convert the similar communication messages into a data set with explicit field names and data types.

[0041] Specifically, the deployment and parsing steps include: publishing the standardized protocol description, which has passed verification (and undergone necessary manual confirmation), to the parsing engine after version management; the parsing engine dynamically generates or loads parsing logic based on the standardized description. The parsing logic can use Just-In-Time (JIT) compilation to compile the JSON description into an efficient parsing function, or use interpreted execution to parse each field one by one for quick rollback and debugging; during parsing, the parsing engine extracts bytes / characters of specified offsets or lengths from the arriving message in the order of protocol units, converts the original bytes into the specified data type according to the field encoding and byte order, and performs semantic replacement according to the mapping rules or enumerations defined in the description; the parsing results are output in key-value pair form (field name-field value) or structured object array for consumption by the upper-layer business system, and records the parsing success rate, latency, abnormal samples, and statistical information at runtime for use by operation and maintenance and model optimization.

[0042] Understandably, the parsing engine should support high concurrency, cache hot protocol descriptions, zero-copy or streaming IO optimization, and canary release / rollback capabilities. For messages that fail to be parsed or fail to pass verification, the engine can adopt lenient parsing, default value filling, or direct marking and reporting to manual review according to the strategy. At the same time, abnormal samples are queued for subsequent model retraining or rule optimization, thus forming a closed-loop parsing system that combines automation and manual intervention.

[0043] In this embodiment, the method further includes a model training step: acquiring a dataset for model training, the dataset containing multiple sets of historical device communication message samples and the corresponding protocol parsing rules for each sample. Preprocessing operations are performed on the historical device communication message samples in the dataset to construct structured training samples for model fine-tuning. The preprocessing operations include structural parsing and standardization of the message samples according to preset structural analysis rules. Based on the structured training samples and their corresponding protocol parsing rules, a pre-trained sequence-to-sequence language model is adjusted, thereby enabling the protocol generation model to learn to output a standardized protocol description format based on the original communication messages.

[0044] Specifically, the model training process includes: dividing the collected or labeled dataset into training, validation, and test sets proportionally; performing data augmentation and pseudo-sample synthesis on the training set to increase sample diversity; selecting an appropriate optimizer and learning rate scheduling strategy (e.g., AdamW and learning rate decay) based on the training / validation results, and employing measures such as early stopping, gradient pruning, and regularization to prevent overfitting; saving periodic checkpoints and recording key metrics (including loss, exact match, BLEU, json_valid_rate, and regression metrics on the validation set) during the training process; using parameter-efficient fine-tuning methods (such as LoRA and Adapter) to reduce computational resources during the fine-tuning phase; and performing regression and performance validation on an independent test set after training, and then versioning and publishing the model to the inference service after passing the validation. To ensure long-term stability, this application also stipulates that the model will undergo a periodic retraining process (including data collection, screening, incremental fine-tuning and regression testing) every three months after initial deployment. In addition to periodic retraining, when online monitoring indicators (such as the parsing success rate dropping beyond a preset threshold or a surge in manual review work orders) trigger an alarm, supplementary retraining or emergency fine-tuning will be initiated immediately.

[0045] Understandably, periodic retraining every three months can serve as a routine maintenance plan or be combined with an active learning mechanism: protocol definitions that are manually confirmed or corrected in the visual editor will be prioritized for inclusion in the training pool as high-quality samples for the next retraining; for resource-constrained scenarios, incremental fine-tuning can be performed on only a few layers or parameters of the model to reduce costs; the retrained model is gradually rolled out through canary release and rollback mechanisms to monitor performance under real traffic and revert to a historical stable version when necessary.

[0046] In this embodiment, the preprocessing operation includes: identifying the communication encoding of the message samples and converting the identified message samples with different encoding forms into a string sequence with a unified representation. Based on the preset structural analysis rules, the string sequence is segmented to identify and separate the structural and data parts of the message. The identified structural and data elements are standardized and labeled, and the length or position information of each segment in the original message is marked.

[0047] Specifically, the preprocessing module first performs encoding detection on each message (such as hex, base64, ASCII, UTF-8, vendor-defined encoding, etc.) and converts them into a unified intermediate representation (such as hexadecimal pairs or printable string streams). Then, according to segmentation rules, the string sequence is divided into several protocol units based on position, length indicator, delimiter, or repetition pattern, and the starting offset, byte length, original encoding, and byte order information are recorded for each unit. The detected check segments (CRC, checksum) are specially marked, and placeholders or anomaly markers are inserted for missing or truncated fields. In the preprocessing stage, enhancement operations such as noise injection, field order perturbation, and truncation simulation can also be performed to enrich the training samples, and key fields (such as device ID and timestamp) are extracted and added to the samples as metadata.

[0048] Understandably, preprocessing can employ rule-based methods (regular expressions, state machines, template matching) or combine lightweight statistical or machine learning methods to improve the recognition rate of abnormal or variant messages. To ensure the quality of training samples, preprocessing results can be automatically verified (e.g., length / offset consistency checks), and abnormal samples can be marked as requiring manual review. Preprocessing templates and rules can be maintained and versioned by device family or vendor to ensure uniform updates and consistency of training data before the three-month periodic retraining.

[0049] Predefined task prompt information is appended before the message sample text in each of the structured training samples to form a complete input sequence for the model training phase. The task prompt information is used to instruct the model during training to learn the task of generating protocol rules from device messages and to constrain the format specifications that the model output must follow.

[0050] Specifically, the task prompt information consists of several parts, including a task description section (describing the business domain or key fields to be extracted), an output structure template section (clarifying the expected JSON Schema or tag format for output), an example pair section (demonstrating several input-output examples in a few-shot manner), and an optional device context section (e.g., device type, known field enumeration, or industry protocol identifier). During the training phase, the above prompts and formatted messages are concatenated as model input. The model's goal is to generate standardized protocol descriptions that match the training samples under given prompt constraints. The prompt template itself is included in version management and can be optimized or replaced before each three-month retraining to reflect the latest industry rules and model capabilities. It is understood that prompt engineering is a key means to improve generation accuracy: in actual deployment, a strategy combining static templates and dynamic contexts can be adopted, and the quantity and quality of example pairs can be tuned to balance input length and model capabilities. The update of the prompt template is managed synchronously with the training dataset and preprocessing rules, and the contribution of prompt improvements to model performance is evaluated simultaneously during periodic retraining.

[0051] In this embodiment, the preset structural analysis rules further include: segmentation rules based on character position for identifying protocol headers or end fields at fixed positions in a string sequence; segmentation rules based on byte length for identifying data fields with fixed or variable lengths in a string sequence; segmentation rules based on delimiter features for identifying delimiters in a string sequence and dividing the string sequence into multiple fields; and segmentation rules based on repetition patterns for identifying groups of fields with repetitive structures in a string sequence.

[0052] Specifically, for rules based on character position, the system directly extracts the corresponding interval from the string sequence according to a predefined offset table and labels its semantics (such as "message type"); for rules based on byte length, the system first parses the value of the length indicator field to determine the boundary of the subsequent data segment and extracts it accordingly; for rules based on delimiters, the system identifies continuous or variable-length delimiters and uses them as boundaries to generate a field list; for repetitive patterns, the system detects repetitive units through sliding window or regular expression pattern matching and abstracts them into array elements, recording the array length or loop termination flag when necessary to assist the parser in generating loop parsing logic.

[0053] Understandably, in actual messages, some fields may satisfy multiple segmentation rules at the same time. In this case, the final segmentation result can be determined by priority rules (e.g., position takes precedence over delimiters) or by combining multiple rule verifications (e.g., verifying length consistency at the same time). For segments that cannot be clearly parsed, the system can mark them as "suspicious fields" and include them in manual confirmation or subsequent model learning samples.

[0054] In this embodiment, bytes or characters of corresponding length are extracted sequentially from similar communication messages according to the protocol unit order defined in the standardized description format. Based on the type and attributes of each protocol unit, the extracted bytes or characters are parsed and converted into the corresponding data type. The parsed data is then associated with the field names defined in the protocol unit to form a data set with clearly defined field names and data types.

[0055] Specifically, the parsing process iteratively executes the protocol units in the standardized description array in order: for each protocol unit, the range to be extracted is first calculated based on the offset or length field in its description, and the byte sequence is converted into a number or string according to the original encoding and byte order of the message; then, the original value is converted and normalized according to the DataType, scaling factor or unit information of the field (for example, dividing the original integer value by 100 to get the degree Celsius); if the field definition contains enumeration or mapping rules, it is replaced with business semantic terms according to the mapping table; finally, the field name is paired with the converted value and added to the output data set until all protocol units are processed.

[0056] Understandably, exceptions should be handled during the parsing process. For example, when the message length is insufficient to meet a certain protocol unit, a default value can be used, an exception can be thrown, or the message can be marked as parsing failure and recorded for subsequent analysis. For variable length or nested structures, the parser can recursively call the corresponding parsing subprocess or use a loop parsing strategy to correctly derive array-type fields.

[0057] In this embodiment of the application, the standardized description format is a JSON-based structured protocol definition array. Each element in the structured protocol definition array is described as a JSON object of a corresponding protocol unit. The type of each protocol unit includes at least one of the following: header, data field, delimiter, terminator, and checksum.

[0058] Specifically, the JSON-based structured protocol definition array follows a predefined JSON Schema. Each protocol unit's JSON object contains basic attribute fields, such as "name" (field name), "type" (protocol unit type), "offset" or "position" (starting offset), "length" (fixed length or length field reference), "endian" (byte order), "encoding" (raw encoding), "mapping" (mapping rules / enumeration), "validate" (validation rules, such as CRC / range check), and "description" (description). This array is arranged according to the order of the protocol units in the message so that the parsing engine can read and execute the corresponding parsing logic in sequence.

[0059] Understandably, this JSON-based standardized description format has good scalability and readability, supports nested structures (represented by children or subfields fields), conditional fields (specifying that it takes effect under a certain condition by condition), version identifiers, and comment fields, and can be directly imported into a visual editor for manual review or exported as a configuration file that can be recognized by the target parser.

[0060] In this embodiment of the application, when the protocol unit is a data field, the JSON object corresponding to the protocol unit includes at least: name, data type, original message encoding, length, and byte order.

[0061] Specifically, the JSON object example corresponding to the data field contains at least the following key-value pairs: "name": a string used to identify the semantics of the field; "dataType": an identifier of the data type (e.g., uint8, int16, float32, string, timestamp, etc.); "encoding": a description of the encoding format of the field in the original message (e.g., hex, ascii, utf-8); "length": a fixed byte length or a reference expression for the length field (e.g., "lengthField":"len1"); "endian": a byte order indicator (e.g., "big" or "little"); additionally, it may optionally include attributes such as "scale" (scaling factor), "offset" (offset), "unit" (unit), and "enum" (enumeration mapping table) to fully describe the conversion process from raw bytes to business meaning.

[0062] Understandably, for complex fields, conversion scripts can be embedded in the JSON object or external conversion modules can be referenced (e.g., specifying a conversion function name or regular expression parsing rule). The "sensitive":true" flag can be used to indicate at runtime that the field should be desensitized or encrypted for storage. When the field is a variable-length string or array, the "length" field can be a dynamic expression or the "terminator" can be used to specify the end marker.

[0063] The specific working process of the method provided in this application is described below using five exemplary embodiments.

[0064] Example 1 (Industrial Water Tank Level Gauge) This embodiment involves a tank monitoring scenario in a chemical plant. It requires real-time reporting of multiple parameters such as liquid level, temperature, and pressure. The message format is an embedded JSON string containing multiple device IDs and various sensor readings. The message structure is complex, with mixed fields, making manual configuration of each field time-consuming and error-prone. An example message is: {msgId:"2091094701305001,209109470130F001",datalist:[{"jl":1.651,"ww":2.999,"wd":10.1,...}]}.

[0065] In this embodiment, the operations and maintenance personnel paste or upload the original message to the access platform. The message acquisition module first identifies and retains fixed text fragments (such as {msgId:") and delimiters for nested JSON, while marking dynamic data segments as variable fields and recording their original encoding and length information. Subsequently, a task prompt explains the expected business field categories to be extracted, such as liquid level (jl) and temperature (wd). After receiving the formatted text and prompt, the sequence-to-sequence model infers and generates a rule-labeled sequence.

[0066] The model output is converted into a JSON-based structured protocol description. Each protocol unit contains attributes such as type, name, dataType, encoding, and length. The platform automatically marks temporary field names in the form of field_n for manual renaming to business names such as "liquid level" and "temperature" with a single click. Finally, the parsing engine can directly perform structured parsing on subsequent messages according to this protocol. This implementation significantly improves access efficiency and reduces manual configuration workload to near zero, lowering the human error rate.

[0067] Example 2 (Integrated Traffic Weather Station for Highways) This embodiment involves comma-separated (CSV) style messages reported by meteorological stations along the route. These messages contain dozens of parameters, including version number, device ID, timestamp, wind speed, wind direction, and visibility. There are numerous fields, and some fields are combinations of "label + value," such as ...,TEMPA,14.9,0,... A sample message is $DATADICK,V202501,00001,...,TEMPA,14.9,0,....

[0068] The platform message acquisition module first identifies and normalizes common delimiters (such as commas) and labeled field fragments, and identifies triples such as LABEL, VALUE, and FLAG as fixed templates based on repetition patterns. Then, the formatted text and example prompts are fed into the protocol generation model. The model uses the "label + value" pattern learned during training to generate a complete JSON protocol definition containing dozens of fields in one go.

[0069] The generated protocol can be directly called by the parsing engine. The output structured data set contains explicit field names (or placeholder names for renaming) and data types (string / decimal), thereby avoiding manual counting and field misalignment problems, and greatly improving the efficiency of on-site debugging and batch access.

[0070] Example 3 (Aging Rack Data Acquisition Board) This embodiment targets an aging test fixture for an electronic production line. To save transmission costs, this type of equipment encodes a large number of sampling points into lengthy hexadecimal sequences, with each data point being a 2-byte integer. The message ends with a CRC16 checksum, making manual analysis and configuration extremely complex. An example message is a long hexadecimal string: AA02E4001BFFFF0001...

[0071] In this embodiment, the message acquisition module identifies "several consecutive 2-byte integer sequences" and marks the array boundaries by repeating patterns and byte length rules. Simultaneously, it detects common checksum patterns at the end of the message and marks them as Checksum units. The sequence-to-sequence model, based on patterns learned from numerous similar samples, can map repeating data segments to array-type Data nodes and automatically generate field definitions of {"dataType":"integer","encoding":"hex","length":2,"byteOrder":"AB"}, and at the end, generates {"type":"Checksum","algorithm":"crc16","length":2}.

[0072] This implementation automates the process that previously required manual configuration of each field, saving time and reducing parsing failures due to manual errors. It is particularly valuable when a large number of similar devices are connected in batches.

[0073] Example 4 (Intelligent Power Analysis Instrument) This embodiment relates to an analytical instrument for high-precision monitoring of power grids. The message consists of several 4-byte hexadecimal blocks, corresponding to floating-point indicators such as multiphase voltage, current, and power factor. The byte order (big-endian / little-endian) is crucial to the accuracy of the results. An example message is 04461EB8C34456F0...

[0074] The platform's message acquisition module first segments the message into 4-byte aligned segments. Based on the distribution characteristics of floating-point numbers and common encoding rules in the training data, it prompts the model to preferentially identify consecutive 4-byte segments as float type. The JSON protocol output by the model will indicate attributes such as {"dataType":"float","encoding":"hex","length":4,"byteOrder":"ABCD"} in each Data item, and may also include scaling factors and unit information to ensure the correct physical meaning of the parsed values.

[0075] The value of this embodiment lies in automatically identifying data type and byte order, avoiding numerical distortion caused by incorrect byte order determination, and ensuring the data reliability and availability of the high-precision power monitoring system.

[0076] Example 5 (Multifunctional Control Instrument for Livestock Farms) This embodiment addresses a type of complex hexadecimal message with a mixed structure, containing both 1-byte and 2-byte integers, along with control flags and end markers. Manual analysis and mapping of this message in a short time is difficult. An example message is 55AA000101000100000000005B02...0D0A.

[0077] In this embodiment, the sequence-to-sequence model, through learning from a large number of heterogeneous messages, can capture potential boundary and length variation patterns, generating a mixed-specification JSON protocol sequence containing multiple length descriptions (such as length:1 and length:2) and necessary type annotations. The preprocessing stage may first perform multi-strategy segmentation (position, length, delimiter, and repetition pattern), and the model, with the assistance of prompts, outputs the most likely protocol structure; subsequently, the platform submits the generated results to a human-based visual editor for quick correction of field meanings and naming.

[0078] This embodiment demonstrates the unique advantages of AI in processing messages with "no significant patterns": through pattern learning and prompt synthesis, it generates a usable protocol draft in a short time, greatly shortening the analysis cycle of human experts, and continuously improving the accuracy of automatic generation through closed-loop learning.

[0079] In this embodiment of the application, the protocol generation model uses google / flan-t5-small as the base model. This choice is based on its specific architectural advantages and task adaptability.

[0080] Specifically, the core reasons for choosing flan-t5-small are as follows: First, its Encoder-Decoder architecture is naturally suited for sequence-to-sequence translation tasks. The encoder is responsible for deeply understanding the complex structure of the input message, while the decoder focuses on generating structured protocol descriptions, resulting in a clear division of labor and high efficiency. Second, the model has undergone extensive instruction tuning, giving it a strong ability to follow natural language instructions. This application transforms the specialized protocol generation task into an instruction-response pattern that the model has already mastered by designing fixed prompts (such as "Translate the following device report into a structured JSON protocol label:"), greatly reducing the difficulty for the model to learn new tasks. Third, its small version has approximately 80 million parameters, ensuring performance while requiring less computing resources, facilitating rapid experimental iteration and low-cost deployment.

[0081] It is understandable that using the flan-t5 series model as a starting point means that this solution can inherit its powerful language understanding and generation capabilities. Those skilled in the art can also, as needed and after verifying the feasibility of the process, replace it with a version with a larger number of parameters (such as base, large) to pursue higher accuracy, or replace it with other open-source models with the same architecture (such as BART, mT5). The scope of protection of this application is not limited to this specific model instance.

[0082] In this embodiment, the preparation of training data is accomplished through an automated data transformation script, which is a key step in ensuring the quality and efficiency of model training.

[0083] Specifically, the script (e.g., named convert_dataset.py) performs the following core logic: reads historical data stored in CSV, database, or text files, each containing an original message and a compact string protocol label in an old format; the script parses the syntax of the old label (e.g., recognizing [ ] boundaries, : and | separators), and converts each old instruction label (e.g., [DE[2]|AB]) into a corresponding structured JSON object (e.g., {"type":"Data","name":"field_2","dataType":"integer","encoding":"hex","length":2,"byteOrder":"AB"}) according to a predefined mapping rule table); for all Data type objects generated by the conversion, the script automatically assigns them a unique and sequential generic name (e.g., field_1, field_2, etc.); finally, the script outputs the converted "message-structured JSON protocol" paired data as a standard JSON Lines format file (e.g., train.jsonl), which is the standard input format for the training framework.

[0084] Understandably, this automated script is not only used for building the initial training set, but also as a protocol format upgrade tool to migrate a large number of old protocol rule bases stored in the legacy system to the new structured JSON format in batches without loss, thus achieving smooth technological evolution and protecting the value of historical assets.

[0085] In this embodiment of the application, the fine-tuning training of the model is executed through a well-packaged training script (e.g., train.py), which integrates the entire process from data loading to model saving.

[0086] Specifically, the workflow of the training script includes: automatically loading the train.jsonl and validation.jsonl datasets; for each data point, concatenating the preset prompt word prefix with the original message string to form a complete input sequence, and then using T5Tokenizer to perform tokenization and encoding on the input sequence and the target JSON sequence; configuring training parameters, such as using the AdamW optimizer, setting the learning rate to 5e-5, the batch size to 16, and training for 10 epochs; during training, the script will periodically calculate and output multiple evaluation metrics on the validation set, including the perfect match rate, BLEU score, and JSON format effective ratio, to comprehensively monitor model performance; after training, the script will automatically save the best-performing model weights on the validation set and related configuration files (such as tokenizer.json, config.json) to a specified directory (such as . / model_export).

[0087] Understandably, the design of this training script improves the reproducibility and engineering sophistication of the experiments. By modifying parameters or model paths within the script, researchers can easily experiment with different base models, hyperparameters, or training techniques. The trained model package (model_export) is a self-contained unit that can be directly loaded and used by subsequent inference services.

[0088] In this embodiment of the application, the visualization protocol builder uses a defined front-end technology stack to implement its interactive functions.

[0089] Specifically, the builder recommends using React as the front-end framework, leveraging its componentization and state management capabilities (such as useState and useEffect Hooks) to maintain synchronization between protocol JSON data and the UI. The core of the drag-and-drop interaction is implemented by the dnd-kit library, which provides high-performance, accessible drag-and-drop primitives for handling drag-and-drop deployment from the component toolkit to the protocol canvas, as well as rearranging the order of "pills" within the canvas. For UI components, mature component libraries such as Ant Design or Material-UI can be used to quickly build forms, buttons, and icons in the property editor, ensuring the aesthetics and consistency of the interface.

[0090] Understandably, this combination of technologies balances development efficiency, user experience, and maintainability. The modern architecture of the dnd-kit library can smoothly handle complex drag-and-drop scenarios, while React's state-driven rendering ensures that any modification to the underlying protocol's JSON array (whether AI-generated, user-drag-and-drop, or form-edited) is reflected instantly and accurately in the visual interface, achieving true two-way data binding and a "what you see is what you get" editing experience.

[0091] In this embodiment, the protocol parsing engine employs two key performance optimization techniques to meet the real-time parsing requirements of massive amounts of device data.

[0092] Specifically, the first optimization is just-in-time compilation: For frequently invoked "hot" protocols (whose JSON descriptions are fixed), the parsing engine can dynamically compile the parsing logic defined in the JSON description into a small piece of efficient native machine code (e.g., implemented through CPython's C extension, Numba, or Cython). When a message matching this protocol is received again, the engine directly calls this compiled function for parsing, thus completely avoiding the overhead of repeatedly interpreting and executing JSON instructions. The second optimization is zero-copy or view parsing: When extracting fields from the original message byte stream, the engine uses memory views (such as Python's memoryview) or byte slices to reference the original data whenever possible, instead of copying the data to a new memory space. This significantly reduces the overhead of memory allocation and copying, and reduces the pressure on garbage collection.

[0093] Understandably, these optimization techniques enable the parsing engine to achieve performance levels approaching or even reaching those of hand-written, hard-coded parsers, while retaining the extreme flexibility of JSON-based descriptions. They are particularly suitable for high-throughput, low-latency core data processing scenarios in the Internet of Things (IoT), ensuring that the system can handle concurrent reporting from millions or even tens of millions of devices even with limited resources.

[0094] Figure 2 This is a schematic diagram of a protocol parsing rule generation system provided in an embodiment of this application. For example... Figure 2 The protocol parsing rule generation system 10 shown includes at least the following parts: a message acquisition module 11, a protocol generation model 12, and a protocol engine calling module 13.

[0095] In this embodiment of the application, the message acquisition module 11 is used to acquire the original communication messages of the device to be accessed.

[0096] It is understood that the message acquisition module 11 can be deployed on the edge side or cloud of the IoT access platform to collect communication data of the device to be accessed through various methods. The acquisition methods include at least real-time monitoring of device uplink data through the IoT access gateway, importing debugging messages from the device debugging interface or serial port tool, batch loading stored communication messages from historical logs or databases, or generating representative message samples through simulators or test benches. In addition, the message acquisition module 11 can also collect and maintain context information associated with the communication message while acquiring the original communication message. The context information includes, but is not limited to, device type, device identifier, communication link parameters, port number, timestamp, and original encoding format, so as to provide necessary input basis and auxiliary information support for subsequent protocol generation model calls and parsing engine execution.

[0097] In this embodiment of the application, the protocol generation model 12 is used to receive the original communication message. The protocol generation model is a sequence-to-sequence language model based on the Encoder-Decoder architecture.

[0098] Understandably, the protocol generation model 12 is the intelligent core of this system. It is a pre-trained language model (e.g., Google / Flan-T5-Small) fine-tuned with large-scale domain data. Its Encoder part is responsible for deeply understanding the input raw communication messages, capturing their complex structural features and contextual relationships; the Decoder part, based on the semantic representation output by the Encoder, autoregressively generates a sequence of labels describing the protocol rules. By learning from massive "message-protocol" sample pairs, this model has mastered the general ability to map any device message to an abstract protocol description. Before inference, the model is input along with text concatenated with task instruction prompts to ensure that its output strictly adheres to preset format requirements (such as specific label syntax or JSON framework), thereby achieving the "translation" from raw communication messages to structured rule descriptions.

[0099] In this embodiment of the application, the protocol engine calling module 13 is used to receive the machine-readable standardized protocol description format output by the protocol generation model 12; and to call the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse the similar communication messages from the device to be accessed according to the standardized description format, and to convert the similar communication messages into a data set with clear field names and data types.

[0100] Understandably, the protocol engine calling module 13 serves as a bridge connecting AI-generated rules with production-level parsing and execution. This module first parses and post-processes the rule tag sequence output by the model, converting it into a rigorous, unambiguous, machine-readable format, such as an array of protocol definitions conforming to a specific JSON Schema. This standardized description precisely defines the name, type, length, offset, byte order, and validation rules for each field. Subsequently, the module publishes or configures this formatted protocol description to a high-performance protocol parsing engine. As the execution unit of the data plane, the parsing engine can dynamically generate or load optimal parsing logic based on this description, performing real-time and accurate parsing of massive concurrent reports from similar devices, directly outputting clean, structured business data for storage, analysis, and display on the upper-layer platform, thus forming a complete automated closed loop from "unknown messages" to "usable data."

[0101] Figure 3 This is an electronic device 20 provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device 20 includes at least the following components: a processor 21 and a memory 22.

[0102] In this embodiment, the memory 22 is used to store executable instructions of the processor 21, which, when configured to execute instructions, implement... Figure 1 The method for generating protocol parsing rules based on machine learning is shown.

[0103] In one embodiment of this application, the program operating in the electronic device 20 may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). The information processed by these devices is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (Flash ROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0104] It should be noted that a portion of the electronic device 20 described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0105] It should be noted that the term "computer" as used here refers to a computer built into electronic device 20, employing hardware including an operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard drives built into the computer.

[0106] Furthermore, a "computer-readable recording medium" can include: a medium that dynamically stores a program for a short period of time, such as a communication line used when transmitting a program via a network such as the Internet or a communication line such as a telephone line; or a medium that stores a program for a fixed period of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining with programs already recorded in the computer.

[0107] Furthermore, the electronic device 20 in the above embodiments can also be implemented as an assembly (device group) composed of multiple devices. Each device constituting the device group can possess some or all of the functions or functional blocks of the electronic device 20 in the above embodiments. As a device group, it is sufficient to have all the functions or functional blocks of the electronic device 20.

[0108] It is understood that the machine learning-based protocol parsing rule generation method, system, and electronic device provided in this application, by training a sequence-to-sequence language model based on an Encoder-Decoder architecture, can automatically "translate" raw device messages into well-structured, machine-readable structured protocol definitions (such as JSON format). This achieves automated generation of protocol parsing rules, resulting in an order-of-magnitude improvement in the access efficiency of large-scale, multi-category devices and significantly reducing the technical threshold for protocol configuration. This allows non-professionals to quickly complete device access by providing message samples. At the same time, protocol rules generated by AI are superior to those written manually in terms of rigor and completeness, reducing data parsing errors caused by rule ambiguity or omissions and enhancing the robustness of the system. Furthermore, structured protocol definitions decouple the parsing engine from specific protocols, significantly enhancing system scalability and facilitating support for new data types and complex protocol structures.

[0109] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A protocol parsing rule generation method based on machine learning, applied to an IoT device access platform, characterized in that, The method includes: Obtain the original communication messages of the device to be connected; The original communication message is input into a pre-trained protocol generation model, which is a sequence-to-sequence language model based on an Encoder-Decoder architecture. Receive the standardized, machine-readable protocol description format output by the protocol generation model; The protocol parsing engine is invoked according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and convert the similar communication messages into a data set with explicit field names and data types.

2. The protocol parsing rule generation method based on machine learning according to claim 1, characterized in that, The method also includes a model training step: Obtain a dataset for model training, which contains multiple sets of historical device communication message samples and the protocol parsing rules corresponding to each sample; Preprocessing operations are performed on historical device communication message samples in the dataset to construct structured training samples for model fine-tuning; wherein, the preprocessing operations include structural parsing and standardization of the message samples according to preset structural analysis rules; Based on the structured training samples and their corresponding protocol parsing rules, a pre-trained sequence-to-sequence language model is adjusted so that the protocol generation model learns to output the standardized protocol description format based on the original communication message.

3. The protocol parsing rule generation method based on machine learning according to claim 2, characterized in that, The preprocessing operation further includes: The message samples are identified by communication encoding, and the identified message samples with different encoding forms are converted into string sequences with a unified representation. Based on the preset structural analysis rules, the string sequence is segmented to identify and separate the structural and data parts of the message; The identified structural and data elements are standardized and labeled, and the length or position information of each segment in the original message is marked.

4. The protocol parsing rule generation method based on machine learning according to claim 2 or 3, characterized in that, Based on the structured training samples and their corresponding protocol parsing rules, a pre-trained sequence-to-sequence language model is adjusted, including: Predefined task prompt information is concatenated before the message sample text in each of the structured training samples to form a complete input sequence for the model training phase. The task prompt information is used to instruct the model to learn the task of generating device messages into protocol rules during training, and to constrain the format specifications that the model output must follow.

5. The protocol parsing rule generation method based on machine learning according to claim 4, characterized in that, The preset structural analysis rules include: The segmentation rule based on character position is used to identify protocol headers or end fields at fixed positions in the string sequence; The byte-length-based segmentation rules are used to identify data fields with fixed or variable lengths in the string sequence; The segmentation rule based on delimiter features is used to identify delimiters in the string sequence and segment the string sequence into multiple fields; The segmentation rule based on repetition pattern is used to identify groups of fields with repetitive structures in the string sequence.

6. The protocol parsing rule generation method based on machine learning according to claim 1, characterized in that, According to the protocol unit order defined in the standardized description format, bytes or characters of corresponding length are extracted sequentially from the same type of communication messages; Based on the type and attributes of each protocol unit, the extracted bytes or characters are parsed and converted into the corresponding data type; The parsed data is associated with the field names defined in the protocol unit to form a data set with explicit field names and data types.

7. The protocol parsing rule generation method based on machine learning according to claim 1, characterized in that, The standardized description format is a JSON-based structured protocol definition array. Each element in the structured protocol definition array is described as a JSON object of a corresponding protocol unit. The type of each protocol unit includes at least one of a header, data field, delimiter, terminator, and checksum.

8. The protocol parsing rule generation method based on machine learning according to claim 7, characterized in that, When the protocol unit is a data field, the JSON object corresponding to the protocol unit includes at least: name, data type, original message encoding, length, and byte order.

9. A protocol parsing rule generation system, applied to implement the protocol parsing rule generation method as described in any one of claims 1 to 8, characterized in that, The system includes: The message acquisition module is used to acquire the original communication messages of the device to be connected. A protocol generation model is used to receive the original communication message. The protocol generation model is a sequence-to-sequence language model based on the Encoder-Decoder architecture. The protocol engine calling module is used to receive the machine-readable standardized protocol description format output by the protocol generation model; and to call the protocol parsing engine according to the standardized description format. The protocol parsing engine is used to parse subsequent similar communication messages from the device to be accessed according to the standardized description format, and to convert the similar communication messages into a data set with explicit field names and data types.

10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the machine learning-based protocol parsing rule generation method according to any one of claims 1 to 8 when executing the instructions.

Images