A data multi-protocol adaptive parsing method, system and storage medium

Abstract

The present invention provides a data multi-protocol adaptive parsing method, system, and storage medium, relating to the field of communications technology. The method comprises receiving a data packet sent by an Internet of Things (IoT) device; stripping the data packet header to obtain a structured data segment of the data packet; extracting features based on the structured data segment to obtain grammatical features of the structured data segment; constructing an abstract syntax tree for the structured data segment based on the grammatical features using a protocol-independent data expression model; serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream for the structured data segment; and integrating all standard data streams to obtain standard data information for the IoT device. The present invention utilizes a protocol-independent data expression model and an abstract syntax tree to improve the data communication management efficiency of an IoT platform.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a data multi-protocol adaptive parsing method, system and storage medium. Background Art

[0002] With the rapid development of the Internet of Things (IoT), an increasing number of IoT devices are connected through it, such as smart homes, smart transportation, and industrial automation. IoT devices typically transmit data based on the communication protocol of the connected platform. However, different platforms often use different communication protocols. Common data parsing methods rely on parsing the protocol header of a data packet to identify the communication protocol used. Once the protocol type is identified, the packet is parsed and processed according to the protocol specifications.

[0003] In related technologies, with the continuous increase in the types of IoT devices, new communication protocols are constantly increasing, making the protocol identification process more and more complicated. In order to achieve interoperability between IoT devices with different protocols, it is necessary to continuously update and maintain the protocol identification model or protocol mapping table to ensure that data packets of various protocol types can be accurately identified. The high frequency of updates and maintenance greatly increases the application of human resources and computing resources. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the data communication management efficiency of the Internet of Things platform.

[0005] To solve the above problems, the present invention provides a data multi-protocol adaptive parsing method, system and storage medium.

[0006] In a first aspect, the present invention provides a data multi-protocol adaptive parsing method, which is applied to an Internet of Things platform, wherein the Internet of Things platform is communicatively connected with an Internet of Things device, and the method comprises:

[0007] Receive a data packet sent by the IoT device;

[0008] Performing header stripping processing on the data packet to obtain a structured data segment of the data packet;

[0009] Performing feature extraction based on the structured data segment to obtain grammatical features of the structured data segment;

[0010] constructing an abstract syntax tree of the structured data segment according to the grammatical features through a protocol-independent data expression model;

[0011] Serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment;

[0012] All the standard data streams are integrated to obtain standard data information of the IoT device.

[0013] Optionally, the performing header stripping on the data packet to obtain the structured data segment of the data packet includes:

[0014] Locating the starting position and length of the header of the data packet based on the protocol header features of the data packet;

[0015] Separating header information from the data packet according to the header starting position and the length, removing the header information from the data packet, and extracting data information following the header information from the data packet as the structured data segment of the data packet;

[0016] Wherein, locating the starting position and length of the header of the data packet includes:

[0017] When the data packet conforms to the preset standard protocol format, the header starting position and the length are directly determined according to the preset standard protocol header characteristic parameters;

[0018] When the data packet does not conform to the preset standard protocol format, feature matching is performed on the data packet through a protocol-independent data feature analysis model to determine the header starting position and the length of the data packet.

[0019] Optionally, the extracting features based on the structured data segment to obtain the grammatical features of the structured data segment includes:

[0020] Performing lexical analysis on the structured data segment to identify vocabulary units in the structured data segment, and performing grammatical analysis on the vocabulary units to determine grammatical relationships between the vocabulary units;

[0021] Extracting grammatical features of the structured data segment according to the grammatical relationship, wherein the grammatical features include the grammatical structure, data type and grammatical rules of the data segment;

[0022] Performing pattern matching on the grammatical features of the extracted structured data segment, identifying a specific grammatical pattern in the structured data segment, and using the specific grammatical pattern as part of the grammatical features;

[0023] performing vectorization processing on the grammatical features of the extracted structured data segment, and converting the grammatical features into vector grammatical feature representations;

[0024] Performing feature fusion on the vector grammatical feature representation to generate a grammatical feature vector of the structured data segment;

[0025] Performing feature dimensionality reduction processing on the grammatical feature vector, and then performing feature enhancement processing on the grammatical feature vector after dimensionality reduction to obtain a final grammatical feature representation;

[0026] The grammatical feature of the structured data segment is obtained according to the specific grammatical pattern and the grammatical feature representation.

[0027] Optionally, constructing an abstract syntax tree of the structured data segment according to the grammatical features using a protocol-independent data expression model includes:

[0028] Initializing the protocol-independent data expression model to obtain a preset syntax tree generation rule and data structure;

[0029] Taking the grammatical features of the structured data segment as input, determining the root node of the abstract syntax tree according to the type and content of the grammatical features through the protocol-independent data expression model;

[0030] Determining, based on the grammatical structure and grammatical rules in the grammatical features, a rule for creating branch nodes and a rule for creating leaf nodes of the abstract syntax tree;

[0031] Adjusting the node structure of the abstract syntax tree according to the specific syntax pattern;

[0032] By using the generation algorithm of the protocol-independent data expression model, based on the grammatical feature vector and the grammatical feature representation, sequentially constructing the hierarchical structure of the abstract syntax tree;

[0033] The abstract syntax tree of the structured data segment constructed is outputted through the protocol-independent data expression model.

[0034] Optionally, the serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment includes:

[0035] Obtaining a preset standard data template, wherein the preset standard data template includes the format, structure, and semantic rules of the data stream;

[0036] Matching the root node of the abstract syntax tree with the root element of the standard data template according to the format, structure and semantic rules of the data stream;

[0037] Converting the branch nodes of the abstract syntax tree and the leaf nodes of the abstract syntax tree into an intermediate data structure that conforms to the standard data template;

[0038] According to the specific grammatical pattern and the grammatical rules, the intermediate data structure is sequentially semantically mapped and format converted to obtain an intermediate data structure after format conversion, and the intermediate data structure after format conversion is verified to determine whether it meets the constraints of the standard data template;

[0039] If so, the verified intermediate data structure is serialized to generate a linear standardized data stream, and the linear standardized data stream is used as the standardized data stream of the structured data segment.

[0040] Optionally, the performing semantic mapping and format conversion on the intermediate data structure in sequence according to the specific grammatical pattern and the grammatical rules to obtain the intermediate data structure after format conversion, and verifying the intermediate data structure after format conversion to determine whether it meets the constraints of the standard data template includes:

[0041] Performing semantic analysis on the intermediate data structure to identify data elements and data relationships in the intermediate data structure;

[0042] Mapping the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships;

[0043] Adjusting the format of the intermediate data structure according to the standardized semantic representation, including data type conversion and data encoding specifications;

[0044] The converted intermediate data structure is format-verified to determine whether it complies with the constraints of the standard data template.

[0045] Optionally, mapping the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships includes:

[0046] Matching the intermediate data structure with a preset standard semantic model to determine the correspondence between the data elements and the data relationships;

[0047] According to the specific grammatical pattern and the grammatical rules, the data elements and the data relations are converted into standardized semantic elements and semantic relations in the preset standard semantic model;

[0048] Preprocessing the converted standardized semantic elements and semantic relations to obtain the standardized semantic elements and semantic relations that meet the requirements of the preset standard semantic model;

[0049] The standardized semantic elements and the semantic relationships that meet the requirements are integrated to form standardized semantic representations of the data elements and the data relationships.

[0050] Optionally, the method further includes: constructing the protocol-independent data expression model;

[0051] The protocol-independent data expression model includes:

[0052] Initialize the model framework of the protocol-independent data expression model and set the basic structure and parameters of the protocol-independent data expression model;

[0053] Acquire a standard communication protocol as a data sample, wherein the data sample is used for model training;

[0054] Preprocessing the data samples, extracting and marking features of the preprocessed data samples to obtain the data samples with feature markings, inputting the data samples with feature markings into the model framework for training, and generating an initial protocol-independent data expression model;

[0055] The initial protocol-independent data expression model is verified and optimized to obtain the final protocol-independent data expression model.

[0056] In a second aspect, the present invention provides a data multi-protocol adaptive parsing system, which is applied to an Internet of Things platform, wherein the Internet of Things platform is communicatively connected with an Internet of Things device, and the system includes:

[0057] A data receiving unit, configured to receive data packets sent by the IoT device;

[0058] A data processing unit, configured to perform header stripping processing on the data packet to obtain a structured data segment of the data packet;

[0059] a feature extraction unit, configured to extract features from the structured data segment to obtain grammatical features of the structured data segment;

[0060] an abstract syntax tree construction unit, configured to construct an abstract syntax tree of the structured data segment according to the syntax features through a protocol-independent data expression model;

[0061] A standardization unit, configured to serialize the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment;

[0062] The integration unit is used to integrate all the standard data streams to obtain standard data information of the Internet of Things devices.

[0063] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned data multi-protocol adaptive parsing method is implemented.

[0064] The data multi-protocol adaptive parsing method, system and storage medium of the present invention avoid the high maintenance costs caused by frequent updates of protocol identification models or protocol mapping tables in the prior art by adopting a protocol-independent data expression model and an abstract syntax tree construction method. In traditional methods, with the continuous update of IoT devices and communication protocols, a large amount of manpower and material resources need to be continuously invested to maintain the relevant content of protocol identification. The method of the present invention can adapt to new communication protocols, significantly reducing the maintenance workload and resource consumption of the platform, thereby reducing the operating and maintenance costs of the IoT platform and improving the data communication management efficiency of the IoT platform. Specifically, the present invention can parse IoT data more efficiently by stripping the header of the data packet, extracting features for the structured data segment and constructing an abstract syntax tree. Compared with the prior art that relies on complex protocol header parsing, the present invention can quickly extract the key grammatical features of the data and convert it into the form of an abstract syntax tree, speeding up the data processing process and improving the efficiency of data processing. In addition, the abstract syntax tree is serialized according to a preset standard data template to generate a standardized data stream, further optimizing the organization and processing of the data, enabling the data to be integrated and utilized more quickly, and improving the data processing performance of the entire IoT platform. The method of the present invention conducts in-depth analysis and understanding of data by constructing an abstract syntax tree, which can more accurately parse data packets sent by IoT devices. Compared with existing technologies that rely solely on protocol header identification, this method can better grasp the essential characteristics of the data, reduce data processing errors caused by protocol incompatibility or complex protocol identification errors, and improve the accuracy of data processing. At the same time, by converting data of different protocols into standardized data sets, it provides a high-quality, unified format data foundation for subsequent data analysis, application development, etc., helps to mine more valuable information, and further improves the data processing quality and application value of the IoT platform. The protocol-independent data expression model and abstract syntax tree construction method of the present invention can adapt to various different communication protocols without the need for large-scale adjustments and updates to the existing system. This makes it easier for the IoT platform to access new IoT devices and protocols, greatly improving the scalability of the system. In addition, by converting data into standardized data sets, it provides strong support for data integration and interoperability of different devices and different protocols, improves the data communication management efficiency of the IoT platform, and enables it to better adapt to the increasingly complex and diverse IoT application scenario requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1Flowchart of a data multi-protocol adaptive parsing method according to an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of the structure of a data multi-protocol adaptive parsing system in another embodiment of the present invention. DETAILED DESCRIPTION

[0067] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0068] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0069] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0070] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0071] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0072] In response to the problems existing in the above-mentioned related technologies, this embodiment provides a data multi-protocol adaptive parsing method, system and storage medium.

[0073] Combine Figure 1As shown, a data multi-protocol adaptive parsing method of the present invention is applied to an Internet of Things platform, and the Internet of Things platform is communicatively connected with an Internet of Things device. The method includes:

[0074] Receive a data packet sent by the IoT device.

[0075] Specifically, the IoT platform establishes a connection with IoT devices through communication interfaces (such as Wi-Fi, Bluetooth, and 4G / 5G networks). When IoT devices generate data, they encapsulate it into packets and send it out according to a specific data transmission protocol (such as CoAP or MQTT). The IoT platform's receiving module is constantly listening, and when it detects an incoming data packet, it begins receiving it. For example, for MQTT-based communication, the IoT platform subscribes to topics published by IoT devices. When a device publishes a data packet to that topic, the platform receives and retrieves the complete packet content, including the header and data portion.

[0076] The data packet is subjected to header stripping processing to obtain a structured data segment of the data packet.

[0077] Specifically, the packet header format must first be parsed. The packet header structure varies across protocols. For example, an Ethernet frame header includes fields such as the source MAC address, destination MAC address, and type / length. In IoT environments, a common protocol like MQTT includes a fixed header (such as the control message type and reserved bits) and a variable header (such as the topic name). By developing a parsing algorithm, the header fields are parsed byte by byte according to the corresponding protocol specifications to determine the header length. The header bytes are then removed from the entire packet, leaving the structured data segment. For example, for an IoT packet transmitted over TCP, the first few bytes are read to determine the TCP header length. From this point on, the subsequent bytes are intercepted as the structured data segment. This structured data segment may exist in a specific data format, such as JSON or XML, to facilitate subsequent processing.

[0078] Feature extraction is performed based on the structured data segment to obtain grammatical features of the structured data segment.

[0079] Specifically, if the structured data segment is in JSON format, the JSON object is first parsed to extract its individual elements. For example, if a JSON data segment is {"temperature":25,"humidity":60}, feature extraction involves identifying the two keys "temperature" and "humidity" and their corresponding numeric values ​​(25 is an integer, 60 is also an integer). For data segments in XML format, the XML tree structure is parsed to identify each tag and its content, and syntactic features such as tag names, attributes, and data types are determined. For other custom structured formats, the corresponding syntactic elements such as identifiers, delimiters, and data type declarations are extracted according to the format's defined rules. These extracted features serve as the basis for subsequently constructing the abstract syntax tree.

[0080] An abstract syntax tree of the structured data segment is constructed according to the grammatical features through a protocol-independent data expression model.

[0081] Specifically, the protocol-independent data representation model is a universal data structure representation method. When constructing an abstract syntax tree, elements of the syntax feature are used as nodes. For example, in the JSON example above, "temperature" and "humidity" are child nodes, and their parent node can be the root node representing the entire data object. Each node contains information such as the element name, data type, and value (if any). These nodes are connected according to the data hierarchy and syntax rules to form a tree structure. This process does not rely on specific data transmission protocols but is based on universal data representation rules. This allows data from different sources and protocols to be represented in a unified abstract syntax tree, facilitating subsequent standardization.

[0082] The abstract syntax tree is serialized according to a preset standard data template to obtain a standardized data stream of the structured data segment.

[0083] Specifically, a pre-set standard data template defines the format and structure of the data. For example, a standard template may require data to be output in a specific XML format, including a fixed root tag, subtag order, and data type declaration format. Serialization involves reorganizing the data according to the tags and format specified in the template, based on the structure of the abstract syntax tree. For example, according to the template requirements, the data for "temperature" and "humidity" are placed in corresponding standard XML tags, with the necessary attributes and formatting added to generate a standard-compliant XML data stream. This unified format of the processed data stream facilitates exchange and processing between different systems and applications.

[0084] All the standard data streams are integrated to obtain standard data information of the IoT device.

[0085] Specifically, the IoT platform collects standardized, serialized data streams from multiple IoT devices. The integration process can combine these data streams by chronological order, device type, or other classification methods. For example, data streams from different devices within the same time interval can be combined into a single dataset based on timestamps, or the devices can be aggregated by region, function, or other categories. This integrated, standardized dataset can be stored in a database to facilitate subsequent data analysis, querying, and application development, providing a unified and standardized data foundation for the IoT system. Each IoT device's unique identifier in the dataset is clearly defined. These identifiers, such as device ID, MAC address, or device name, have been standardized during the data standardization process. These identifiers serve as query keys to efficiently extract data for the corresponding device from the dataset. These identifiers are then used to perform structured queries within the standardized dataset to locate and isolate data records belonging to each device. This step involves categorizing and retrieving the dataset to ensure that the extracted data accurately corresponds to a specific device. Next, the extracted data for each device is parsed. Since the data is standardized, it can be parsed according to pre-set standard data templates. Standard data templates define the data format and structure, including information such as field names and data types. Parsing can transform raw data into a format that is easier to understand and use. For example, timestamps can be converted to a readable date and time format, or numeric data can be converted to values ​​in specific units. During data parsing, integrity checks and validation are also performed to ensure that the extracted data is complete and accurate, free of missing or corrupted data due to errors during data transmission or processing. If data anomalies or missing data are detected, interpolation, statistical estimation, and other methods can be used to correct and supplement the data to ensure its integrity and reliability. Finally, the parsed and validated data is consolidated into standardized data for each IoT device. This includes arranging the data in chronological order or categorizing it by functional modules to facilitate subsequent analysis and application. For example, environmental monitoring data such as temperature and humidity can be consolidated to form a comprehensive dataset on environmental conditions, or device status and fault information can be consolidated to form a monitoring dataset on device operational status. In this way, users can easily obtain comprehensive, accurate, and standardized data for each IoT device, supporting various applications and services on the IoT platform.

[0086] The data multi-protocol adaptive parsing method of the present invention avoids the high maintenance costs caused by frequent updates of protocol identification models or protocol mapping tables in the prior art by adopting a protocol-independent data expression model and an abstract syntax tree construction method. In traditional methods, with the continuous update of IoT devices and communication protocols, a large amount of manpower and material resources need to be continuously invested to maintain the relevant content of protocol identification. The method of the present invention can adapt to new communication protocols, significantly reducing the maintenance workload and resource consumption of the platform, thereby reducing the operating and maintenance costs of the IoT platform and improving the data communication management efficiency of the IoT platform. Specifically, the present invention can parse IoT data more efficiently by stripping the header of the data packet, extracting features for the structured data segment and constructing an abstract syntax tree. Compared with the prior art that relies on complex protocol header parsing, the present invention can quickly extract the key grammatical features of the data and convert it into the form of an abstract syntax tree, speeding up the data processing process and improving the efficiency of data processing. In addition, the abstract syntax tree is serialized according to a preset standard data template to generate a standardized data stream, further optimizing the organization and processing of the data, so that the data can be integrated and utilized more quickly, and improving the data processing performance of the entire IoT platform. The method of the present invention conducts in-depth analysis and understanding of data by constructing an abstract syntax tree, which can more accurately parse data packets sent by IoT devices. Compared with existing technologies that rely solely on protocol header identification, this method can better grasp the essential characteristics of the data, reduce data processing errors caused by protocol incompatibility or complex protocol identification errors, and improve the accuracy of data processing. At the same time, by converting data of different protocols into standardized data sets, it provides a high-quality, unified format data foundation for subsequent data analysis, application development, etc., helps to mine more valuable information, and further improves the data processing quality and application value of the IoT platform. The protocol-independent data expression model and abstract syntax tree construction method of the present invention can adapt to various different communication protocols without the need for large-scale adjustments and updates to the existing system. This makes it easier for the IoT platform to access new IoT devices and protocols, greatly improving the scalability of the system. In addition, by converting data into standardized data sets, it provides strong support for data integration and interoperability of different devices and different protocols, improves the data communication management efficiency of the IoT platform, and enables it to better adapt to the increasingly complex and diverse IoT application scenario requirements.

[0087] Optionally, the performing header stripping on the data packet to obtain the structured data segment of the data packet includes:

[0088] Locating the starting position and length of the header of the data packet based on the protocol header features of the data packet;

[0089] Separating header information from the data packet according to the header starting position and the length, removing the header information from the data packet, and extracting data information following the header information from the data packet as the structured data segment of the data packet;

[0090] Wherein, locating the starting position and length of the header of the data packet includes:

[0091] When the data packet conforms to the preset standard protocol format, the header starting position and the length are directly determined according to the preset standard protocol header characteristic parameters;

[0092] When the data packet does not conform to the preset standard protocol format, feature matching is performed on the data packet through a protocol-independent data feature analysis model to determine the header starting position and the length of the data packet.

[0093] Specifically, header stripping first requires determining the starting position and length of the packet's header. When a packet matches a pre-set standard protocol format, the protocol's header signature parameters are directly used to determine the header's starting position and length. For example, in the MQTT protocol, the packet's fixed header length is 2 bytes, starting at the beginning of the packet. This allows for rapid location and extraction of the header information. When a packet does not conform to the pre-set standard protocol format, a protocol-independent data feature analysis model is employed. Rather than relying on specific protocol rules, this model analyzes common packet features, such as repetitive patterns, fixed-length fields, or specific identifiers. By matching these features with known protocol header signatures, the starting position and length of the header can be inferred. For example, if a packet contains a fixed two-byte sequence, this may indicate that it is part of the header. The model uses these features to determine the header's location. After locating the header information, the header is separated from the packet and removed, leaving the structured data segment.

[0094] For example, consider a data packet sent by an IoT device that follows a custom communication protocol that doesn't conform to common standard protocols (such as MQTT or CoAP). The data packet structure for this custom protocol is as follows: the first two bytes are the device type identifier (fixed at 0xAA55), the next two bytes are the data length (including the total length of the header and data), the next byte is the custom protocol version number, followed by a one-byte reserved field, and finally the actual data segment (structured data segment).

[0095] When the IoT platform receives this data packet, since it doesn't conform to the pre-set standard protocol format, it activates a protocol-independent data signature analysis model for processing. The model first analyzes the data packet's signature, identifying the fixed pattern 0xAA55 in the first two bytes, which typically represents the start identifier of a custom protocol. The model further analyzes the packet's structure, discovering that the next two bytes represent the data length, a one-byte protocol version number, a one-byte reserved field, and finally the data segment. Through signature matching, the model determines the starting position of the header (the beginning of the packet) and its length (the first six bytes). The platform then uses this information to separate the header information from the packet and extract the data following the header as a structured data segment for further processing.

[0096] In this embodiment of the present invention, this positioning method based on protocol header features ensures the accuracy and efficiency of header stripping. For standard protocol packets, directly using preset parameters for header stripping allows for rapid and accurate processing, reducing unnecessary calculations and processing time. Furthermore, for non-standard protocol packets, the protocol-independent data feature analysis model provides flexibility and adaptability, enabling it to handle packets from various unknown or custom protocols, thereby improving the versatility and robustness of the entire system.

[0097] Optionally, the extracting features based on the structured data segment to obtain the grammatical features of the structured data segment includes:

[0098] Performing lexical analysis on the structured data segment to identify vocabulary units in the structured data segment, and performing grammatical analysis on the vocabulary units to determine grammatical relationships between the vocabulary units;

[0099] Extracting grammatical features of the structured data segment according to the grammatical relationship, wherein the grammatical features include the grammatical structure, data type and grammatical rules of the data segment;

[0100] Performing pattern matching on the grammatical features of the extracted structured data segment, identifying a specific grammatical pattern in the structured data segment, and using the specific grammatical pattern as part of the grammatical features;

[0101] performing vectorization processing on the grammatical features of the extracted structured data segment, and converting the grammatical features into vector grammatical feature representations;

[0102] Performing feature fusion on the vector grammatical feature representation to generate a grammatical feature vector of the structured data segment;

[0103] Performing feature dimensionality reduction processing on the grammatical feature vector, and then performing feature enhancement processing on the grammatical feature vector after dimensionality reduction to obtain a final grammatical feature representation;

[0104] The grammatical feature of the structured data segment is obtained according to the specific grammatical pattern and the grammatical feature representation.

[0105] Specifically, a lexical analyzer (such as Lex) scans a structured data segment and identifies lexical units such as keywords, identifiers, numbers, and operators. For example, in a JSON-formatted data segment, delimiters such as "{," "}," ":," and "," are identified, as well as lexical units such as key names and key values. A parser (such as Yacc) then performs grammatical analysis on these lexical units, determining the grammatical relationships between them based on predefined grammatical rules. For example, in an XML-formatted data segment, the correct nesting of tags and whether elements conform to the defined structure are determined. Based on the results of the grammatical analysis, grammatical features of the structured data segment are extracted. The grammatical structure includes the overall organization of the data segment, such as a JSON object consisting of key-value pairs or an XML document consisting of nested elements. The data type refers to the specific type of each data item, such as integer, floating-point number, or string. Syntax rules are the rules that constrain the structure and content of the data segment, such as tags must be closed in XML and key names must be enclosed in double quotes in JSON. Pattern matching is performed on the extracted grammatical features to identify specific grammatical patterns. For example, in log data, common error and warning log patterns can be identified by matching regular expressions. These specific grammatical patterns can be used as part of grammatical features, providing more valuable information for subsequent data processing. The extracted grammatical features are vectorized and converted into vector grammatical feature representations. One-hot encoding can be used to assign a dimension to each possible grammatical feature. If a feature exists, the corresponding dimension value is 1, otherwise it is 0. For example, for data type features, if there are three types: integer, floating-point number, and string, the corresponding vector dimension is 3. When the data item is an integer, the vector representation is [1, 0, 0]. The vector grammatical feature representations are fused to generate a grammatical feature vector for the structured data segment. This may involve concatenating or weighted summing grammatical feature vectors of different types to obtain a comprehensive feature vector. The grammatical feature vector is then subjected to feature dimensionality reduction to reduce the feature dimension and remove redundant information. Dimensionality reduction algorithms such as principal component analysis (PCA) can be used to retain the main feature components and reduce data complexity. Finally, feature enhancement is performed to highlight important feature information and improve feature separability and discrimination. For example, feature scaling and normalization are used to keep feature values ​​within a consistent range, or new, more representative features are generated through feature construction. Based on specific grammatical patterns and grammatical feature representations, the grammatical features of the structured data segment are synthesized. These grammatical features can comprehensively and accurately describe the grammatical properties of the structured data segment, providing strong support for subsequent data processing and analysis.

[0106] In an embodiment of the present invention, through lexical analysis and grammatical analysis, the vocabulary units and their grammatical relations in the structured data segment can be accurately identified, thereby extracting detailed grammatical features. Accurate basic information is provided for subsequent data processing and analysis. Pattern matching of grammatical features can identify specific grammatical patterns, which often have important business significance. At the same time, through vectorization, feature fusion, dimensionality reduction and enhancement, key features can be highlighted, and the analyzability and comprehensibility of the data can be improved. In addition, this optional embodiment is applicable to a variety of structured data formats, such as JSON, XML, CSV, etc. By extracting and processing the grammatical features of different data formats, it can achieve unified analysis and management of different types of data, and has strong adaptability and versatility.

[0107] Optionally, constructing an abstract syntax tree of the structured data segment according to the grammatical features using a protocol-independent data expression model includes:

[0108] Initializing the protocol-independent data expression model to obtain a preset syntax tree generation rule and data structure;

[0109] Taking the grammatical features of the structured data segment as input, determining the root node of the abstract syntax tree according to the type and content of the grammatical features through the protocol-independent data expression model;

[0110] Determining, based on the grammatical structure and grammatical rules in the grammatical features, a rule for creating branch nodes and a rule for creating leaf nodes of the abstract syntax tree;

[0111] Adjusting the node structure of the abstract syntax tree according to the specific syntax pattern;

[0112] By using the generation algorithm of the protocol-independent data expression model, based on the grammatical feature vector and the grammatical feature representation, sequentially constructing the hierarchical structure of the abstract syntax tree;

[0113] The abstract syntax tree of the structured data segment constructed is outputted through the protocol-independent data expression model.

[0114] Specifically, the model first loads the pre-set syntax tree generation rules and data structure. These rules define the general structure and node types of the abstract syntax tree (AST). The data structure is used to store and manage the nodes and their relationships within the AST. For example, a node class can be defined, containing properties such as node type, node value, and child node list. The model then takes the grammatical features of the structured data segment as input and determines the root node of the AST based on the type and content of the grammatical features. For example, if the grammatical features represent a JSON object, the root node can be a node representing the object; if it represents an XML document, the root node can be a node representing the document's root element. Based on the grammatical structure and grammatical rules within the grammatical features, the model determines the rules for creating branch and leaf nodes. For example, for key-value pairs in a JSON object, the key can serve as a branch node, and the value as a leaf node; for XML elements, the element name can serve as a branch node, and the element content can serve as a leaf node. The model also considers nested relationships and grammatical rules, such as nested objects or arrays in JSON objects and nested elements in XML. Based on specific grammatical patterns, the model adjusts the node structure of the AST. For example, if a specific pattern is detected in the data, such as the consecutive occurrence of multiple elements of the same type, the model may create a branch node representing a list, with these elements as its child nodes. This optimizes the tree structure and makes it more clearly reflect the logical structure of the data. The protocol-independent data representation model's generation algorithm sequentially constructs a hierarchical abstract syntax tree based on the grammatical feature vectors and grammatical feature representations. Starting from the root node, the model creates branch nodes and leaf nodes layer by layer, following the hierarchical relationships in the grammatical features, until the grammatical structure of the structured data segment is fully represented. For example, for a JSON data segment containing nested objects, the model first creates a root node (representing the entire object), then creates branch nodes and leaf nodes based on key-value pairs, and further creates sub-branches for nested objects. Finally, the protocol-independent data representation model outputs the completed abstract syntax tree. This abstract syntax tree represents the grammatical structure of the structured data segment in a protocol-independent manner and can be used for subsequent data processing and analysis.

[0115] In an embodiment of the present invention, by using a protocol-independent data expression model, structured data of different protocols and formats can be processed without having to design syntax tree generation rules separately for each protocol, thereby improving the versatility and adaptability of the system. By constructing an abstract syntax tree based on grammatical features, the grammatical structure and logical relationship of structured data can be accurately represented, providing a clear data model for subsequent data processing and analysis. By adjusting the node structure according to a specific grammatical pattern, the representation of the abstract syntax tree can be optimized to make it more consistent with the actual logical structure of the data, thereby improving the accuracy and readability of the data representation. As a form of data abstraction, the abstract syntax tree can simplify complex data structures into a tree structure, facilitate operations such as data query, conversion and analysis, and improve the efficiency and effectiveness of data processing. When processing data from different IoT devices using different protocols, this embodiment can provide a unified data representation method, which is conducive to data integration and interoperability, and has important technical value for data processing and analysis on the IoT platform.

[0116] Optionally, the serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment includes:

[0117] Obtaining a preset standard data template, wherein the preset standard data template includes the format, structure, and semantic rules of the data stream;

[0118] Matching the root node of the abstract syntax tree with the root element of the standard data template according to the format, structure and semantic rules of the data stream;

[0119] Converting the branch nodes of the abstract syntax tree and the leaf nodes of the abstract syntax tree into an intermediate data structure that conforms to the standard data template;

[0120] According to the specific grammatical pattern and the grammatical rules, the intermediate data structure is sequentially semantically mapped and format converted to obtain an intermediate data structure after format conversion, and the intermediate data structure after format conversion is verified to determine whether it meets the constraints of the standard data template;

[0121] If so, the verified intermediate data structure is serialized to generate a linear standardized data stream, and the linear standardized data stream is used as the standardized data stream of the structured data segment.

[0122] Specifically, the preset standard data template defines the format, structure, and semantic rules of the target data flow. For example, the preset standard data template specifies that the data flow adopts XML format, including specific root elements, subelements, and their attributes, as well as the hierarchical relationships and semantic meanings between elements. The root node of the abstract syntax tree is matched with the root element of the standard data template. For example, if the root node of the abstract syntax tree represents a device data object, and the root element of the standard data template is "device_data," the root node is mapped to the "device_data" element. The branch nodes and leaf nodes of the abstract syntax tree are converted into intermediate data structures that conform to the standard data template. For example, a branch node representing a device attribute can be converted into an intermediate object containing information such as the attribute name and value. Leaf nodes can be converted into specific values ​​or simple data items. Semantic mapping and format conversion are performed on the intermediate data structures based on specific syntax patterns and rules. For example, the date format is converted from "YYYYMMDD" to the "YYYY-MM-DD" format required by the standard template. After conversion, the intermediate data structure is validated to check for compliance with the constraints of the standard data template, such as data type, value range, and required fields. If the intermediate data structure passes validation, it is serialized to generate a linear, standardized data stream. The serialization process converts the intermediate data structure into a continuous stream of bytes or characters, following the format and structure of the standard data template. For example, the intermediate object is converted into an XML-formatted string, organizing the data according to the element order and nesting relationships specified by the template. Finally, the serialization process outputs a standardized data stream of structured data segments.

[0123] In an embodiment of the present invention, by serializing an abstract syntax tree according to a preset standard data template, data of different protocols and formats can be converted into a unified standardized data stream, helping to achieve data interoperability and integration, and facilitating data exchange and sharing between different systems and applications. Semantic mapping and format conversion are performed according to the semantic rules of the standard data template, ensuring the semantic accuracy and consistency of the converted data, helping to avoid data ambiguity and misunderstanding, and improving data quality and reliability. Verifying the intermediate data structure during the serialization process ensures that the generated standardized data stream conforms to the constraints of the preset standard data template, helping to ensure data integrity and correctness, and avoiding data errors and inconsistencies. This embodiment can adapt to different standard data templates, simply by adjusting the corresponding mapping rules and verification conditions to achieve serialization processing for different target formats. This makes the system highly flexible and adaptable, able to meet diverse data processing needs. By combining the abstract syntax tree with the preset standard data template, standardized processing of structured data segments is achieved, providing strong support for the integration, analysis, and application of IoT data.

[0124] Optionally, the performing semantic mapping and format conversion on the intermediate data structure in sequence according to the specific grammatical pattern and the grammatical rules to obtain the intermediate data structure after format conversion, and verifying the intermediate data structure after format conversion to determine whether it meets the constraints of the standard data template includes:

[0125] Performing semantic analysis on the intermediate data structure to identify data elements and data relationships in the intermediate data structure;

[0126] Mapping the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships;

[0127] Adjusting the format of the intermediate data structure according to the standardized semantic representation, including data type conversion and data encoding specifications;

[0128] The converted intermediate data structure is format-verified to determine whether it complies with the constraints of the standard data template.

[0129] Specifically, by traversing the intermediate data structure, the data elements (such as numerical values, character strings, etc.) and data relationships (such as hierarchical relationships and association relationships between elements) therein are identified. For example, a numerical element representing temperature and a string element representing time are found in the intermediate data structure, and an association relationship exists between them, representing the temperature measured at a specific time. The identified data elements and data relationships are mapped to a preset standard semantic model. The preset standard semantic model defines a standardized semantic representation of data, such as mapping temperature to "temperature" and time to "timestamp". By looking up mapping rules or using a semantic dictionary, the data elements and relationships in the intermediate data structure are converted to a standardized semantic representation. According to the standardized semantic representation, the format of the intermediate data structure is adjusted. This includes data type conversion (such as converting a string to an integer or floating-point number) and data encoding specifications (such as a unified date format, character encoding, etc.). For example, converting a temperature value from the string "25" to the floating-point number 25.0, or converting a time string from "2024 / 01 / 01 12:00" to the standard ISO 8601 format "2024-01-01T12:00:00Z," performs format verification on the converted intermediate data structure according to the constraints of the standard data template. This verification includes ensuring the correct data type, compliance with the data format, and the completeness of required fields. For example, the system checks whether the temperature value is a floating-point number, whether the time format complies with the ISO 8601 standard, and whether any required fields are missing. Furthermore, during semantic analysis of the intermediate data structure, grammar rules provide a basis for identifying data elements and data relationships. For example, according to the definitions in the grammar rules, it is recognized that numeric elements represent specific measurement values, such as temperature and humidity, while string elements represent timestamps and device names. Furthermore, grammar rules help clarify the hierarchical and associative relationships between elements. For example, the correspondence between a temperature value and a timestamp indicates the temperature measured at a specific time.

[0130] In an embodiment of the present invention, the data elements and relationships in the intermediate data structure are converted into standardized semantic representations through semantic mapping, which ensures the semantic consistency and accuracy of the data and facilitates different systems and applications to understand and process the data. The intermediate data structure is formatted and verified to conform to the format and encoding specifications of the preset standard data template, which helps to improve the readability and processability of the data and reduce errors and misunderstandings caused by format inconsistencies. The strict format verification process can promptly detect and correct errors and inconsistencies in the data, ensure the quality and reliability of the data, and provide a solid foundation for subsequent data processing and analysis. Data that has undergone semantic mapping and format conversion can better interoperate with other systems and applications, promote data sharing and integration, and improve the efficiency and collaboration capabilities of the entire Internet of Things system.

[0131] Optionally, mapping the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships includes:

[0132] Matching the intermediate data structure with a preset standard semantic model to determine the correspondence between the data elements and the data relationships;

[0133] According to the specific grammatical pattern and the grammatical rules, the data elements and the data relations are converted into standardized semantic elements and semantic relations in the preset standard semantic model;

[0134] Preprocessing the converted standardized semantic elements and semantic relations to obtain the standardized semantic elements and semantic relations that meet the requirements of the preset standard semantic model;

[0135] The standardized semantic elements and the semantic relationships that meet the requirements are integrated to form standardized semantic representations of the data elements and the data relationships.

[0136] Specifically, grammatical rules guide how to map identified data elements and data relationships to a pre-set standard semantic model. For example, grammatical rules specify that temperature-related elements should be mapped to "temperature" and time-related elements should be mapped to "timestamp." By following these grammatical rules, the accuracy and consistency of semantic mappings are ensured, ensuring that the data semantically conforms to the requirements of the standard model. The pre-set standard semantic model is a model that includes standardized semantic elements and semantic relationships. It defines various possible data elements, their corresponding semantic meanings, and the semantic relationships between data. Data elements and data relationships in the intermediate data structure are matched against this model to determine their corresponding relationships. For example, if a data element in the intermediate data structure is "tmp," matching indicates that it corresponds to the element "temperature" in the pre-set standard semantic model. Based on specific grammatical patterns, the data elements and data relationships in the intermediate data structure are converted to the standardized semantic elements and semantic relationships in the pre-set standard semantic model. This may involve operations such as replacing element names and adjusting data structures. For example, "tmp" can be converted to "temperature," "tm" (time) can be converted to "timestamp," and the relationships between them can be converted to the standard "measured at" relationship. The converted standardized semantic elements and semantic relationships are preprocessed to ensure that they meet the requirements of the preset standard semantic model. Preprocessing may include removing redundant information, supplementing missing information, and unifying data formats. For example, removing extra spaces, standardizing the date format to a standard format, and supplementing missing unit information. The preprocessed and qualified standardized semantic elements and semantic relationships are integrated to form a standardized semantic representation of data elements and data relationships. The integrated semantic representation is a complete data structure that conforms to the standard semantic model and can be used for subsequent serialization or other data processing operations. For example, the integrated semantic representation may be an XML or JSON object containing "temperature" and "timestamp" elements and their relationships.

[0137] In an embodiment of the present invention, by mapping data elements and data relationships to a preset standard semantic model, the semantic consistency and standardization of the data are ensured, and semantic confusion and misunderstanding caused by the use of different terminology in different data sources are avoided. The conversion and preprocessing process makes the data elements and semantic relationships conform to standardization requirements, improves the standardization and quality of the data, and facilitates subsequent data processing and analysis. The standardized semantic representation makes the data easier to understand and process by different systems and applications, enhances the interoperability of the system, and promotes data sharing and integration. The integrated standardized semantic representation provides a unified basis for further processing and analysis of the data, facilitating operations such as data query, statistical analysis, and data mining.

[0138] Optionally, the method further includes: constructing the protocol-independent data expression model;

[0139] The protocol-independent data expression model includes:

[0140] Initialize the model framework of the protocol-independent data expression model and set the basic structure and parameters of the protocol-independent data expression model;

[0141] Acquire a standard communication protocol as a data sample, wherein the data sample is used for model training;

[0142] Preprocessing the data samples, extracting and marking features of the preprocessed data samples to obtain the data samples with feature markings, inputting the data samples with feature markings into the model framework for training, and generating an initial protocol-independent data expression model;

[0143] The initial protocol-independent data expression model is verified and optimized to obtain the final protocol-independent data expression model.

[0144] Specifically, protocol-independent data representation models can be based on recurrent neural networks (RNNs) or their variants in deep learning, such as long short-term memory (LSTM) or gated recurrent units (GRUs), as these models excel at processing sequential data and capturing structure and dependencies within the data. First, initialize the basic model structure, including the input layer, hidden layers, and output layers. For example, for an LSTM model, set the input layer dimensions to receive preprocessed protocol data, the number of hidden layer units to control model complexity, and the output layer dimensions to generate a grammatical feature vector. Collect data from a variety of standard communication protocols as training samples, such as packets from MQTT, CoAP, and HTTP. These data samples should cover a variety of typical protocol formats and data structures to ensure the model can learn a wide range of protocol features. Preprocess the acquired data samples, including noise removal and data format standardization. Then, perform feature extraction on the preprocessed data samples to identify key grammatical features, such as packet header features, data structure, and specific grammatical patterns. These features will serve as the model input. Simultaneously, label the data samples to create feature-labeled data samples to guide model learning. For example, mark the header information in the data packet, the start and end positions of the data segment, the data type, etc. Input the data samples with feature labels into the initialized model framework for training. During the training process, the model will learn how to extract grammatical features from the input data and generate the corresponding grammatical feature vectors. By adjusting the model parameters, such as weights and biases, the output of the model is made as close as possible to the predefined feature labels, thereby optimizing the model performance. Use the validation dataset to verify the initial protocol-independent data expression model obtained through training to evaluate its accuracy and generalization ability. Based on the validation results, optimize the model, such as adjusting the model structure, modifying hyperparameters, adding regularization, etc., to improve the performance and robustness of the model. Finally, a verified and optimized protocol-independent data expression model is obtained.

[0145] For example, choose LSTM as a protocol-independent data representation model. Initialize an LSTM model with an input layer dimension of 128 (assuming the feature dimension of each time step of the preprocessed protocol data is 128), a hidden layer unit of 256, and an output layer dimension of 64 (to generate a 64-dimensional grammatical feature vector). Collect 10,000 packets from protocols such as MQTT, CoAP, and HTTP as training samples. Preprocess the data samples, removing irrelevant information from the packets (such as transport layer headers), converting the binary content of the packets into integer sequences (each byte represented by an integer from 0 to 255), and truncating or padding each packet to a fixed length of 100 bytes. Extract grammatical features from each packet, such as header length, data segment start position, and data type, and convert them into label vectors. For example, use an integer to represent the header length, an index to represent the data segment start position, and a one-hot encoding to represent the data type. The preprocessed packets and their corresponding label vectors form a training dataset and are input into the LSTM model for training. Using the cross-entropy loss function and the Adam optimizer, the model was trained for 10 epochs with a batch size of 32. A validation dataset of 2,000 packets from various protocols was used to validate the trained initial model, calculating metrics such as precision, recall, and F1 score. Based on the validation results, the model's hidden layer number was adjusted to 512, and a dropout layer was added to prevent overfitting. The model was then retrained and validated again until satisfactory performance was achieved, resulting in the final protocol-independent data representation model.

[0146] In an embodiment of the present invention, the constructed protocol-independent data expression model is capable of processing data from multiple standard communication protocols. In an IoT environment, devices may use a variety of different communication protocols, such as MQTT, CoAP, HTTP, etc. By using data samples from multiple protocols for training, the model can learn the characteristics of data from different protocols, thereby achieving universal support for multiple protocols. For example, the model can identify the header features and data segment structure of data packets in different protocols, even if these protocols have significant differences in format and semantics. In addition, the adaptability of the model is also reflected in the processing of new or custom protocols. When encountering new protocol data, the model can use its learned general feature extraction capabilities to try to parse and process this data without the need for large-scale retraining of the model. This enables the IoT platform to more flexibly access and manage different types of devices and protocols.

[0147] This model efficiently extracts grammatical features from protocol data and converts them into a unified grammatical feature vector representation. Through training with deep learning models (such as LSTM), the model automatically learns complex patterns and dependencies within the data, capturing key grammatical features such as packet structure, data type, and specific patterns. This unified grammatical feature vector representation facilitates subsequent data processing steps. For example, when constructing an abstract syntax tree, these feature vectors can be used to quickly determine the node creation rules and structure, eliminating the need to design parsing logic for each protocol. This improves the efficiency and consistency of the entire data processing process.

[0148] During model training, by learning from a large number of feature-labeled data samples, the model accurately identifies and extracts grammatical features from protocol data. Verification and optimization steps further enhance the model's performance, making it more accurate and reliable when processing data from various protocols. Accurate grammatical feature extraction is crucial for subsequent data processing. For example, during data serialization, accurate grammatical features ensure that the generated standardized data stream conforms to the pre-set standard data template, avoiding data errors and inconsistencies, thereby improving the reliability and stability of the entire system. The protocol-independent data representation model provides a unified representation for data from different protocols, enabling data from different devices and protocols to be processed and analyzed within the same framework, facilitating data integration. This unified data representation improves system interoperability, allowing different systems and applications to exchange and share data based on this unified representation without having to worry about the data's original protocol format. For example, a data analysis application can process model-processed data from various protocols without having to adapt to each protocol, simplifying system integration and application development.

[0149] In a second aspect, the present invention provides a data multi-protocol adaptive parsing system, which is applied to an Internet of Things platform, wherein the Internet of Things platform is communicatively connected with an Internet of Things device, and the system includes:

[0150] A data receiving unit, configured to receive data packets sent by the IoT device;

[0151] A data processing unit, configured to perform header stripping processing on the data packet to obtain a structured data segment of the data packet;

[0152] a feature extraction unit, configured to extract features from the structured data segment to obtain grammatical features of the structured data segment;

[0153] an abstract syntax tree construction unit, configured to construct an abstract syntax tree of the structured data segment according to the syntax features through a protocol-independent data expression model;

[0154] A standardization unit, configured to serialize the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment;

[0155] The integration unit is used to integrate all the standard data streams to obtain standard data information of the Internet of Things devices.

[0156] The advantages of the data multi-protocol adaptive parsing system of the present invention over the prior art are the same as the advantages of the above-mentioned data multi-protocol adaptive parsing method over the prior art, and will not be repeated here.

[0157] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned data multi-protocol adaptive parsing method is implemented.

[0158] The advantages of the computer-readable storage medium of the present invention over the prior art are the same as the advantages of the above-mentioned data multi-protocol adaptive parsing method over the prior art, and will not be repeated here.

[0159] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A data multi-protocol adaptive parsing method, characterized in that: The method is applied to an Internet of Things platform, wherein the Internet of Things platform is communicatively connected with an Internet of Things device, and the method includes: Receive a data packet sent by the IoT device; Performing header stripping processing on the data packet to obtain a structured data segment of the data packet; Performing feature extraction based on the structured data segment to obtain grammatical features of the structured data segment; Constructing the protocol-independent data expression model; the protocol-independent data expression model includes: Initialize the model framework of the protocol-independent data expression model and set the basic structure and parameters of the protocol-independent data expression model; Acquire a standard communication protocol as a data sample, wherein the data sample is used for model training; Preprocessing the data samples, extracting and marking features of the preprocessed data samples to obtain the data samples with feature markings, inputting the data samples with feature markings into the model framework for training, and generating an initial protocol-independent data expression model; Verifying and optimizing the initial protocol-independent data expression model to obtain a final protocol-independent data expression model; constructing an abstract syntax tree of the structured data segment according to the grammatical features through a protocol-independent data expression model; Specifically including: initializing the protocol-independent data expression model to obtain preset syntax tree generation rules and data structure; Taking the grammatical features of the structured data segment as input, determining the root node of the abstract syntax tree according to the type and content of the grammatical features through the protocol-independent data expression model; Determining, based on the grammatical structure and grammatical rules in the grammatical features, a rule for creating branch nodes and a rule for creating leaf nodes of the abstract syntax tree; Adjusting the node structure of the abstract syntax tree according to a specific syntax pattern; By using the generation algorithm of the protocol-independent data expression model, based on the grammatical feature vector and the grammatical feature representation, sequentially constructing the hierarchical structure of the abstract syntax tree; Outputting the constructed abstract syntax tree of the structured data segment through the protocol-independent data expression model; Serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment; All the standardized data streams are integrated to obtain standard data information of the IoT devices.

2. The data multi-protocol adaptive parsing method according to claim 1, characterized in that: The step of stripping the header of the data packet to obtain the structured data segment of the data packet includes: Locating the starting position and length of the header of the data packet based on the protocol header features of the data packet; Separating header information from the data packet according to the header starting position and the length, removing the header information from the data packet, and extracting data information following the header information from the data packet as the structured data segment of the data packet; Wherein, locating the starting position and length of the header of the data packet includes: When the data packet conforms to the preset standard protocol format, the header starting position and the length are directly determined according to the preset standard protocol header characteristic parameters; When the data packet does not conform to the preset standard protocol format, feature matching is performed on the data packet through a protocol-independent data feature analysis model to determine the header starting position and the length of the data packet.

3. The data multi-protocol adaptive parsing method according to claim 2, characterized in that: The extracting features based on the structured data segment to obtain the grammatical features of the structured data segment includes: Performing lexical analysis on the structured data segment to identify vocabulary units in the structured data segment, and performing grammatical analysis on the vocabulary units to determine grammatical relationships between the vocabulary units; Extracting grammatical features of the structured data segment according to the grammatical relationship, wherein the grammatical features include the grammatical structure, data type and grammatical rules of the data segment; Performing pattern matching on the grammatical features of the extracted structured data segment, identifying a specific grammatical pattern in the structured data segment, and using the specific grammatical pattern as part of the grammatical features; performing vectorization processing on the grammatical features of the extracted structured data segment, and converting the grammatical features into vector grammatical feature representations; Performing feature fusion on the vector grammatical feature representation to generate a grammatical feature vector of the structured data segment; Performing feature dimensionality reduction processing on the grammatical feature vector, and then performing feature enhancement processing on the grammatical feature vector after dimensionality reduction to obtain a final grammatical feature representation; The grammatical feature of the structured data segment is obtained according to the specific grammatical pattern and the grammatical feature representation.

4. The data multi-protocol adaptive parsing method according to claim 1, characterized in that: The step of serializing the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment includes: Obtaining a preset standard data template, wherein the preset standard data template includes the format, structure, and semantic rules of the data stream; Matching the root node of the abstract syntax tree with the root element of the standard data template according to the format, structure and semantic rules of the data stream; Converting the branch nodes of the abstract syntax tree and the leaf nodes of the abstract syntax tree into an intermediate data structure that conforms to the standard data template; According to the specific grammatical pattern and the grammatical rules, the intermediate data structure is sequentially semantically mapped and format converted to obtain an intermediate data structure after format conversion, and the intermediate data structure after format conversion is verified to determine whether it meets the constraints of the standard data template; If so, the verified intermediate data structure is serialized to generate a linear standardized data stream, and the linear standardized data stream is used as the standardized data stream of the structured data segment.

5. The data multi-protocol adaptive parsing method according to claim 4, characterized in that: The step of sequentially performing semantic mapping and format conversion on the intermediate data structure according to the specific grammatical pattern and the grammatical rules to obtain an intermediate data structure after format conversion, and verifying the intermediate data structure after format conversion to determine whether it meets the constraints of the standard data template includes: Performing semantic analysis on the intermediate data structure to identify data elements and data relationships in the intermediate data structure; Mapping the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships; Adjusting the format of the intermediate data structure according to the standardized semantic representation, including data type conversion and data encoding specifications; The converted intermediate data structure is format-verified to determine whether it complies with the constraints of the standard data template.

6. The data multi-protocol adaptive parsing method according to claim 5, characterized in that: The mapping of the data elements and the data relationships to a preset standard semantic model according to the specific grammatical pattern and the grammatical rules to obtain standardized semantic representations of the data elements and the data relationships includes: Matching the intermediate data structure with a preset standard semantic model to determine the correspondence between the data elements and the data relationships; According to the specific grammatical pattern and the grammatical rules, the data elements and the data relations are converted into standardized semantic elements and semantic relations in the preset standard semantic model; Preprocessing the converted standardized semantic elements and semantic relations to obtain the standardized semantic elements and semantic relations that meet the requirements of the preset standard semantic model; The standardized semantic elements and the semantic relationships that meet the requirements are integrated to form standardized semantic representations of the data elements and the data relationships.

7. A data multi-protocol adaptive parsing system, characterized in that: The system is applied to an Internet of Things platform, which is communicatively connected with an Internet of Things device. The system includes: A data receiving unit, configured to receive data packets sent by the IoT device; A data processing unit, configured to perform header stripping processing on the data packet to obtain a structured data segment of the data packet; a feature extraction unit, configured to extract features from the structured data segment to obtain grammatical features of the structured data segment; An abstract syntax tree construction unit is used to construct the protocol-independent data expression model; the protocol-independent data expression model includes: Initialize the model framework of the protocol-independent data expression model and set the basic structure and parameters of the protocol-independent data expression model; Acquire a standard communication protocol as a data sample, wherein the data sample is used for model training; Preprocessing the data samples, extracting and marking features of the preprocessed data samples to obtain the data samples with feature markings, inputting the data samples with feature markings into the model framework for training, and generating an initial protocol-independent data expression model; Verifying and optimizing the initial protocol-independent data expression model to obtain a final protocol-independent data expression model; constructing an abstract syntax tree of the structured data segment according to the grammatical features through a protocol-independent data expression model; Specifically including: initializing the protocol-independent data expression model to obtain preset syntax tree generation rules and data structure; Taking the grammatical features of the structured data segment as input, determining the root node of the abstract syntax tree according to the type and content of the grammatical features through the protocol-independent data expression model; Determining, based on the grammatical structure and grammatical rules in the grammatical features, a rule for creating branch nodes and a rule for creating leaf nodes of the abstract syntax tree; Adjusting the node structure of the abstract syntax tree according to a specific syntax pattern; By using the generation algorithm of the protocol-independent data expression model, based on the grammatical feature vector and the grammatical feature representation, sequentially constructing the hierarchical structure of the abstract syntax tree; Outputting the constructed abstract syntax tree of the structured data segment through the protocol-independent data expression model; A standardization unit, configured to serialize the abstract syntax tree according to a preset standard data template to obtain a standardized data stream of the structured data segment; The integration unit is used to integrate all the standardized data streams to obtain standard data information of the Internet of Things devices.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the data multi-protocol adaptive parsing method according to any one of claims 1 to 6 is implemented.

Images