A data frame extraction method and system based on declarative description

Abstract

The application discloses a data frame extraction method and system based on a declarative description, and the method comprises the following steps: S100, acquiring a declarative format description file, wherein the declarative format description file is used for declaring the structure of a target binary data stream; S200, generating a parsing plan tree, wherein a general parsing engine dynamically loads the declarative format description file at runtime, and compiles the declarative format description file into a parsing plan tree in memory; S300, processing a data stream, wherein a binary data stream is accessed, the general parsing engine executes the parsing plan tree in a depth-first traversal manner, recursively processes structure body and array nodes, and uniformly processes atomic field nodes; and S400, standardizing output, wherein the extracted data frame is encapsulated into a standardized data frame object and output. Through complete decoupling of data format description and a parsing engine, the application realizes runtime dynamic parsing of non-standard and diversified structure binary data and standardized data frame extraction.

Description

Technical Field

[0001] This invention relates to the field of network data transmission and processing technology, and specifically to a data frame extraction method and system based on declarative description. Background Technology

[0002] In areas such as network security monitoring, performance analysis, troubleshooting, and protocol reverse engineering, in-depth analysis of network traffic data is a core task. In real-world applications, the diversity and complexity of data sources far exceed standard scenarios. Analysts often deal not with standard PCAP files, but with binary data files from various network devices, embedded systems, industrial control systems (ICS), Internet of Things (IoT) devices, or proprietary software logs. These files typically end with custom suffixes such as .dat, .log, .bin, and .trace. While these binary files encapsulate raw network communication frames internally, their overall file structure, metadata format, and frame encapsulation methods are non-standard and highly customized.

[0003] To extract valuable network frame information from these heterogeneous data sources, existing technologies mainly employ the following schemes, but all of them have significant limitations:

[0004] Hard-coded parsing: For each specific binary format, a dedicated parser is written using languages ​​such as C / C++ or Python. This method hard-codes the specific details of the data format (such as field offsets, lengths, and byte order) into the program logic. Its disadvantages include high development costs and long development cycles. Furthermore, any slight change in the data format necessitates modifying the source code and recompiling, testing, and deploying, resulting in a heavy maintenance burden and an inability to adapt to rapidly changing analysis needs.

[0005] Scripted rapid parsing: Parsing logic is written using scripting languages ​​such as Python. While this reduces the initial development cost, it doesn't change the fundamental nature of "parsing logic being coupled with format definition." Format changes still require script modifications, and in scenarios involving massive amounts of data or demanding high real-time performance, script execution efficiency often becomes a bottleneck.

[0006] Parser generator frameworks employ tools like Kaitai Struct, using a domain-specific language (DSL) to describe data structures. The tool then compiles this DSL to generate parser source code in the target language (e.g., Python, C++). This approach separates the format description from the code, representing a significant advancement. However, it is essentially an "offline code generation" model. After supporting new formats or modifying existing format descriptions, the entire process of "regenerating code -> integrating into the project -> recompiling and deploying" is still required, failing to achieve dynamic runtime adaptation. Once deployed, the range of formats the system can parse is fixed and limited.

[0007] In summary, existing technologies generally suffer from fundamental drawbacks when processing diverse and non-standard network binary data streams, including strong coupling between parsing logic and data format, inability to dynamically extend at runtime, high development and maintenance thresholds, and long development cycles. Therefore, there is an urgent need in this field for an innovative technical solution that can achieve zero-code, runtime dynamic adaptation to various custom binary formats, and efficiently and stream-wise extract standardized network frame information, thus bridging the gap between raw data and analytical applications. Summary of the Invention

[0008] To address the technical problems of high development costs and inability to dynamically adapt at runtime caused by hard-coded coupling between parsing logic and data format in existing binary data parsing technologies, this invention provides a data frame extraction method and system based on declarative description.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] Firstly, a data frame extraction method based on declarative description includes the following steps:

[0011] S100. Obtain the declarative format description file: The declarative format description file is used to declare the structure of the target binary data stream;

[0012] S200, Parsing Plan Tree Generation: The general parsing engine dynamically loads the declarative format description file at runtime and compiles it into a parsing plan tree in memory; the parsing plan tree is a tree-like executable data structure, which consists of a series of internal data structure nodes corresponding to different data logic units, and the data encapsulation hierarchy is represented by the parent-child relationship between nodes.

[0013] The general parsing engine is the general binary parsing engine UBPE;

[0014] The data structure nodes in the parsing plan tree include at least: file nodes, channel nodes, and frame nodes;

[0015] S300, Data Stream Processing: Upon receiving the binary data stream, the general parsing engine executes the parsing plan tree in a depth-first traversal manner, recursively processes the structure and array nodes, and uniformly processes the atomic field nodes;

[0016] Specifically, data frames are extracted from the binary data stream using a streaming method.

[0017] The processing of atomic field nodes includes support for determining field length based on dynamic references;

[0018] S400, Standardized Output: Encapsulates the extracted data frames into standardized data frame objects and outputs them.

[0019] Furthermore, the declarative format description file is defined based on YAML syntax and is used to declare the structure of the target binary data stream, which is a raw data stream with a non-standard encapsulation structure generated by a network device or embedded system.

[0020] Furthermore, the declarative format description file is used at least to declare:

[0021] Atomic field declarations include an unsigned integer type for representing the length of a network data frame, an integer type for representing a timestamp, and a byte array type for representing the frame payload;

[0022] Structure declarations are used to define composite data structures through an ordered sequence of fields, where each field defines its type by referencing the underlying data type or by nesting other structures / arrays;

[0023] An array declaration is used to define the repetition of the structure.

[0024] Domain-specific keyword declarations, the keywords including at least frame_payload for identifying the start of the frame payload, frame_cap_len for declaring the actual capture length, and frame_orig_len for declaring the original network length;

[0025] A dynamic reference declaration is used to make the parsed parameters of a field dependent on the value of a previously parsed field through a reference statement. The format of the reference statement is {ref: "target field name"}.

[0026] Further, the file node corresponds to the logical root container of the entire binary data file or data stream; the frame node corresponds to the parsing template of a complete network data frame; the channel node is used to represent multiplexed or logically separated data sub-streams; wherein, the file node may contain one or more channel nodes, and each channel node contains one or more frame nodes.

[0027] Furthermore, step S300 specifically includes the following stages:

[0028] S310, Initialization and Root Node Processing Phase: Starting from the file node of the parsing plan tree, read the global attributes defined therein to initialize the parsing context, and place the read pointer of the binary data stream at the starting position;

[0029] S320, Optional Channel Processing Stage: If a channel node exists in the parsing plan tree, it is identified and synchronized to the channel structure in the data stream; otherwise, it directly enters the frame processing stage.

[0030] S330, Frame Processing Stage: Based on the dynamic frame length, identify and synchronize the frame to the beginning of the frame structure in the data stream, and then traverse and process the subtree under the frame node to extract and encapsulate the frame data.

[0031] Furthermore, the specific steps in step S330 for traversing and processing the subtree under this frame node include:

[0032] Iterate through the list of field definitions under the current node in sequence; for each field, perform the corresponding parsing operation based on its node type:

[0033] If it is a structure node, then all its child field nodes are entered sequentially, and the processing steps of this node are recursively called for each child field node.

[0034] If it is an array node, then according to the number of array iterations defined, the process will loop through each array element node and recursively call the processing steps of this node for each element node.

[0035] If it is an atomic field node, then the atomic field parsing sub-step is executed.

[0036] Furthermore, the atomic field parsing sub-step includes:

[0037] Calculate field length: If the length or value of the atomic field depends on another field specified by dynamic reference, the field length is obtained by querying the value of the referenced field from the maintained parsing context;

[0038] Reading and conversion steps: Based on the data type defined in the field definition and the field length obtained in the previous step, read bytes of a specified length from the current position in the binary data stream and convert them according to the defined data type and byte order;

[0039] Storage association steps: The converted value is associated with the position corresponding to the field name in the standardized data frame object being constructed. On the other hand, based on the definition of the semantic keyword, it is decided whether to store it in the parsing context for subsequent dynamic reference and query.

[0040] Furthermore, the method also includes maintaining a parsing context for storing field values ​​determined during the parsing process. The parsing plan tree achieves fully dynamic parsing capability at runtime based on the determined field values. This capability includes: field length determination based on dynamic references, array iteration count determination based on dynamic references, and frame boundary synchronization based on dynamic values.

[0041] Furthermore, the step S330, which identifies and synchronizes to the beginning of the frame structure, includes: parsing a specified length field based on the frame boundary of a dynamic value, and dynamically determining the end boundary of the frame payload according to the value of the field, so as to complete the frame synchronization;

[0042] In the specific steps of traversing and processing the subtree under the frame node, the number of array iterations is determined based on the number of array iterations of the dynamically referenced array;

[0043] In the atomic field parsing sub-step, the field length is determined based on the field length of the dynamic reference.

[0044] On the other hand, based on the above method, this application also provides a data frame extraction system based on declarative description, for implementing the above method, including:

[0045] Format description interface, data stream interface, parsing engine, standardized output interface;

[0046] The format description interface is used to obtain the declarative format description file;

[0047] The data stream interface is used to provide the binary data stream to the parsing engine;

[0048] The parsing engine is connected to both the format description interface and the data stream interface, and is used to dynamically load and compile the declarative format description file at runtime to generate a parsing plan tree in memory, and extract data frames from the binary data stream by executing the parsing plan tree; wherein, the execution includes determining the length of atomic fields based on dynamic references;

[0049] The standardized output interface is used to output the standardized data frame object.

[0050] The technical solution provided by this invention has the following advantages compared with the known prior art:

[0051] The core of this invention lies in the design of a highly flexible declarative format description language (YFDL) and a corresponding general-purpose binary parsing engine (UBPE). The YFDL externalizes the structure of any binary stream, and at runtime, the general-purpose parsing engine dynamically compiles this description into an executable parsing plan tree in memory. By interpreting and executing this plan tree, it achieves streaming data frame extraction and standardized output that adapts to new data formats without code modification, realizing "configuration equals adaptation," greatly reducing system maintenance costs and shortening the response cycle to data format changes.

[0052] The Universal Binary Parsing Engine (UBPE) has the ability to dynamically load and compile description files and generate parsing plan trees during system runtime, adding support for new data formats to the system, achieving zero-code extension, breaking down the technical barriers that developers must intervene in coding in traditional solutions, significantly reducing the technical threshold and improving business agility.

[0053] High performance and high throughput are guaranteed through efficient streaming processing and a unified memory model. The parsing engine uses a depth-first traversal approach to execute the parsing plan tree in memory, processing binary data in a streaming manner without loading the entire file into memory, making it particularly suitable for handling large or real-time network data streams. Simultaneously, all parsing operations are performed under the guidance of a unified executable data structure (parsing plan tree), avoiding the performance overhead of traditional script parsing or multiple data transformations. This provides high flexibility while ensuring excellent parsing performance and processing efficiency. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0055] Figure 1 This is a flowchart illustrating a data frame extraction method based on declarative description.

[0056] Figure 2 Examples of core elements included in the YAML language;

[0057] Figure 3 This is an example of a declarative format description language (YFDL).

[0058] Figure 4 This is a schematic diagram of the parsing plan tree structure;

[0059] Figure 5This is a schematic diagram of the packaging structure.

[0060] Figure 6 A flowchart illustrating the data stream processing of S300;

[0061] Figure 7 A flowchart illustrating the sub-steps for parsing atomic fields;

[0062] Figure 8 This is a block diagram of a data frame extraction system based on declarative description. Detailed Implementation

[0063] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0064] like Figure 1 This embodiment provides a data frame extraction method based on declarative description, including the following steps:

[0065] S100. Obtain the declarative format description file: The declarative format description file is used to declare the structure of the target binary data stream;

[0066] S200, Parsing Plan Tree Generation: The general parsing engine dynamically loads the declarative format description file at runtime and compiles it into a parsing plan tree in memory; the parsing plan tree is a tree-like executable data structure, which consists of a series of internal data structure nodes corresponding to different data logic units, and the encapsulation level of data is represented by the parent-child relationship between nodes.

[0067] The common parsing engine is UBPE;

[0068] The data structure nodes in the parsing plan tree must include at least: file nodes, channel nodes, and frame nodes;

[0069] S300, Data Stream Processing: When a binary data stream is received, the general parsing engine executes the parsing plan tree in a depth-first traversal manner, recursively processes the structure and array nodes, and uniformly processes the atomic field nodes;

[0070] Among them, data frames are extracted from the binary data stream in a streaming manner;

[0071] Processing atomic field nodes includes support for determining field length based on dynamic references;

[0072] S400, Standardized Output: The extracted data frames are encapsulated into standardized data frame objects and output. Outputting standardized data objects (UFOs) greatly simplifies the upper-layer application and analysis process. Regardless of the diverse formats of the underlying binary data sources, this invention ultimately encapsulates all extracted network data frames into standardized data frame objects (UFOs) with a unified structure (including standardized timestamps, length, tags, payload, etc.) for output. This provides a stable and consistent data interface for upper-layer applications such as data analysis, visualization, intrusion detection, and performance monitoring. Application developers no longer need to worry about the diversity of data sources and can focus on core business logic, effectively improving the efficiency and reliability of the entire data processing chain.

[0073] This embodiment first defines the structure of any binary data stream and dynamically compiles this description into an executable parsing plan tree at runtime, fundamentally changing the traditional hard-coded parsing mode. When the data format changes or new formats need to be supported, only the description file needs to be modified or added; there is no need to modify, recompile, or deploy the parsing engine code, achieving "configuration-as-adaptation." This greatly reduces system maintenance costs, shortens the response cycle to data format changes, and overcomes the drawbacks of existing technologies caused by the strong coupling between format and code.

[0074] The target binary data stream consists of binary data files from various network devices, embedded systems, industrial control systems (ICS), Internet of Things (IoT) devices, or proprietary software logs. These files typically end with custom suffixes such as .dat, .log, .bin, and .trace. The structural information of the binary files (such as field types, lengths, offsets, byte order, etc.) is directly embedded in the parser's source code in a hard-coded form, usually manifested as C / C++ structure definitions or Python's struct.unpack() format strings. When any binary format changes, the parser must be modified, recompiled, and deployed accordingly, making configuration-level dynamic adaptation impossible. To address the strong coupling between the parsing logic and the data format description, this approach is necessary.

[0075] In another embodiment, the declarative format description file is defined based on YAML syntax and is used to declare the structure of the target binary data stream, which is a raw data stream with a non-standard encapsulation structure generated by a network device or embedded system.

[0076] The Declarative Format Description Language (YFDL) defines a rich set of syntactic elements to characterize diverse binary data structures. Its syntactic scope covers, but is not limited to: basic data types (such as fixed-length and variable-length integers, floating-point numbers, byte arrays, and strings), complex structures (such as structs, arrays, and optional fields), control and semantic keywords (such as `endianness` for identifying byte order, `magic` for frame synchronization, and `frame_payload` for marking the start of the payload), and key reference statements for enabling dynamic parsing (such as `{ref: "target field name"}`). Furthermore, the language supports advanced features, including bit-level field parsing, conditional branching paths, data validation assertions, and declarations of custom data encodings.

[0077] The core elements of this language include, Figure 2 As shown, a complete YFDL description file example is as follows: Figure 3 As shown, at least for declarations:

[0078] Atomic field declarations include an unsigned integer type for representing the length of a network data frame, an integer type for representing a timestamp, and a byte array type for representing the frame payload;

[0079] Structure declarations are used to define composite data structures through an ordered sequence of fields, where each field defines its type by referencing a basic data type or by nesting other structures / arrays;

[0080] Array declarations are used to define the repetition of a struct.

[0081] Domain-specific keyword declarations, including at least frame_payload for identifying the start of the frame payload, frame_cap_len for declaring the actual capture length, and frame_orig_len for declaring the original network length;

[0082] Dynamic reference declarations are used to make the parsed parameters of a field dependent on the value of a previously parsed field through a reference statement. The format of the reference statement is {ref: "target field name"}.

[0083] This embodiment utilizes a dynamic referencing mechanism to achieve accurate parsing of complex and variable data structures. The declarative description language supports dynamic referencing statements such as `{ref: "target field name"}`, allowing key parsing parameters such as field length, array iteration count, and even frame boundaries to be dynamically determined based on the values ​​of previously parsed fields. This mechanism fundamentally solves the problem of accurately parsing complex scenarios common in binary formats, such as "TLV" (type-length-value) structures, variable-length arrays, and nested dependencies, greatly enhancing the solution's ability to characterize and parse complex, non-standard data formats in the real world.

[0084] YFDL uses a general-purpose parsing engine to dynamically load declarative format description files at runtime and compile them into a parsing plan tree in memory. The parsing plan tree is a tree structure consisting of a series of internal executable data structures and the relationships between them.

[0085] Parse the plan tree structure as follows Figure 4 As shown in the diagram. The file node represents a file node, the channel node represents a channel node, and the frame node represents a frame node. A file node corresponds to the logical root container of the entire binary data file or data stream; a frame node corresponds to the parsing template of a complete network data frame; a file node can directly contain one or more frame nodes; a channel node is used to represent multiplexed or logically separated data sub-streams; a file node can contain one or more channel nodes, and each channel node contains one or more frame nodes.

[0086] A channel is a concept used to process multiplexed or logically separated data streams. A file structure can have multiple channel structures, and a channel structure can have multiple frame structures. For example... Figure 5 As shown, the encapsulation structure can be determined by YFDL. If there is no channel data, such as the entire file being a single channel, then it can be directly encapsulated by a container (filecontainer).

[0087] The parsing plan tree is a runtime dynamic blueprint for the General Binary Parsing Engine (UBPE) to perform data extraction tasks. It is generated in memory after being loaded and compiled using the YFDL declarative description language. It precisely maps the inherent logic and physical layout of the target binary data stream in a hierarchical executable node structure. The General Parsing Engine employs streaming processing algorithms to process large binary streams efficiently in real time. The S300 steps are as follows: Figure 6 As shown, it specifically includes the following stages:

[0088] S310, Initialization and Root Node Processing Phase: Starting from the file node of the parsing plan tree, read the global attributes defined therein to initialize the parsing context, and place the read pointer of the binary data stream at the starting position.

[0089] In this embodiment, the starting point of execution is the root node of the parsing plan tree, namely the file node;

[0090] Context initialization: UBPE first reads the global attributes defined in the file node, such as the endianness of the entire data stream, synchronization bytes, and tag information. This information will serve as the initial context for the parsing engine and will be effective throughout the parsing process.

[0091] Data stream access: The general binary parsing engine binds the target binary data file or data stream to the current execution session and places the internal read pointer at the beginning of the data stream (offset of 0).

[0092] S320, Optional Channel Processing Stage: If a channel node exists in the parsing plan tree, it is identified and synchronized to the channel structure in the data stream; otherwise, it directly enters the frame processing stage.

[0093] In this embodiment, after processing the root node, the engine will check whether there are any channel nodes under it;

[0094] Multi-channel processing: If a channel node exists, the engine will identify and synchronize the channel structure data according to the rules defined by the channel. This is usually achieved by searching for a synchronization header (magic) consisting of one or more bytes in the data stream. Of course, the identification and synchronization of the channel structure data can also be achieved by the length.

[0095] Single-channel (no-channel) processing: If the data format does not contain a logical channel, the channel structure does not need to be defined in YFDL. In this case, the general binary parsing engine will skip this stage and directly use the frame structure sequence under the file node as the main processing flow.

[0096] S330, Frame Processing Stage: Based on the dynamic frame length, identify and synchronize the frame to the beginning of the frame structure in the data stream, and then traverse and process the subtree under the frame node to extract and encapsulate the frame data.

[0097] A frame node defines a complete and meaningful unit of data and serves as a template for generating a Universal Frame Object.

[0098] Frame synchronization: Dynamic frame length is typically used to indicate the length of the frame payload. Once the length is determined, the engine considers it to have found the start and end of a frame. Simultaneously, the frame timestamp can be obtained using a high-low byte calculation method.

[0099] This embodiment combines the parsing context and executes the parsing plan tree, which can extract data frames in real time and calculate and parse relevant information, such as file-level or channel-level tags, frame numbers, frame timestamps, frame labels, etc., and encapsulate them into UFO objects.

[0100] In another embodiment, the specific steps for traversing and processing the subtree under the frame node in step S330 are as follows: Figure 7 As shown, it includes:

[0101] Iterate through the list of field definitions under the current node in sequence; for each field, perform the corresponding parsing operation based on its node type:

[0102] If it is a structure node, then all its child field nodes are entered sequentially, and the processing steps of this node are recursively called for each child field node.

[0103] In the parsing plan tree, a struct node is represented as an ordered set of child nodes (fields). The engine processes it as a strict serialization process.

[0104] Sequential traversal: The general binary parsing engine visits each sub-field node under the struct node one by one according to the field order defined in YFDL.

[0105] If it is an array node, then according to the number of array iterations defined, the process will loop through each array element node and recursively call the processing steps of this node for each element node.

[0106] Array nodes define recurring data patterns, which general binary parsing engines process in a loop. First, the number of iterations is determined; the number of iterations for an array is determined by its definition.

[0107] Fixed number of times: Specify a constant directly in YFDL (e.g., count: 10).

[0108] Dynamic iteration count: The number of iterations is dynamically determined by the value of a previously parsed field. For example, count: header.element_count. The general binary parsing engine will backtrack to find the value of header.element_count in the memory image of the current data frame object and use it as the loop count.

[0109] Termination condition: The loop can continue to execute until the read pointer reaches the boundary of the current frame, which is determined by the length definition of the parent frame node.

[0110] If it is an atomic field node, then the atomic field parsing sub-step is executed.

[0111] For atomic field parsing sub-steps such as Figure 7 As shown, it includes:

[0112] Calculate field length: For each field node of an atomic type (such as uint8, int32, float, double, string, etc.), if the length or value of the atomic field depends on another field specified by dynamic reference, the field length is obtained by querying the value of the referenced field from the maintained parsing context.

[0113] Reading and conversion steps: Based on the data type defined in the field definition and the field length obtained in the previous step, read bytes of a specified length from the current position of the binary data stream and convert them according to the defined data type and byte order; specifically, based on the globally or locally defined byte order, convert the read binary data block into the corresponding numeric, string or other type.

[0114] The storage association step involves associating the transformed value with the corresponding field name in the standardized data frame object being constructed, and then deciding whether to store it in the parsing context for subsequent dynamic reference and query based on the definition of the semantic keywords.

[0115] Alternatively, the converted value can be associated with the field's name and stored in the memory image of the generic data frame object being constructed.

[0116] Pointer advance: Moves the read pointer forward by the corresponding number of bytes to prepare for parsing the next field.

[0117] Nested structure processing: If a field itself is a struct, the general binary parsing engine will recursively enter the sub-struct node and repeat the above serialization process.

[0118] To achieve fully dynamic parsing capability, in another embodiment, the method further includes maintaining a parsing context for storing field values ​​determined during the parsing process. The parsing plan tree implements fully dynamic parsing capability at runtime based on the determined field values. This capability includes: field length determination based on dynamic references, array iteration count determination based on dynamic references, and frame boundary synchronization based on dynamic values.

[0119] Step S330 identifies and synchronizes to the beginning of the frame structure, including: parsing a specified length field based on the frame boundary synchronously with dynamic values, and dynamically determining the end boundary of the frame payload according to the value of the field to complete frame synchronization;

[0120] In the specific steps of traversing and processing the subtree under this frame node, the number of array iterations is determined based on the number of iterations of the dynamically referenced array;

[0121] In the atomic field parsing sub-step, the field length is determined based on the field length of the dynamic reference.

[0122] The following embodiments use a simplified network data acquisition file format as an example to illustrate in detail how the present invention achieves fully dynamic parsing capability by maintaining a parsing context, specifically including determining field length based on dynamic references, determining array iteration counts, and synchronizing frame boundaries.

[0123] Assume the target binary file format contains a series of data frames, each with the following structure (in little-endian byte order):

[0124] frame_header:

[0125] magic (2 bytes, unsigned integer): fixed at 0x5A5A, used for frame synchronization;

[0126] payload_length (2 bytes, unsigned integer): Represents the actual length in bytes of the following payload_data field;

[0127] item_count (1 byte, unsigned integer): Represents the number of data items contained in payload_data;

[0128] payload_data (frame payload): An array consisting of item_count data items.

[0129] Each data item has the following structure: item_id (2 bytes, unsigned integer);

[0130] item_value_length (1 byte, unsigned integer): Represents the length in bytes of the immediately following item_value field;

[0131] item_value (variable-length byte array): The length is item_value_length bytes.

[0132] The corresponding dynamic resolution process is as follows:

[0133] a) The general engine begins parsing the file, creating an empty parsing context (e.g., a key-value pair mapping table). The context is used to store the parsed field values ​​that have been marked as referential.

[0134] b) Parse the frame_header of the first frame.

[0135] The engine searches for the magic value 0x5A5A in the data stream to synchronize with the start of the frame.

[0136] Sequential parsing of the frame_header structure:

[0137] The magic code has been analyzed and verified.

[0138] Parse payload_length (assuming its value is 100). After parsing, the engine stores the key-value pair "frame_header.payload_length": 100 in the parsing context.

[0139] Parse item_count (let's assume it's 5). After parsing, the engine stores the key-value pair "frame_header.item_count": 5 in the parsing context.

[0140] c) Dynamically determine and parse payload_data (based on the field length of dynamic references)

[0141] The engine is preparing to parse the payload_data field. According to the YFDL description, its size attribute is {ref: "frame_header.payload_length"}.

[0142] The engine queries and parses the context, searches for the key "frame_header.payload_length", and obtains the value 100.

[0143] Based on this, the engine dynamically determines that 100 bytes need to be read from the current stream position as the raw byte block of payload_data. At this time, the engine only treats it as raw bytes and does not perform internal structured parsing.

[0144] d) Dynamically iterate and parse the `parsed_items` array (the number of iterations is determined based on the array's dynamic references).

[0145] The engine is preparing to parse the parsed_items array. According to the YFDL description, its count attribute is {ref: "frame_header.item_count"}.

[0146] The engine queries and parses the context, searches for the key "frame_header.item_count", and obtains the value 5.

[0147] Based on this, the engine dynamically determines that it needs to parse 5 item structures in a loop.

[0148] For each item (e.g., the first one): parse the item_id.

[0149] The `item_value_length` is parsed (let's say it's 25). After parsing, the engine stores the key-value pair "item_value_length": 25 in the local parsing context of the current frame (or manages it through scope).

[0150] The `item_value` field is parsed. Its `size` attribute is `{ref: "item_value_length"}`. The engine queries the parsing context (within the current scope) and obtains the value 25, thus dynamically determining to read 25 bytes as `item_value`.

[0151] Complete the parsing of one item, and repeat the process until it is completed 5 times.

[0152] e) Frame boundary synchronization completed and context reset (frame boundary synchronization based on dynamic values)

[0153] In step c), the payload_length=100 obtained by querying the context not only determines the read length of payload_data, but also implicitly determines the end boundary of the entire frame. Because payload_data is the last field of the frame in this format definition, after reading 100 bytes, the engine's read pointer reaches the end of the frame, aligning with the start position of the magic of the next frame, thus achieving precise frame boundary synchronization.

[0154] Once the frame is parsed, a standardized Data Frame Object (UFO) is generated. The parsing context clears or resets the content in the relevant scope for the parsing of the next frame, ensuring that dynamic references do not incorrectly reference other frames.

[0155] In this embodiment, the parsing context is used as a "memory unit": the values ​​of key fields (such as payload_length, item_count) are continuously stored during the parsing process.

[0156] Dynamic references serve as "query instructions": in subsequent field descriptions (such as size, count), the {ref:...} statement points to the key in the context, enabling value passing.

[0157] Implementation of fully dynamic parsing capability:

[0158] Field lengths are dynamically determined: the lengths of payload_data and each item_value are determined by the values ​​of the preceding fields.

[0159] The number of array iterations is dynamically determined: the number of times the parsed_items array is parsed is determined by the value of the preceding item_count field.

[0160] Frame boundary dynamic synchronization: Automatic and accurate inter-frame synchronization is achieved by reading a fixed-length payload based on payload_length.

[0161] This mechanism greatly enhances flexibility: it can adapt to various binary formats with complex dependencies between fields simply by modifying the reference relationships in the YFDL description, without having to rewrite the parsing logic code.

[0162] On the other hand, based on the above method, this application also provides a data frame extraction system based on declarative description, for implementing the above method, including:

[0163] Format description interface, data stream interface, parsing engine, standardized output interface;

[0164] The format description interface is used to obtain a declarative format description file;

[0165] The data stream interface is used to provide binary data streams to the parsing engine;

[0166] The parsing engine connects to both the format description interface and the data stream interface. It is used to dynamically load and compile the declarative format description file at runtime to generate a parsing plan tree in memory, and to extract data frames from the binary data stream by executing the parsing plan tree. The execution includes determining the length of atomic fields based on dynamic references.

[0167] The standardized output interface is used to output standardized data frame objects.

[0168] like Figure 8 As shown, the parsing engine includes a description file parsing module, a parsing plan module, a data stream processor, and a frame extractor and wrapper.

[0169] A description file parser is used to dynamically load and parse the declarative format description file to generate a structured format definition;

[0170] The parsing plan module is used to compile the structured format definition into a parsing plan tree in memory. The parsing plan tree is a tree-like executable data structure, which consists of a series of internal data structure nodes corresponding to different data logic units. The data encapsulation hierarchy is represented by the parent-child relationship between the nodes.

[0171] A data stream processor is used to read binary data streams input from a data stream access unit in a streaming manner and provide precise byte blocks as requested.

[0172] A frame extractor and wrapper is used to execute the parsing plan tree in a depth-first traversal manner, recursively process structure and array nodes, and uniformly process atomic field nodes to extract data frames from the binary data stream in a streaming manner; wherein, processing atomic field nodes includes supporting the determination of field length based on dynamic references.

[0173] This embodiment, through architectural innovation, systematically solves the core pain points of existing binary data parsing technologies, such as high development and maintenance costs, poor dynamic adaptability, high technical threshold, and difficulty in balancing processing performance and flexibility. It provides a highly efficient, flexible, easy-to-use, and powerful general data frame extraction solution for fields such as network data analysis, protocol reverse engineering, and security monitoring.

[0174] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0175] The above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A data frame extraction method based on declarative description, comprising the following steps: S100. Obtain the declarative format description file: The declarative format description file is used to declare the structure of the target binary data stream; S200, Parsing Plan Tree Generation: The general parsing engine dynamically loads the declarative format description file at runtime and compiles it into a parsing plan tree in memory; the parsing plan tree is a tree-like executable data structure, which consists of a series of internal data structure nodes corresponding to different data logic units, and the data encapsulation hierarchy is represented by the parent-child relationship between nodes. The general parsing engine is the general binary parsing engine UBPE; The data structure nodes in the parsing plan tree include at least: file nodes, channel nodes, and frame nodes; S300, Data Stream Processing: Upon receiving the binary data stream, the general parsing engine executes the parsing plan tree in a depth-first traversal manner, recursively processes the structure and array nodes, and uniformly processes the atomic field nodes; Specifically, data frames are extracted from the binary data stream using a streaming method. The processing of atomic field nodes includes support for determining field length based on dynamic references; S400, Standardized Output: Encapsulates the extracted data frames into standardized data frame objects and outputs them.

2. The data frame extraction method based on declarative description according to claim 1, characterized in that, The declarative format description file is defined based on YAML syntax and is used to declare the structure of the target binary data stream, which is a raw data stream with a non-standard encapsulation structure generated by a network device or embedded system.

3. The data frame extraction method based on declarative description according to claim 2, characterized in that, The declarative format description file is used at least to declare: Atomic field declarations include an unsigned integer type for representing the length of a network data frame, an integer type for representing a timestamp, and a byte array type for representing the frame payload; A structure declaration is used to define a composite data structure through an ordered sequence of fields, where each field defines its type by referencing the underlying data type or by nesting other structures / arrays; An array declaration is used to define the repetition of the structure. Domain-specific keyword declarations, the keywords including at least frame_payload for identifying the start of the frame payload, frame_cap_len for declaring the actual capture length, and frame_orig_len for declaring the original network length; A dynamic reference declaration is used to make the parsed parameters of a field dependent on the value of a previously parsed field through a reference statement. The format of the reference statement is {ref: "target field name"}.

4. The data frame extraction method based on declarative description according to claim 1, characterized in that, The file node corresponds to the logical root container of the entire binary data file or data stream; The frame node corresponds to the parsing template of a complete network data frame; The channel node is used to represent multiplexed or logically separated data sub-streams; wherein, the file node may contain one or more of the channel nodes, and each channel node contains one or more of the frame nodes.

5. The data frame extraction method based on declarative description according to claim 1, characterized in that, The S300 step specifically includes the following stages: S310, Initialization and Root Node Processing Phase: Starting from the file node of the parsing plan tree, read the global attributes defined therein to initialize the parsing context, and place the read pointer of the binary data stream at the starting position; S320, Optional Channel Processing Stage: If a channel node exists in the parsing plan tree, it is identified and synchronized to the channel structure in the data stream; otherwise, it directly enters the frame processing stage. S330, Frame Processing Stage: Based on the dynamic frame length, identify and synchronize the frame to the beginning of the frame structure in the data stream, and then traverse and process the subtree under the frame node to extract and encapsulate the frame data.

6. The data frame extraction method based on declarative description according to claim 5, characterized in that, The specific steps for traversing and processing the subtree under the frame node in step S330 include: Iterate through the list of field definitions under the current node in sequence; for each field, perform the corresponding parsing operation based on its node type: If it is a structure node, then all its child field nodes are entered sequentially, and the processing steps of this node are recursively called for each child field node. If it is an array node, then according to the number of array iterations defined, the process will loop through each array element node and recursively call the processing steps of this node for each element node. If it is an atomic field node, then the atomic field parsing sub-step is executed.

7. The data frame extraction method based on declarative description according to claim 6, characterized in that, The atomic field parsing sub-step includes: Calculate field length: If the length or value of the atomic field depends on another field specified by dynamic reference, the field length is obtained by querying the value of the referenced field from the maintained parsing context; Reading and conversion steps: Based on the data type defined in the field definition and the field length obtained in the previous step, read bytes of a specified length from the current position in the binary data stream and convert them according to the defined data type and byte order; Storage association steps: The converted value is associated with the position corresponding to the field name in the standardized data frame object being constructed. On the other hand, based on the definition of the semantic keyword, it is decided whether to store it in the parsing context for subsequent dynamic reference and query.

8. The data frame extraction method based on declarative description according to claim 7, characterized in that, The method also includes maintaining a parsing context for storing field values ​​determined during the parsing process. The parsing plan tree achieves fully dynamic parsing capability at runtime based on the determined field values. This capability includes: field length determination based on dynamic references, array iteration count determination based on dynamic references, and frame boundary synchronization based on dynamic values.

9. The data frame extraction method based on declarative description according to claim 8, characterized in that, The step S330, which identifies and synchronizes to the beginning of the frame structure, includes: parsing a specified length field based on the frame boundary of a dynamic value, and dynamically determining the end boundary of the frame payload according to the value of the field, so as to complete the frame synchronization. In the specific steps of traversing and processing the subtree under the frame node, the number of array iterations is determined based on the number of array iterations of the dynamically referenced array; In the atomic field parsing sub-step, the field length is determined based on the field length of the dynamic reference.

10. A data frame extraction system based on declarative description, used to implement the method as described in any one of claims 1-9, characterized in that, include: Format description interface, data stream interface, parsing engine, standardized output interface; The format description interface is used to obtain the declarative format description file; The data stream interface is used to provide the binary data stream to the parsing engine; The parsing engine is connected to both the format description interface and the data stream interface, and is used to dynamically load and compile the declarative format description file at runtime to generate a parsing plan tree in memory, and extract data frames from the binary data stream by executing the parsing plan tree; wherein, the execution includes determining the length of atomic fields based on dynamic references; The standardized output interface is used to output the standardized data frame object.

Images