A unified declarative mapping and conversion method and system for heterogeneous data formats and their internal structures

Abstract

The application discloses a kind of unified declarative mapping conversion method and system for heterogeneous data format and its internal structure, it is related to data format conversion and data structure transformation technical field.The method is by obtaining the declarative mapping file containing format prefix and unified path expression, it is generated format-independent mapping execution plan by parsing and semantic binding, JSON object and array structure, XML element / attribute / text node, Binary structure and bit segment structure are uniformly abstracted as executable access plan and target write plan;According to mapping execution plan, the corresponding format data access adapter is called to read field value in execution phase, data write adapter writes mapping result, and conversion between heterogeneous data formats and structure transformation within the same format are completed simultaneously in the same mapping task.The application can realize the unified mapping modeling of multi-format data, and improve the consistency, maintainability and execution flexibility of data conversion.

Description

Technical Field

[0001] This invention relates to the field of data format conversion and data structure transformation technology, specifically to a unified declarative mapping conversion method and system for heterogeneous data formats and their internal structures. Background Technology

[0002] In various scenarios such as industrial control, automotive systems, IoT, device protocol processing, and business data processing, the same data content often needs to be expressed in different formats or with different internal structures. Input data may be a JSON object tree, an XML element tree, or a fixed binary protocol packet. Output data may need to be converted to another data format, or it may need to undergo structural transformations within the same format, such as adjusting field paths, changing object hierarchy, adapting array lengths, converting field types, truncating byte ranges, or adjusting bit layout. Common methods for heterogeneous data conversion in the industry include: writing a separate conversion program for each combination of input and output formats; configuring data flow links through a general ETL or integration platform; first converting various types of input data into a unified intermediate object and then having business code reorganize fields and write them into the target format; using a dedicated parser or manually writing offset read / write logic for binary protocols; and using independent path expression systems, templates, or scripts to complete the conversion process for JSON and XML. The above-mentioned solutions generally lack a unified mapping semantic system. JSON, XML and Binary data are often processed separately. Binary protocol data cannot be used as a data source and output target of the same level as JSON, XML and other structured text data in the same mapping model. At the same time, format conversion between different data formats and various structural transformations within the same data format usually need to be broken down into multiple independent execution steps. They cannot be uniformly described and driven by the same mapping rule in the same mapping task. This leads to low reusability of mapping rules, long conversion links, high development and maintenance costs, and difficulty in ensuring semantic consistency and execution efficiency in the process of multi-format conversion. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the purpose of this invention is to provide a unified declarative mapping and conversion method and system for heterogeneous data formats and their internal structures, thereby achieving unified modeling and execution of heterogeneous data format conversion and internal structure transformation, and improving the consistency, maintainability, and execution flexibility of data conversion.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] This application provides a unified declarative mapping transformation method for heterogeneous data formats and their internal structures, comprising: obtaining a declarative mapping file, wherein the declarative mapping file declares access to the structure of different data formats through a format prefix and a unified path expression, wherein the different data formats include at least one of JSON, XML, and Binary; parsing and semantically binding the declarative mapping file to generate a format-independent mapping execution plan, wherein the mapping execution plan uniformly represents the object and array structure of JSON, the element, attribute and text node structure of XML, and the structure, field offset and bit field structure of Binary as an executable access plan, expression plan and target write plan; according to the mapping execution plan, calling the data access adapter corresponding to the input source format type to read field values, and calling the data write adapter corresponding to the output target format type to write the mapping result; wherein the mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

[0006] Furthermore, the declarative mapping file includes input declarations, output declarations, model declarations, and variable declarations; the input declaration is used to declare the input source name, input source type, and optional data structure description; the output declaration is used to declare the output target name, output target type, output conditions, and output fields, with field values ​​calculated from input access paths, variables, context references, literals, or expressions; the unified path expression method includes a format prefix and a path body, the format prefix is ​​used to identify JSON, XML, Binary, or context data, and the path body is used to locate object fields, array elements, XML elements, XML attributes, XML text nodes, Binary structure fields, Binary byte ranges, or Binary bit fields.

[0007] Furthermore, the mapping execution plan includes: an input source table, recording the input source number, name, format type, and data structure description reference; a variable table, recording the variable number, name, expression, and variable value type; an output plan table, recording the output target number, name, format type, output conditions, and predicate rules; an access plan, recording field access steps; an expression plan, recording literals, input access, variable references, context references, function calls, and formula expressions; a target write plan, recording the output target path and corresponding value; and a predicate plan, recording checking, assertion, and skip rules.

[0008] Furthermore, the Binary input source or Binary output target is bound to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. When generating the mapping execution plan, the offset, field length, array step size, and bit field position of the Binary fields are calculated based on the binary structure description file, so that the Binary protocol data, as a data format with structural semantics, participates in the conversion in the same declarative mapping semantics as JSON and XML.

[0009] Furthermore, the structural transformations within the same data format include at least one of field renaming, path migration, object reorganization, array pruning, array padding, nested structure expansion, nested structure merging, field combination, type conversion, byte range truncation, or bit segment processing; the declarative mapping file simultaneously declares multiple input sources and multiple output targets in the same mapping file, the multiple input sources include at least two of JSON format, XML format, and Binary format, and the multiple output targets include at least two of JSON format, XML format, and Binary format, realizing a multi-source to multi-target combined mapping transformation in a mapping task.

[0010] Furthermore, the method includes two execution modes: Interpreted execution mode: the interpreted execution engine reads the mapped execution plan at runtime, calls different data access adapters for different input formats, and calls different data write adapters for different output formats. Expression evaluation, predicate control, and variable handling follow a unified semantics. Code generation execution mode: the code generation backend converts the mapped execution plan into C language source code. The generated C language source code includes a fixed entry function, input reading function, expression evaluation logic, predicate judgment logic, and output writing function. It does not require re-parsing of the declarative mapping file at runtime.

[0011] Furthermore, in the interpreted execution mode, the execution engine analyzes the input sources that the output target depends on based on the access plan. When an assertion for a certain input source fails, only the output target that depends on that input source is blocked, without affecting the normal execution of other output targets. In the code generation execution mode, the generated C language source code includes functions for reading binary input fields, writing binary output fields, extracting JSON input fields, extracting XML input fields, variable initialization code, expression evaluation code, predicate judgment code, and construction functions for each output target.

[0012] Accordingly, this application also provides a unified declarative mapping conversion system for heterogeneous data formats and their internal structures, comprising: a declarative mapping file acquisition module, used to acquire a declarative mapping file, wherein the declarative mapping file declares access to the structure of different data formats through a format prefix and a unified path expression, wherein the different data formats include at least one of JSON, XML, and Binary; and a parsing and semantic binding module, connected to the declarative mapping file acquisition module, used to parse and semantically bind the declarative mapping file, and generate a format-independent mapping execution plan, wherein the mapping execution plan includes JSON object and array structures, XML, and Binary structures. The elements, attributes, and text node structures of L, the structure of Binary, field offsets, and bit segment structures are uniformly represented as an executable access plan and a target write plan. The adapter execution module, connected to the parsing and semantic binding module, is used to call the data access adapter corresponding to the input source format type to read field values ​​and call the data write adapter corresponding to the output target format type to write the mapping result according to the mapping execution plan. The mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

[0013] Furthermore, the mapping execution plan generated by the parsing and semantic binding module includes an input source table, a variable table, an output plan table, an access plan, an expression plan, a target write plan, and a predicate plan; the adapter execution module includes: a data access adapter unit, including a JSON data access adapter, an XML data access adapter, and a Binary data access adapter, respectively used to read field values ​​from input sources of corresponding formats; and a data write adapter unit, including a JSON data write adapter, an XML data write adapter, and a Binary data write adapter, respectively used to write the mapping results to output targets of corresponding formats; both the data access adapter unit and the data write adapter unit are subject to unified scheduling by the mapping execution plan, and expression evaluation, predicate control, and variable processing follow unified semantics.

[0014] Furthermore, the Binary data access adapter or the Binary data writing adapter binds to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. The parsing and semantic binding module calculates the offset, field length, array step size, and bit field position of the Binary fields based on the binary structure description file, enabling the Binary protocol data to participate in the conversion as a data format with structural semantics, in the same declarative mapping semantics as JSON and XML. The system supports interpreted execution mode and code-generated execution mode. The interpreted execution mode reads the mapping execution plan and executes it dynamically at runtime, while the code-generated execution mode pre-converts the mapping execution plan into C language source code, and directly calls the generated code at runtime to complete the mapping conversion.

[0015] The beneficial effects of this invention are:

[0016] 1. It can use the same declarative mapping file to describe the conversion relationships between various heterogeneous data formats such as JSON, XML, and Binary;

[0017] 2. It can use the same declarative mapping file to describe field renaming, path migration, object reorganization, array pruning, array padding, nested structure expansion or merging, type conversion, and bit field processing within the same format;

[0018] 3. It can support multiple input sources and multiple output targets in a single mapping task, realize the combined structure transformation from multiple sources to multiple targets, and enable Binary protocol data to participate as a first-class structure object in the mapping between JSON, XML, and between Binary or between internal structures of Binary.

[0019] 4. It can uniformly express path access, target writing, expression evaluation, conditional output, checking, assertion, skipping, and variable reference;

[0020] 5. It can isolate mapping semantics and format adaptation through format-independent mapping execution plans, thereby improving scalability;

[0021] 6. It can support interpreted execution and AOT code generation execution, enabling the same structure mapping rule to have different execution forms. Attached Figure Description

[0022] Figure 1 A flowchart illustrating a unified declarative mapping and conversion method for heterogeneous data formats and their internal structures, provided for embodiments of this application;

[0023] Figure 2 This is a schematic diagram of the structure of the mapping execution plan provided in the embodiments of this application;

[0024] Figure 3 This is a schematic diagram of the structure of the binary structure description file provided in the embodiments of this application;

[0025] Figure 4 This is a schematic diagram of the execution flow of the dynamic execution data mapping and transformation provided in the embodiments of this application;

[0026] Figure 5 A schematic diagram of the C language source code provided in the embodiments of this application;

[0027] Figure 6 This is a schematic diagram of a unified declarative mapping and conversion system for heterogeneous data formats and their internal structures, provided as an embodiment of this application. Detailed Implementation

[0028] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0029] In this invention, the terms "system" and "network" are used interchangeably. "Multiple" refers to two or more; therefore, in this invention, "multiple" can also be understood as "at least two." "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be understood that in the description of this invention, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0030] Example 1:

[0031] like Figures 1-5As shown, this application provides a unified declarative mapping transformation method for heterogeneous data formats and their internal structures, including: obtaining a declarative mapping file, which declares access to the structure of different data formats through a format prefix and a unified path expression, wherein the different data formats include at least one of JSON, XML, and Binary; parsing and semantically binding the declarative mapping file to generate a format-independent mapping execution plan, wherein the mapping execution plan uniformly represents the object and array structure of JSON, the element, attribute and text node structure of XML, and the structure, field offset and bit structure of Binary as an executable access plan, expression plan and target write plan; according to the mapping execution plan, calling the data access adapter corresponding to the input source format type to read field values, and calling the data write adapter corresponding to the output target format type to write the mapping result; wherein, the mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

[0032] In another possible embodiment, the method provided in this embodiment includes the following steps: Step S101, obtaining a declarative mapping file. The declarative mapping file declares structural access to different data formats through format prefixes and uniform path expressions. The different data formats include at least one of JSON, XML, and Binary. A declarative mapping file is a configuration file written in a domain-specific language (DSL), which identifies different data sources through format prefixes and locates specific data fields through uniform path bodies. The declarative mapping file can be carried in XML, JSON, or custom text formats; Step S102, parsing and semantically binding the declarative mapping file. The system first reads the text content of the declarative mapping file and parses out the input source declaration, output target declaration, model declaration, variable declaration, and field mapping rules. Then, a semantic binding process is performed, including verifying the existence of the input source name, verifying the existence of the output target name, verifying the validity of variable references, and verifying the matching of expression parameter types. After the semantic binding is successful, the system generates a format-independent mapping execution plan. The mapping execution plan uniformly represents the object and array structures of JSON, the element, attribute and text node structures of XML, and the structure, field offsets and bit structures of Binary as executable access plans and target write plans; Step S103: For JSON format, its object structure {key:value} and array structure [element1, element2, ...] are mapped to object field access and array index access steps in the access plan; for XML format, its element nesting structure <element> ...< / element>The attribute structure (attribute=value) and text node structure are mapped to element path access, attribute path access, and text node access steps in the access plan. For the Binary format, its structure definition, field byte offsets, and bit field positions in the protocol message are mapped to offset reading and bit field extraction steps in the access plan. The three different format structure units are uniformly entered into the same access plan item in the same mapping execution plan through their respective access adapters, and are no longer processed separately by independent conversion codes for each format. Step S104: According to the mapping execution plan, the data access adapter corresponding to the input source format type is called to read field values, and the data writing adapter corresponding to the output target format type is called to write the mapping result. For JSON input sources, the system calls the JSON data access adapter to extract field values ​​from the JSON data tree according to the object field path or array index in the access plan; for XML input sources, the system calls the XML data access adapter to extract field values ​​from the XML document tree according to the element path, attribute path, or text node path in the access plan; for Binary input sources, the system calls the Binary data access adapter to read field values ​​from the binary byte stream according to the byte offset and bit field position in the access plan. Similarly, for the output target, the system calls the corresponding data writing adapter according to the target format type to write the mapping calculation result into the target data structure. The mapping transformation includes format conversion between different data formats and structural transformation within the same data format. Format conversion refers to converting data from one format (such as JSON) to another (such as XML or Binary), while structural transformation refers to operations such as field renaming, path migration, object reorganization, array pruning or padding, and nested structure expansion or merging within the same format. Furthermore, the structural transformation and format conversion are described and executed by the same mapping execution plan in the same mapping task; that is, a single declarative mapping file can simultaneously describe the rules for cross-format conversion and intra-format structural transformation, eliminating the need to maintain two separate sets of conversion logic.

[0033] The unified declarative mapping transformation method in this embodiment can describe the transformation relationships between multiple heterogeneous data formats and the structural transformation rules within the same format using a single mapping file. JSON object arrays, XML element attribute text, and Binary structure field offsets are unified into the access plan and target write plan in the mapping execution plan, eliminating semantic differences in mapping between different data formats. Compared to existing technologies that require writing separate transformation programs for each format, this method improves the maintainability and consistency of data transformation while reducing the development cost of multi-format mapping.

[0034] In embodiments of this application, the declarative mapping file includes input declarations, output declarations, model declarations, and variable declarations; the input declaration is used to declare the input source name, input source type, and optional data structure description; the output declaration is used to declare the output target name, output target type, output conditions, and output fields, with field values ​​calculated from input access paths, variables, context references, literals, or expressions; the unified path expression method includes a format prefix and a path body, the format prefix is ​​used to identify JSON, XML, Binary, or context data, and the path body is used to locate object fields, array elements, XML elements, XML attributes, XML text nodes, Binary structure fields, Binary byte ranges, or Binary bit fields.

[0035] In another possible embodiment, the declarative mapping file includes four types of declaration structures: input declarations, output declarations, model declarations, and variable declarations. The input declarations are used to declare the input source name, input source type, and optional data structure description. Their declaration syntax is as follows:

[0036] <input name="输入源名称" type="格式类型" [schema="结构描述文件路径" ]>

[0037] The `name` attribute specifies the identifier name of the input source, which is globally unique within the mapping file; the `type` attribute specifies the data format type of the input source, with values ​​including json, xml, and binary; the `schema` attribute is optional, and when `type` is binary, it is used to specify the path to the binary structure description file.

[0038] The output declaration is used to declare the output target name, output target type, output conditions, and output fields. Its declaration syntax is as follows:

[0039] <output name="输出目标名称" type="格式类型" when="输出条件表达式"> <fields> <field target="目标路径"> <value kind="值类型" expr="表达式" / > < / field> < / fields> < / output>

[0040] The `name` attribute specifies the identifier name of the output target; the `type` attribute specifies the data format type of the output target; the `when` attribute specifies the output condition, and the writing to the output target is executed when the condition is true; the `target` attribute in the `field` tag specifies the target path to be written; the `kind` attribute in the `value` tag specifies the type of the value (such as object, array, expr, struct, etc.), and the `expr` attribute specifies the expression for calculating the field value.

[0041] Field values ​​can be calculated in the following ways: (1) Input access path, i.e., reading field values ​​directly from the input source through the format prefix and path body; (2) Variable, i.e. referencing intermediate calculation results defined in the variable declaration; (3) Context reference, i.e. reading parameter values ​​from the runtime context; (4) Literal, i.e. directly using constant values; (5) Expression, i.e. calculated through function calls and operator combinations.

[0042] The unified path representation consists of two parts: a format prefix and a path body. The format prefix identifies the format type of the data source, specifically including:

[0043] $json: indicates that data is read from a JSON input source. The path body is an object path expression enclosed in curly braces, such as $json:req{ / device / id}, which means reading the id field of the device object from a JSON input source named req.

[0044] $xml: indicates that data is read from an XML input source. The path body can be an element path, an attribute path (using the @ prefix), or a text node path (using #text). For example, $xml:meta{ / routing / @trace} means reading the trace attribute under the routing element from an XML input source named meta.

[0045] $binary: Identifies data read from the Binary input source. The path body locates the structure field defined in the binary structure description file. For example, $binary:calib{ / flags} means reading the value of the flags field from the Binary input source named calib.

[0046] Context prefixes (e.g., $ctx): identify when parameter data is read from the runtime context.

[0047] The path body is used to locate at least one of the following structural units: object fields (such as JSON keys), array elements (such as the index position of a JSON array), XML elements (such as nested element paths), XML attributes (such as @attributeName), XML text nodes (such as #text), Binary structure fields (such as named fields in a structure), Binary byte ranges (such as a specified range in a raw byte stream), and Binary bit fields (such as a specific range in a byte or integer).

[0048] The following is a complete example of a declarative mapping file (using DSL v4.1 XML hosting) demonstrating a mapping declaration from multi-format input to multi-format output:

[0049] <dsl version="4.1"> <inputs> <input name="req" type="json"> <input name="meta" type="xml"> <input name="calib" type="binary" schema="calib.schema.xml"> < / inputs> <outputs> <output name="out_json" type="json" when="true"> <value kind="object"> <fields> <field target=". / payload / deviceId"> <value kind="expr" expr="$json:req{ / device / id}" / > < / field> <field target=". / payload / trace"> <value kind="expr" expr="$xml:meta{ / routing / @trace}" / > < / field> <field target=". / payload / mode"> <value kind="expr" expr="#bits($binary:calib{ / flags}, 0, 3)" / > < / field> < / fields> < / value> < / output> <output name="out_xml" type="xml" when="true"> <value kind="object"> <fields><field target=". / report / @device "> <value kind="expr" expr="$json:req{ / device / id}" / > <field target=". / report / trace / #text"> <value kind="expr" expr="$xml:meta{ / routing / @trace}" / > < / field> < / fields> < / value> < / output> <output name="out_bin" type="binary" when="true"> <value kind="struct"> <fields> <field target=". / header / device_id" type="uint32"> <value kind="expr" expr="$json:req{ / device / id}" / > < / field> <field target=". / header / mode" type="uint8"> <value kind="expr" expr="#bits($binary:calib{ / flags}, 0, 3)" / > < / fields> < / value> < / output> < / outputs> < / dsl>

[0050] In the example above, `json:req{ / device / id}` indicates reading object fields from the JSON input source `req`, `xml:meta{ / routing / @trace}` indicates reading attributes from the XML input source `meta`, and `$binary:calib{ / flags}` indicates reading fields defined by the binary structure description file from the binary input source `calib`. After semantic binding, these three elements generate access plan items respectively. Their correspondence is shown in the table below:

[0051] $json:req{ / device / id} JSON object fields Access Plan Items $xml:meta{ / routing / @trace} XML attributes Access Plan Items $binary:calib{ / flags} Binary structure fields Access Plan Items . / payload / deviceId JSON target field Write the target into the plan item . / report / @device XML target attributes Write the target into the plan item . / header / device_id Binary target structure fields Write the target into the plan item

[0052] The declarative mapping file structure and unified path expression method in this embodiment enable the description of access to multiple data sources such as JSON, XML, Binary, and context using a unified format prefix and path body syntax, eliminating the differences in path expression methods between different formats. The declarative mapping file, through four types of declaration structures—input declaration, output declaration, model declaration, and variable declaration—fully expresses the semantics of the mapping task, making the mapping rules readable, maintainable, and verifiable, and allowing the description of complex multi-format mapping relationships without relying on hand-written code.

[0053] In embodiments of this application, the mapping execution plan includes: an input source table, recording input source numbers, names, format types, and data structure description references; a variable table, recording variable numbers, names, expressions, and variable value types; an output plan table, recording output target numbers, names, format types, output conditions, and predicate rules; an access plan, recording field access steps; an expression plan, recording literals, input access, variable references, context references, function calls, and formula expressions; a target write plan, recording output target paths and corresponding values; and a predicate plan, recording checking, assertion, and skip rules. The mapping execution plan does not depend on specific input data instances but rather describes the executable semantics of the mapping task.

[0054] In another possible embodiment, the mapping execution plan includes the following seven types of plan structures:

[0055] (1) Input source table: Records the input source number, name, format type and data structure description reference. Each input source occupies one row in the input source table, recording its unique number (for quick location during subsequent execution), input source name (corresponding to the name attribute in the declarative mapping file), format type (json / xml / binary) and optional schema reference (such as the binary structure description file path).

[0056] (2) Variable table: Records variable number, name, expression, and variable value type. Variables are used to store intermediate calculation results during the mapping process to avoid duplicate calculations. Each variable records its unique number, variable name, expression for calculating the variable value (which can be accessed by input, literal, function call, etc.), and the data type of the variable value.

[0057] (3) Output plan table: Record the output target number, name, format type, output conditions and predicate rules. Each output target occupies one line in the output plan table, recording its unique number, output target name, target format type, output condition expression (corresponding to the when attribute in the declarative mapping file) and predicate rules (including assertions, checks and skip conditions).

[0058] (4) Access Plan: Records the field access steps. Each access step describes how to read a specific field value from a specific input source. Types of access steps include: object field access (for JSON format), array index access (for JSON arrays), element path access (for XML format), attribute path access (for XML attributes), text node access (for XML text nodes), offset reading (for Binary format), and bit field extraction (for Binary bit fields).

[0059] (5) Expression Plan: Records literals, input accesses, variable references, context references, function calls, and formula expressions. The expression plan describes the calculation logic of field values ​​and supports nested combinations. For example, an expression can first reference the input access result and then perform type conversion or bit field extraction through a function call.

[0060] (6) Target Write Plan: Records the output target path and corresponding value. Each write step describes where the calculation result will be written to the target data structure. The format of the target path depends on the data format of the output target: JSON target path is an object field path or array element position, XML target path is an element path, attribute path or text node path, and Binary target path is a structure field position, byte range or bit range position.

[0061] (7) Predicate Plan: Records check, assertion, and skip rules. Check rules are used to verify the validity of input data, assertion rules are used to block specific outputs when conditions are not met, and skip rules are used to skip writing specific fields when conditions are met.

[0062] The following declarative mapping file serves as an example to illustrate the process of generating a mapping execution plan:

[0063] After parsing the above DSL example, the system generates the following mapping execution plan:

[0064] Input source table:

[0065] 0 req json - 1 meta xml - 2 calib binary calib.schema.xml

[0066] Variable table: There are no variable declarations in this example, so it is an empty table.

[0067] Output schedule:

[0068] 0 out_json json true 1 out_xml xml true 2 out_bin binary true

[0069] The access plan includes the following access steps:

[0070] Step A0: Access the path / device / id from input source 0 (req) → access the plan item (JSON object field);

[0071] Step A1: Access the path / routing / @trace from input source 1 (meta) → access the plan item (XML attribute);

[0072] Step A2: Access path / flags from input source 2 (calib) → access plan item (Binary structure field).

[0073] The target write plan includes the following write steps:

[0074] W0: Write the result of step A0 to the output target 0's . / payload / deviceId → target write to plan item;

[0075] W1: Write the result of step A1 to the output target 0's . / payload / trace → target to plan item;

[0076] W2: Write the result of step A2 processed by the #bits function into the . / payload / mode → target write plan item of output target 0;

[0077] W3: Write the result of step A0 to output target 1's . / report / @device → target write to plan item;

[0078] W4: Write the result of step A1 to output target 1 in . / report / trace / #text → Write the target to the plan item;

[0079] W5: Write the result of step A0 to output target 2's . / header / device_id (type=uint32) → Write the target to the plan item;

[0080] W6: Write the result of step A2 processed by the #bits function to output target 2's . / header / mode (type=uint8) → target is written to the plan item.

[0081] The mapping execution plan does not depend on specific input data instances, but rather describes the executable semantics of the mapping task. That is, the mapping execution plan is generated solely based on the rule definitions in the declarative mapping file, without involving the actual input data content. When the actual input data arrives, the execution engine executes the steps described in the mapping execution plan sequentially, completing data reading, expression calculation, and target writing.

[0082] The mapping execution plan generation method in this embodiment compiles the mapping rules in the declarative mapping file into seven types of structured execution plans, achieving complete separation of mapping semantics and format adaptation. The mapping execution plan is independent of specific input data instances; it can be pre-generated and cached after the mapping rules are defined, and executed directly according to the plan at runtime, improving execution efficiency. Furthermore, the format-independent mapping execution plan means that adding new data formats only requires adding corresponding access and write adapters, without changing the structure and semantics of the execution plan, significantly improving the system's scalability.

[0083] In embodiments of this application, the Binary input source or Binary output target is bound to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. When generating the mapping execution plan, the offset, field length, array step size, and bit field position of the Binary fields are calculated based on the binary structure description file, so that the Binary protocol data, as a data format with structural semantics, participates in the conversion in the same declarative mapping semantics as JSON and XML.

[0084] In another possible embodiment, when the input source or output target is in binary format, the system reads a binary structure description file. The binary structure description file includes:

[0085] (1) Root structure: Defines the top-level data structure of the binary protocol;

[0086] (2) Field Name: Defines the identifier name for each field;

[0087] (3) Field type: Define the data type of each field, such as uint8, uint16, uint32, int8, int16, int32, float, etc.;

[0088] (4) Field order: Defines the order in which fields are arranged in the binary message. The field offset is calculated sequentially according to the field order and field type.

[0089] (5) Array length: Defines the number of elements in a fixed-length array;

[0090] (6) Byte length: Defines the number of bytes in a byte array or character array;

[0091] (7) Bit field information: Defines the bit width and bit offset of a specific range in a field.

[0092] The following is an example of a binary structure description file using Binary Schema v1.1:

[0093] <binary-schema version="1.1" endian="little" alignment="1" root="Packet"> <structures> <structure name="Header"> <fields> <field name="device_id" type="uint32" / > <field name="mode" type="uint8" / > < / fields> < / structure> <structure name="Point"> <fields> <field name="x" type="uint16" / > <field name="y" type="uint16" / > < / fields> < / structure> <structure name="Packet"> <fields> <field name="header" type="Header" / > <field name="points" type="Point[3]" / > <field name="payload" type="uint8

[16] " / > < / fields> < / structure> < / structures> < / binary-schema>

[0094] In the example above, the binary structure description file defines three structures: the Header structure contains a uint32 type device_id field (4 bytes) and a uint8 type mode field (1 byte); the Point structure contains two uint16 type fields x and y (2 bytes each); and the Packet, as the root structure, contains a Header type header field, a Point[3] type points array (3 Point structures), and a uint8

[16] type payload byte array.

[0095] When generating the mapping execution plan, the byte offset, field size, array length, and array stride are calculated based on the binary structure description file. Taking the Packet structure above as an example, under the conditions of little-endian and alignment=1 (no alignment padding):

[0096] header field: offset = 0, size = 5 bytes (4 bytes device_id + 1 byte mode)

[0097] The `points` field has the following parameters: offset = 5, single `Point` structure size = 4 bytes (2 bytes x + 2 bytes y), array length = 3, array step size = 4 bytes, and total size = 12 bytes.

[0098] payload field: offset=17, type=uint8

[16] , size=16 bytes

[0099] The total size of the Packet structure is 33 bytes.

[0100] For binary input, field values ​​can be read based on offsets during execution: for example, reading header.device. — When dealing with the `id`, read 4 bytes from byte offset 0 and parse them into a `uint32` integer value according to little-endian byte order. For binary output, field values ​​can be written based on the offset during execution: for example, writing `header.device`.— When setting id, the uint32 value is written to a 4-byte space starting at byte offset 0 in little-endian byte order.

[0101] In bit-field processing scenarios, the binary structure description file can first declare the bytes or integers carrying the bit fields as ordinary fields, such as uint8 or uint16; the field value rules in the declarative mapping file are then used to perform bit reads or bit writes through bit manipulation functions. Specifically, the system provides the following bit-field processing functions:

[0102] #bits(source,startBit,endBit): Extracts the integer value represented by bits startBit to endBit from source. For example, #bits($binary:calib{ / flags}, 0, 3) extracts the integer value represented by bits 0 to 3 (i.e., the lower 4 bits) from the Binary input field flags.

[0103] #bit(source, bitIndex): Extracts the value (0 or 1) of the bit-th bit from source.

[0104] #withBits(target,startBit,endBit,value): Writes value to bits startBit through endBit of target, leaving other bits unchanged;

[0105] #withBit(target,bitIndex,value): Sets the bit index of target to value.

[0106] In byte range truncation scenarios, declarative mapping files can manipulate the raw byte stream of Binary using the following functions:

[0107] #binaryRange(sourceName, startOffset, endOffset): Extracts the byte range from startOffset to endOffset from the raw byte stream of the Binary input source;

[0108] #sliceBytes(byteValue,start,length): Slices the byte value, extracting length bytes starting from the start position.

[0109] For example, the following declaration extracts and slices a range of bytes from a binary input source.

[0110] <field target=". / segment" type="bytes[2]"> <value kind="expr" expr="#sliceBytes(#binaryRange("packet"”, 1, 5), 1, 3)" / > < / field>

[0111] Therefore, the Binary protocol is no longer just a plain byte stream, but a data format with structural semantics, participating in transformations alongside JSON and XML in a unified mapping semantic layer. This mechanism can be used for format conversion between JSON, XML, and Binary, as well as for protocol structure transformations within the Binary format, such as field rearrangement, byte fragment truncation, bit field reading, and bit field writing.

[0112] Through the binary structure description file-driven mechanism of this embodiment, binary protocol data can participate in unified mapping transformation as a first-class data object with structural semantics, eliminating the semantic gap between binary protocol data and structured text data. When generating the mapping execution plan, field offsets, field lengths, array steps, and bit positions are automatically calculated based on the binary structure description file, eliminating the need for developers to manually write offset read / write logic, thus reducing the development difficulty and error probability of binary protocol processing. Simultaneously, binary data participates in the transformation with JSON and XML data within the same declarative mapping semantics, enabling cross-format combination mapping (such as JSON to Binary, Binary to XML, etc.) without requiring additional independent protocol parsing code.

[0113] In the embodiments of this application, the structural transformation within the same data format includes at least one of field renaming, path migration, object reorganization, array pruning, array padding, nested structure expansion, nested structure merging, field combination, type conversion, byte range truncation, or bit segment processing; the declarative mapping file declares multiple input sources and multiple output targets simultaneously in the same mapping file, the multiple input sources include at least two of JSON format, XML format, and Binary format, and the multiple output targets include at least two of JSON format, XML format, and Binary format, thereby realizing a multi-source to multi-target combined mapping transformation in one mapping task.

[0114] In another possible embodiment, the structural transformation within the same data format includes at least one of the following types:

[0115] (1) Field renaming: Write the input field to the output target with a different name. For example, rename the field "device_id" in the JSON input to the field "deviceId" in the JSON output, which can be achieved by using different access paths and target paths in the declarative mapping file.

[0116] (2) Path migration: Migrate input fields from the original object level to different levels in the output target. For example, migrate the field located at . / device / id in the JSON input to the position located at . / payload / header / deviceId in the JSON output to reorganize the data hierarchy.

[0117] (3) Object reorganization: Combine multiple fields from different input sources into the same new object in the output target. For example, read the device number from JSON input, the tracking identifier from XML input, and the device status from Binary input, and combine the three into the . / device object in the JSON output.

[0118] (4) Array pruning: When the number of elements in the source array exceeds the capacity of the target array, the excess elements are pruned according to the target capacity. In the declarative mapping file, the capacity of the target array is specified through the capacity attribute of the value tag. When the number of elements in the source array exceeds the capacity, pruning is performed according to the capacity during the execution phase.

[0119] (5) Array padding: When the number of elements in the source array is less than the capacity of the target array, the remaining positions are filled according to the padding rules. In the declarative mapping file, the padding method is specified by the pad attribute, such as pad="zero" means padding with zero values, and pad="null" means padding with null values.

[0120] (6) Nested structure expansion: Expands nested object structures into flat structures. For example, it expands the nested {"device":{"id":123,"name":"sensor1"}} in the JSON input into a flat element structure in the XML output. <deviceid> 123< / deviceid> <devicename> sensor1< / devicename> .

[0121] (7) Nested structure merging: Merging multiple flat fields into a nested object structure. For example, merging the flat fields id, name, lat, and lng in the JSON input into a nested structure {"device":{"id":..., "name":...},"location":{"lat":..., "lng":...}} in the JSON output.

[0122] (8) Field Combination: Combine the values ​​of multiple input fields into one output field after calculation by an expression. For example, combine the last name and first name fields into a full name field using a string concatenation function.

[0123] (9) Type conversion: Convert field values ​​from one data type to another, such as converting a string type numeric value to an integer type, or an integer type to a floating-point type.

[0124] (10) Byte range truncation: Extracts a specified byte range from the raw byte stream of the Binary input source as the value of the output field.

[0125] (11) Bit processing: Extract a specific bit range of values ​​from a field of the Binary input source, or write the values ​​into a specific bit range of a field of the Binary output target.

[0126] In a specific application scenario, the above-mentioned structural transformations can be combined in the same declarative mapping file. For example, in an IoT device data reporting scenario:

[0127] Internal to JSON output: The device information fields from JSON input, XML input, and Binary input are migrated to a new object level (path migration), and the number of elements in the reported data array is pruned (array pruning, capacity="100"). When the reported data is less than 100, it is padded with zero values ​​(array padding, pad="zero").

[0128] Internal XML output: Maps the device number field in the JSON input to XML attributes (format conversion), and maps the device status field in the Binary input to the text content of XML elements;

[0129] Internally, the Binary output writes the device number from the JSON input into a fixed offset field of the target protocol as a uint32 type, and writes the configuration parameters from the XML input into the flag bytes of the target protocol as bit fields.

[0130] The declarative mapping file declares multiple input sources and multiple output targets within the same mapping file. The multiple input sources include at least two of the following formats: JSON, XML, and Binary. The multiple output targets also include at least two of the following formats: JSON, XML, and Binary. This enables a combined mapping transformation from multiple sources to multiple targets within a single mapping task. Taking the mapping file in Example 2 as an example, the same declarative mapping file declares three input sources (req for JSON, meta for XML, and calib for Binary) and three output targets (out_json for JSON, out_xml for XML, and out_bin for Binary). In a single execution, the combined mapping from the three input sources to the three output targets is completed, and the internal data structure reorganization of each output target is also completed.

[0131] The internal structure transformation method in this embodiment can express multiple structure transformation operations using a unified declarative mapping semantics, eliminating the need to write additional hand-written logic for each transformation. By declaring multiple input sources and multiple output targets simultaneously in the same mapping file, multi-source to multi-target combined mapping transformation and internal structure reorganization within the format can be completed in a single mapping task. This avoids the problem in existing technologies that require multiple calls to different transformation programs or the use of complex intermediate data flow logic to achieve the same effect, significantly improving the processing efficiency and maintainability of complex data exchange scenarios.

[0132] In the embodiments of this application, the method includes two execution modes: interpreted execution mode: the interpreted execution engine reads the mapped execution plan at runtime, calls different data access adapters for different input formats, and calls different data write adapters for different output formats. Expression evaluation, predicate control, and variable processing follow a unified semantics. Code generation execution mode: the code generation backend converts the mapped execution plan into C language source code. The generated C language source code includes a fixed entry function, input reading function, expression evaluation logic, predicate judgment logic, and output writing function. It does not require re-parsing of the declarative mapping file at runtime.

[0133] In another possible embodiment, the execution engine reads the mapped execution plan at runtime and dynamically performs the data mapping transformation. Its execution flow is as follows:

[0134] Step E1: Load the data from each input source according to the input source table. For JSON input sources, call the JSON parser to parse the JSON text into an object tree; for XML input sources, call the XML parser to parse the XML text into an element tree; for Binary input sources, directly receive the binary byte stream and establish a mapping relationship between fields and offsets based on the binary structure description file.

[0135] Step E2: Calculate the variable values ​​according to the variable table. Calculate the value of each variable in the order they are declared in the variable table. Variable expressions can reference input access, literals, function calls, and other calculated variables.

[0136] To further illustrate the specific processing steps when the variable table is not empty, the following is an example of a declarative mapping file containing variable declarations. In this example, the mapping file declares a JSON input source and declares four variables in the variable declaration section to store intermediate calculation results such as string length calculation, string-to-integer type conversion, string concatenation, and bit segment extraction. A snippet of its declarative mapping file is shown below:

[0137] < <dsl>

[0138] <inputs>

[0139] <input name="req" type="json" src="device_report.json">

[0140] < / inputs>

[0141] <variables>

[0142] <variable name="id_len" expr="#len($json:req{ / device / id})" type="uint16" / >

[0143] <variable name="mode_num" expr="#int($json:req{ / device / mode})" type="uint32" / >

[0144] <variable name="id_tag" expr="#concat('DEV-', $json:req{ / device / id})" type="string" / >

[0145] <variable name="mode_low4" expr="#bits($mode_num, 0, 3)" type="uint8" / >

[0146] < / variables>

[0147] <outputs>

[0148] <output name="out_json" type="json">

[0149] <value>

[0150] <fields>

[0151] <field target=". / meta / idLength"> <value kind="expr" expr="$id_len" / > < / field>

[0152] <field target=". / meta / idTag"> <value kind="expr" expr="$id_tag" / > < / field>

[0153] <field target=". / meta / modeLow"> <value kind="expr" expr="$mode_low4" / > < / field>

[0154] < / fields>

[0155] < / value>

[0156] < / output>

[0157] < / outputs>

[0158] < / dsl>

[0159] After parsing and semantically binding the above mapping file, the system generates a variable table containing the following four specific records, as shown in the table below:

[0160] 0 id_len #len($json:req{ / device / id}) uint16 1 mode_num #int($json:req{ / device / mode}) uint32 2 id_tag #concat('DEV-',$json:req{ / device / id}) string 3 mode_low4 #bits($mode_num,0,3) uint8

[0161] The specific meanings of the expressions for each variable in the table are as follows: The expression #len($json:req{ / device / id}) for the variable id_len indicates that the length function #len is called to calculate the string length of the / device / id field value in the JSON input source req; the expression #int($json:req{ / device / mode}) for the variable mode_num indicates that the type conversion function #int is called to convert the string value of the / device / mode field in the JSON input source req to an integer; the expression #concat('DEV-',$json:req{ / device / id}) for the variable id_tag ​​indicates that the string concatenation function #concat is called to concatenate the literal 'DEV-' with the / device / id field value to form a new string; the expression #bits($mode_num,0,3) for the variable mode_low4 indicates that the bit segment extraction function #bits is called to extract the lower 4 bits of the integer value represented by the 0th to 3rd bits from the already calculated variable mode_num.

[0162] In step E2, the execution engine calculates the value of each variable in the order they are declared in the variable table. Taking specific input data as an example: when the actual value of the / device / id field in the JSON input source req is the string "SN-2026-001" and the actual value of the / device / mode field is the string "12", the calculation process of the variable id_len is: #len("SN-2026-001")=12, that is, id_len=12; the calculation process of the variable mode_num is: #int("12")=12, that is, mode_num=12; the calculation process of the variable id_tag ​​is: #concat('DEV-',"SN-2026-001")="DEV-SN-2026-001", that is, id_tag="DEV-SN-2026-001"; the expression of the variable mode_low4 references the previously declared and calculated variable mode_num, and its calculation process is: #bits(12,0,3)=12, that is, mode_low4=12. Because variables are evaluated sequentially according to their declaration order, when a later-declared variable references an earlier-declared variable, the referenced variable has already been calculated, thus ensuring the validity of the variable expression reference. Subsequently, the `. / meta / idLength`, `. / meta / idTag`, and `. / meta / modeLow` fields in the output target `out_json` reference the calculation results of variables `$id_len`, `$id_tag`, and `$mode_low4`, respectively. When the variable table is empty, step E2 does not perform any variable calculations and proceeds directly to step E3.

[0163] Step E3: Determine the output-level conditions for each output target. Read the output condition expression (when attribute) for each output target in the output plan table and evaluate the expression. If the condition is true, continue writing for that output target; if the condition is false, skip that output target.

[0164] Step E4: Perform field-level conditions, checks, assertions, and skip logic on the output fields. Based on the rules in the predicate plan, perform checks, assertions, and skip checks on each output field. If a check fails, record diagnostic information; if an assertion fails, block subsequent writes to the current output target; if the skip condition is met, skip the write to the current field.

[0165] Step E5: Read field values ​​from the corresponding input source according to the access plan. Different data access adapters are invoked for different input formats: the JSON data access adapter extracts field values ​​from the JSON object tree by path, the XML data access adapter extracts element, attribute, or text node values ​​from the XML element tree by path, and the Binary data access adapter reads field values ​​by offset or extracts bit values ​​by bit position from the binary byte stream.

[0166] Step E6: Perform calculations according to the expression plan. This includes type conversions (such as string to integer), conditional functions (such as ternary expressions), concatenation functions (such as object concatenation), string concatenation, length calculation, byte range extraction (#binaryRange and #sliceBytes), and bit operations (#bits and #bit).

[0167] Step E7: Write the calculation results to the corresponding output format according to the target write plan. Different data write adapters are called for different output formats: the JSON data write adapter writes the field values ​​to the specified path in the JSON object tree, the XML data write adapter writes the field values ​​to the specified element, attribute, or text node in the XML element tree, and the Binary data write adapter writes the field values ​​to the specified position in the binary byte stream according to the type and offset.

[0168] Step E8: Summarize diagnostic information and output results. Summarize the results and diagnostic information of all output targets (including check failures, assertion failures, skipped records, etc.) and return them to the caller.

[0169] In the interpreted execution model, expression evaluation, predicate control, and variable handling follow a unified semantics. That is, regardless of the combination of input and output formats, the way the expression is evaluated, the control logic of the predicate, and the rules for handling variables remain consistent.

[0170] The code generation backend pre-converts the mapping execution plan into C source code. The generated C source code includes the following:

[0171] (1) Fixed entry function: The function signature is fixed and is used to receive the input buffer array, input length array, output buffer array, output length array, and diagnostic information. The entry function is the call entry point for the entire mapping transformation, and the caller only needs to pass in the parameters according to the agreement.

[0172] (2) Input reading functions: including Binary input field reading function (reads field values ​​from the input buffer based on offset and field type), JSON input field extraction function (extracts field values ​​from a specified path from the JSON text buffer), and XML input field extraction function (extracts element, attribute, or text node values ​​from a specified path from the XML text buffer).

[0173] (3) Variable initialization code: Generate variable declaration and initialization assignment statements based on the variable table.

[0174] (4) Expression evaluation code: Generate corresponding C language expression calculation code based on the expression plan, including arithmetic operations, string operations, type conversion, bit operations and condition judgments, etc.

[0175] (5) Predicate judgment code: Generate C language condition judgment code for checking, asserting and skipping logic based on the predicate plan.

[0176] (6) Output write functions: including Binary output field write function (writes values ​​to the output buffer according to the offset and field type), JSON output build function (builds JSON text according to the target write plan) and XML output build function (builds XML text according to the target write plan).

[0177] In the code-generated execution model, the runtime does not need to load declarative mapping files or run a full DSL compiler. The execution environment only needs to call the generated entry function to complete the mapping conversion between formats and the structural transformation within the format. This implementation can be used in execution environments that are sensitive to runtime dependencies and startup overhead, such as embedded systems, vehicle controllers, and industrial PLCs.

[0178] This implementation is one execution mode of the unified mapping transformation method, and it does not change the core semantics of the present invention regarding data format and internal structure mapping transformation. The output results of the two execution modes are consistent; that is, for the same input data and the same mapping execution plan, the output data and diagnostic information generated by the interpretation execution mode and the code generation execution mode are exactly the same.

[0179] Through the dual execution mode design in this embodiment, the same mapping execution plan can be dynamically executed at runtime via the interpreted execution engine (suitable for development and debugging phases, supporting dynamic loading and hot updates), or it can be pre-compiled into C language source code and run directly (suitable for production environments, eliminating runtime dependencies on mapping files, reducing startup latency and memory consumption). The two execution modes ensure consistent mapping semantics, allowing developers to quickly debug mapping rules using the interpreted execution mode during development and switch to the code generation mode during deployment for higher runtime performance, without needing to modify the mapping rules themselves.

[0180] In the embodiments of this application, in the interpreted execution mode, the execution engine analyzes the input sources that the output target depends on according to the access plan. When the assertion of a certain input source fails, only the output target that depends on that input source is blocked, without affecting the normal execution of other output targets. In the code generation execution mode, the generated C language source code includes a Binary input field reading function, a Binary output field writing function, a JSON input field extraction function, an XML input field extraction function, variable initialization code, expression evaluation code, predicate judgment code, and construction functions for each output target.

[0181] In another possible embodiment, in the interpreted execution mode, the execution engine analyzes the input sources that the output target depends on based on the access plan. Specifically, for each output target, the execution engine traverses all write steps in its target write plan, analyzes the access plan items referenced by each write step, and determines the input sources associated with those access plan items. This establishes a dependency graph from the output target to the input sources.

[0182] When data from a certain input source fails in the assertion check (for example, the message length of the Binary input source does not match the structure size defined in the binary structure description file, or a required field is missing in the JSON input source), the execution engine only blocks write operations to output targets that depend on that input source, without affecting the normal execution of other output targets.

[0183] The specific execution logic is as follows:

[0184] For each input source, maintain an assertion status flag (pass / fail);

[0185] For each output target, check the assertion state of all input sources it depends on before performing the write;

[0186] If all dependent input source assertions pass, the write to the output target will proceed normally.

[0187] If any dependent input source assertion fails, writes to that output target are blocked, diagnostic information (including the reason for the block and the failed dependent input source) is logged, but other output targets continue to be processed.

[0188] Taking a mapping file as an example: Assume the output target `outjson` depends on the input sources `req` and `meta`, the output target `outxml` depends on the input source `meta`, and the output target `outbin` depends on the input sources `req` and `calib`. When the assertion of the input source `calib` fails (e.g., the Binary protocol packet data is incomplete), only the output target `outbin` is blocked, while the output targets `outjson` and `outxml` can still be executed normally.

[0189] This mechanism is of great value in multi-input, multi-output scenarios: even if the data quality of a certain input source is poor or some data is missing, the system can still complete the output target without relying on that input source, realize local fault isolation in multi-output scenarios, and improve the system's fault tolerance and availability.

[0190] In the code generation and execution mode, the generated C language source code includes the following functions and code segments:

[0191] (1) Fixed entry function: the function prototype is void mapping — The `execute(const uint8_t* input_buffers, const size_t input_lengths, uint8t* output_buffers, size_t output_lengths, diag_t* diag)` method is used, where `input_buffers` is an array of pointers to input buffers, `input_lengths` is an array of lengths for each input buffer, `output_buffers` is an array of pointers to output buffers, `output_lengths` is an array of actual write lengths for each output buffer, and `diag` is a pointer to a diagnostic information structure.

[0192] (2) Binary input field reading function: Reads the value of the specified field from the input buffer based on the offset and field type calculated in the binary structure description file. The function selects the corresponding reading method according to the field type (uint8, uint16, uint32, etc.) and handles byte order conversion according to the byte order parameter.

[0193] (3) Binary output field write function: Writes the value to the specified position in the output buffer according to the offset and field type specified in the target write plan. The function selects the corresponding write mode according to the field type and handles byte order conversion.

[0194] (4) JSON input field extraction function: Extracts field values ​​from the JSON text buffer according to the path specified in the target write plan (e.g., / payload / deviceId). The function implements lightweight JSON path traversal logic internally.

[0195] (5) XML input field extraction function: Extracts element or attribute values ​​from the XML text buffer according to the path specified in the target write plan (e.g., / report / @device). The function implements lightweight XML path traversal logic internally.

[0196] (6) Variable initialization code: Generate corresponding C language variable declaration and initialization assignment statements according to the variable table to ensure that the variables have been correctly assigned before use. For example of the above non-empty variable table, the variable initialization C language code generated by the code generation backend is as follows: uint16_t id_len; uint32_t mode_num; char id_tag

[64] ; uint8_tmode_low4; id_len=len_func(json_get(req," / device / id")); mode_num=int_func(json_get(req," / device / mode")); concat_func(id_tag,"DEV-",json_get(req," / device / id")); mode_low4=bits_func(mode_num,0,3); The above code completes the variable declaration and initialization assignment before each variable is referenced by the subsequent expression evaluation code to ensure that the variables have been correctly assigned before use.

[0197] (7) Expression evaluation code: Generates corresponding C language expression calculation code based on the expression plan, supporting arithmetic operations, string concatenation, condition judgment, type conversion, bit operations (bit extraction and bit writing), byte range truncation and other operations.

[0198] (8) Predicate judgment code: Generate corresponding C language condition judgment code based on the predicate plan, including output-level condition judgment (corresponding to the when attribute), field-level check, assertion and skip logic.

[0199] (9) Construction functions for each output target: Based on the data format type of the output target, generate JSON output construction functions, XML output construction functions, and Binary output construction functions respectively. Each construction function writes the field values ​​into the output data structure in sequence according to the writing steps in the target writing plan.

[0200] In the generated C language source code, the execution logic of all functions is consistent with the semantics in the interpreted execution mode. That is, for the same input data and the same mapping execution plan, the two execution modes produce the same output results and diagnostic information.

[0201] Through the assertion failure local blocking mechanism in this embodiment, the system achieves fine-grained fault isolation in multi-input multi-output scenarios. When the data from a certain input source is abnormal, only the output targets that depend on that input source are affected, without affecting the normal execution of other output targets, thus improving the system's fault tolerance and data processing reliability. The detailed code structure of the code generation execution form in this embodiment compiles the mapped execution plan into independent C language source code. At runtime, it does not require loading declarative mapping files and a DSL compiler, eliminating runtime dependencies, reducing startup latency and memory consumption, and making it suitable for performance-sensitive execution environments such as embedded systems and real-time control.

[0202] Example 2:

[0203] Reference Figure 6 This application also provides a unified declarative mapping and transformation system for heterogeneous data formats and their internal structures, comprising: a declarative mapping file acquisition module, used to acquire a declarative mapping file, wherein the declarative mapping file declares access to the structure of different data formats through a format prefix and a unified path expression method, wherein the different data formats include at least one of JSON, XML, and Binary; a parsing and semantic binding module, connected to the declarative mapping file acquisition module, used to parse and semantically bind the declarative mapping file, and generate a format-independent mapping execution plan, wherein the mapping execution plan uniformly represents the object and array structure of JSON, the element, attribute and text node structure of XML, and the structure, field offset and bit field structure of Binary as an executable access plan and a target write plan; and an adapter execution module, connected to the parsing and semantic binding module, used to call the data access adapter corresponding to the input source format type to read field values ​​and call the data write adapter corresponding to the output target format type to write the mapping result according to the mapping execution plan; wherein the mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

[0204] In another possible embodiment, the system includes three core modules: a declarative mapping file acquisition module, a parsing and semantic binding module, and an adapter execution module.

[0205] The declarative mapping file acquisition module is used to acquire declarative mapping files. These declarative mapping files declare access to structures in different data formats through format prefixes and uniform path expressions. These different data formats include at least one of JSON, XML, and Binary. The declarative mapping files can be acquired by: reading configuration files from the file system, receiving mapping definitions from a network interface, extracting mapping rules from a database, or loading pre-compiled mapping plans from a memory cache.

[0206] The parsing and semantic binding module is connected to the declarative mapping file acquisition module. The connection is as follows: the declarative mapping file acquisition module passes the acquired declarative mapping file content to the parsing and semantic binding module as its input. The parsing and semantic binding module parses and semantically binds the declarative mapping file, generating a format-independent mapping execution plan. This mapping execution plan uniformly represents JSON object and array structures, XML elements, attributes and text node structures, and Binary structures, field offsets, and bit structures as executable access plans and target write plans.

[0207] The specific functions of the parsing and semantic binding module include:

[0208] Lexical analysis: Decomposes the text content of a declarative mapping file into a sequence of tokens;

[0209] Syntax parsing: Organizes the token sequences into a syntax tree according to the DSL syntax rules of the declarative mapping file;

[0210] Semantic validation: Validates whether the input source name exists, whether the output target name exists, whether the variable reference is valid, whether the expression parameter types match, and whether the binary input and output are bound to the binary structure description file, etc.

[0211] Execution plan generation: Converts the syntax tree into a format-independent mapping execution plan, including the input source table, variable table, output plan table, access plan, expression plan, target write plan, and predicate plan.

[0212] The adapter execution module is connected to the parsing and semantic binding module. The connection is as follows: the parsing and semantic binding module passes the generated mapping execution plan to the adapter execution module as its execution basis. According to the mapping execution plan, the adapter execution module calls the data access adapter corresponding to the input source format type to read field values, and calls the data writing adapter corresponding to the output target format type to write the mapping result.

[0213] The workflow of the adapter execution module is as follows:

[0214] Based on the input source table in the mapping execution plan, load the actual data for each input source;

[0215] Calculate the intermediate variable values ​​based on the variable table;

[0216] For each output target, read the field values ​​from the corresponding input source according to the access plan;

[0217] The values ​​of the read fields are calculated according to the expression plan;

[0218] According to the target write plan, the calculation results are written to the output target through the corresponding data write adapter.

[0219] The mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

[0220] This embodiment employs a three-module system architecture, achieving a clear layering of declarative mapping file acquisition, semantic parsing and binding, and adapter execution. The declarative mapping file acquisition module is responsible for obtaining mapping rules, the parsing and semantic binding module is responsible for compiling the mapping rules into a format-independent execution plan, and the adapter execution module is responsible for completing the actual data read and write operations according to the execution plan. The connections between modules are clear, and each module has independent responsibilities. When new data formats need to be supported, only the corresponding data access adapter and data write adapter need to be added to the adapter execution module; the logic of the declarative mapping file acquisition module and the parsing and semantic binding module does not need to be modified, significantly improving the system's scalability and maintainability.

[0221] In the embodiments of this application, the mapping execution plan generated by the parsing and semantic binding module includes an input source table, a variable table, an output plan table, an access plan, an expression plan, a target writing plan, and a predicate plan; the adapter execution module includes: a data access adapter unit, including a JSON data access adapter, an XML data access adapter, and a Binary data access adapter, respectively used to read field values ​​from input sources of corresponding formats; and a data writing adapter unit, including a JSON data writing adapter, an XML data writing adapter, and a Binary data writing adapter, respectively used to write the mapping results to output targets of corresponding formats; both the data access adapter unit and the data writing adapter unit are subject to unified scheduling by the mapping execution plan, and expression evaluation, predicate control, and variable processing follow unified semantics.

[0222] In another possible embodiment, the mapping execution plan generated by the parsing and semantic binding module includes seven types of plan structures:

[0223] (1) Input source table: records the input source number, name, format type, and data structure description reference;

[0224] (2) Variable table: records the variable number, name, expression, and variable value type;

[0225] (3) Output plan table: Record the output target number, name, format type, output conditions and predicate rules;

[0226] (4) Access plan: Record field access steps, including object field access, array index access, XML element access, XML attribute access, XML text node access, Binary offset reading and Binary bit field extraction;

[0227] (5) Expression plan: Records literals, input accesses, variable references, context references, function calls, and formula expressions;

[0228] (6) Target write plan: Record the output target path and corresponding value, including JSON target field writing, XML target attribute writing, XML target element writing, XML target text node writing and Binary target field writing;

[0229] (7) Predicate plan: record checks, assertions and skip rules.

[0230] The adapter execution module includes two sub-units:

[0231] Data access adapter unit: includes JSON data access adapter, XML data access adapter and Binary data access adapter, which are used to read field values ​​from input sources of the corresponding formats.

[0232] JSON Data Access Adapter: Receives JSON formatted data, extracts the corresponding field values ​​from the JSON object tree according to the object field path (such as / device / id) or array index position specified in the access plan, and converts the field values ​​into a unified internal data type representation.

[0233] XML Data Access Adapter: Receives XML formatted data, extracts the corresponding values ​​from the XML document tree according to the element path (e.g., / routing / header), attribute path (e.g., / routing / @trace), or text node path (e.g., / report / trace / #text) specified in the access plan, and converts the values ​​into a unified internal data type representation.

[0234] Binary Data Access Adapter: Receives binary format data and the corresponding binary structure description file. Based on the field names specified in the access plan, and according to the offsets and field types calculated in the binary structure description file, it reads the field values ​​from the binary byte stream. For bit field fields, it extracts the bit field values ​​from the carrying byte according to the bit field position and bit width.

[0235] Data write adapter unit: includes JSON data write adapter, XML data write adapter and Binary data write adapter, which are used to write the mapping results to the corresponding output target.

[0236] JSON data write adapter: According to the target path specified in the target write plan (such as . / payload / deviceId), create or modify the corresponding field node in the JSON object tree, and write the mapping calculation result to the field.

[0237] XML Data Write Adapter: Creates or modifies the corresponding element node, attribute node, or text node in the XML document tree according to the target path specified in the target write plan, and writes the mapping calculation results to the node.

[0238] Binary Data Write Adapter: Based on the target path and field type specified in the target write plan, and according to the offset calculated in the binary structure description file, writes the mapping calculation results to the corresponding positions in the binary byte stream according to the specified data type and byte order. For bit field fields, it writes the value to a specific bit range of the carrying byte according to the bit field position and bit width.

[0239] Both the data access adapter unit and the data write adapter unit are subject to unified scheduling by the mapping execution plan. During execution, the adapter execution module reads field values ​​according to the access plan described in the mapping execution plan, calculates results according to the expression plan, and writes results according to the target write plan. Throughout the execution process, expression evaluation, predicate control, and variable processing follow a unified semantics and do not change the calculation logic due to different data formats.

[0240] When it is necessary to extend support for new data formats (such as CSV, Protobuf, Avro, etc.), simply add new adapters for the corresponding formats to the data access adapter unit and the data write adapter unit, respectively. Since the mapping semantics are carried by the format-independent mapping execution plan, adding new formats does not require changes to the expression semantics, variable semantics, predicate semantics, and multi-output scheduling semantics.

[0241] Through the adapter unit architecture design in this embodiment, the system encapsulates the data format access and writing logic in independent adapters, and the mapping execution plan is responsible for uniformly scheduling all adapters. This design allows the data formats supported by the system to be independently extended. Adding a new data format only requires adding the corresponding adapter implementation, without modifying the structure of the mapping execution plan and the core execution logic. The six adapters (three access adapters and three write adapters) cover the three main data formats: JSON, XML, and Binary, which can meet the data format conversion needs in most industrial control and data processing scenarios.

[0242] In the embodiments of this application, the Binary data access adapter or the Binary data writing adapter is bound to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. The parsing and semantic binding module calculates the offset, field length, array step size, and bit field position of the Binary fields based on the binary structure description file, so that the Binary protocol data, as a data format with structural semantics, participates in the conversion with JSON and XML in the same declarative mapping semantics. The system supports interpreted execution mode and code-generated execution mode. The interpreted execution mode reads the mapping execution plan and performs dynamic execution at runtime, while the code-generated execution mode pre-converts the mapping execution plan into C language source code, and directly calls the generated code at runtime to complete the mapping conversion.

[0243] In another possible embodiment, in the system, a binary data access adapter or a binary data write adapter is bound to a binary structure description file, the binary structure description file comprising:

[0244] (1) Structure: Defines the data structure of the binary protocol and supports nested structures, that is, a field of one structure can be another defined structure type;

[0245] (2) Field name: Defines the identifier name of each field in the structure, which is used to reference it in the declarative mapping file;

[0246] (3) Field type: Define the data type of each field, supporting basic types (uint8, uint16, uint32, int8, int16, int32, float, double, etc.) and composite types (structure type reference, fixed-length array, byte array, character array);

[0247] (4) Field order: Defines the order in which fields are arranged in the structure. The byte offset of the fields in the binary message is calculated sequentially according to the field order and field type.

[0248] (5) Array length: Defines the number of elements in a fixed-length array;

[0249] (6) Byte length: Defines the number of bytes in a byte array or character array;

[0250] (7) Bit field information: Define the bit width and bit offset of a specific range in a field through bit manipulation functions (#bits, #bit, etc.) in the declarative mapping file.

[0251] The binary structure description file also includes global attributes:

[0252] endian: byte order, which can be either little (little endian) or big (big endian), affecting how multi-byte fields are parsed;

[0253] alignment: Alignment method, takes a positive integer value, representing the number of bytes to align between fields, a value of 1 indicates no alignment padding;

[0254] root: The name of the root structure, specifying the top-level data structure of the binary protocol.

[0255] The parsing and semantic binding module calculates the offset, field length, array step size, and bit segment position of the Binary field based on the binary structure description file. The specific calculation rules are as follows:

[0256] For basic type fields: the field length equals the number of bytes of that type (e.g., uint8 is 1 byte, uint16 is 2 bytes, uint32 is 4 bytes), the field offset equals the offset of the previous field plus the length of the previous field, and the offset of the first field is 0. If the alignment is greater than 1, the field offset must be rounded up according to the alignment rules.

[0257] For structure type fields: the field length is equal to the sum of the lengths of all fields in the structure (including alignment padding), and the field offset is calculated according to the rules for basic type fields.

[0258] For fixed-length array fields: the array step is equal to the length of a single element, the total field length is equal to the array step multiplied by the array length, and the field offset is calculated according to the rules for basic type fields.

[0259] Through the above calculations, each Binary field obtains explicit byte offset, field length, and bit position information. This allows Binary protocol data to participate in conversion within the same declarative mapping semantics as a data format with structured semantics, alongside JSON and XML. In the mapping execution plan, the access plan items for Binary fields have the same structure and status as those for JSON and XML fields; the only difference lies in the specific reading method.

[0260] The system supports two execution modes: interpreted execution and code-generated execution.

[0261] The interpreted execution mode dynamically executes the mapped execution plan at runtime. The interpreted execution engine loads data from the input source table in the mapped execution plan, calls the corresponding data access adapter to read field values ​​according to the access plan, calculates the result according to the expression plan, and calls the corresponding data write adapter to write the result according to the target write plan. In interpreted execution mode, the declarative mapping file and the mapped execution plan are loaded at runtime, supporting dynamic updates and hot replacement, making it suitable for development and debugging phases.

[0262] The code generation execution mode pre-converts the mapping execution plan into C language source code. The code generation backend reads the mapping execution plan and generates corresponding C language data reading code, expression calculation code, and data writing code according to the access plan, expression plan, and target write plan in the execution plan. At runtime, the generated code is directly called to complete the mapping conversion, without needing to load declarative mapping files and mapping execution plans, or run a DSL compiler. In this code generation execution mode, runtime dependencies are minimized, startup latency is low, and memory usage is small, making it suitable for performance- and resource-sensitive production environments such as embedded systems, automotive controllers, and industrial PLCs.

[0263] Both execution modes share the same mapping execution plan, ensuring semantic consistency. For the same input data and the same mapping execution plan, the output data and diagnostic information generated by the interpretation execution mode and the code generation execution mode are exactly the same.

[0264] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0265] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0266] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A unified declarative mapping and transformation method for heterogeneous data formats and their internal structures, characterized in that, include: Obtain a declarative mapping file, which declares access to structures of different data formats through a format prefix and a uniform path expression, wherein the different data formats include at least one of JSON, XML and Binary; The declarative mapping file is parsed and semantically bound to generate a format-independent mapping execution plan. The mapping execution plan uniformly represents JSON object and array structures, XML elements, attributes and text node structures, and Binary structures, field offsets and bit structures as executable access plans, expression plans and target write plans. According to the mapping execution plan, the data access adapter corresponding to the input source format type is called to read the field value, and the data write adapter corresponding to the output target format type is called to write the mapping result; The mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

2. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 1, characterized in that, The declarative mapping file includes input declarations, output declarations, model declarations, and variable declarations; The input declaration is used to declare the input source name, input source type, and optional data structure description; the output declaration is used to declare the output target name, output target type, output conditions, and output fields, and the field values ​​are calculated from the input access path, variables, context references, literals, or expressions. The unified path expression method includes a format prefix and a path body. The format prefix is ​​used to identify JSON, XML, Binary, or context data, and the path body is used to locate object fields, array elements, XML elements, XML attributes, XML text nodes, Binary structure fields, Binary byte ranges, or Binary bit fields.

3. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 1, characterized in that, The mapping execution plan includes: The input source table records the input source number, name, format type, and data structure description reference; The variable table records the variable number, name, expression, and variable value type; Output plan table, recording the output target number, name, format type, output conditions and predicate rules; Access plan, records the steps for accessing fields; Expression plans record literals, input accesses, variable references, context references, function calls, and formula expressions; The target is written into the plan, recording the output target path and corresponding value; Predicate plans record checks, assertions, and skip rules.

4. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 1, characterized in that, The binary input source or binary output target is bound to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. When generating the mapping execution plan, the offset, field length, array step size, and bit position of the Binary field are calculated based on the binary structure description file, so that the Binary protocol data, as a data format with structured semantics, participates in the conversion in the same declarative mapping semantics as JSON and XML.

5. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 1, characterized in that, The structural transformations within the same data format include at least one of the following: field renaming, path migration, object reorganization, array pruning, array padding, nested structure expansion, nested structure merging, field combination, type conversion, byte range truncation, or bit segment processing. The declarative mapping file declares multiple input sources and multiple output targets in the same mapping file. The multiple input sources include at least two of the following formats: JSON, XML, and Binary. The multiple output targets include at least two of the following formats: JSON, XML, and Binary. This enables a combined mapping transformation from multiple sources to multiple targets in a single mapping task.

6. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 1, characterized in that, The method includes two execution modes: Interpreted execution mode: The interpreted execution engine reads the mapped execution plan at runtime, calls different data access adapters for different input formats, and calls different data write adapters for different output formats. Expression evaluation, predicate control, and variable handling follow a unified semantics. Code generation execution mode: The code generation backend converts the mapping execution plan into C language source code. The generated C language source code includes a fixed entry function, input reading function, expression evaluation logic, predicate judgment logic, and output writing function. It does not require re-parsing of the declarative mapping file at runtime.

7. The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures according to claim 6, characterized in that, In the interpreted execution mode, the execution engine analyzes the input sources that the output target depends on based on the access plan. When the assertion of a certain input source fails, it only blocks the output target that depends on that input source and does not affect the normal execution of other output targets. In the code generation and execution mode, the generated C language source code includes functions for reading binary input fields, writing binary output fields, extracting JSON input fields, extracting XML input fields, variable initialization code, expression evaluation code, predicate judgment code, and construction functions for each output target.

8. A unified declarative mapping and transformation system for heterogeneous data formats and their internal structures, characterized in that, The unified declarative mapping and transformation method for heterogeneous data formats and their internal structures, as described in any one of claims 1-7, includes: A declarative mapping file acquisition module is used to acquire a declarative mapping file, wherein the declarative mapping file declares access to the structure of different data formats through a format prefix and a uniform path expression method, wherein the different data formats include at least one of JSON, XML and Binary; The parsing and semantic binding module, connected to the declarative mapping file acquisition module, is used to parse and semantically bind the declarative mapping file to generate a format-independent mapping execution plan. The mapping execution plan uniformly represents JSON object and array structures, XML elements, attributes and text node structures, and Binary structures, field offsets and bit structures as executable access plans and target write plans. The adapter execution module, connected to the parsing and semantic binding module, is used to call the data access adapter corresponding to the input source format type to read field values ​​according to the mapping execution plan, and call the data writing adapter corresponding to the output target format type to write the mapping result. The mapping transformation includes format conversion between different data formats and structural transformation within the same data format, and the structural transformation and the format conversion are described and executed by the same mapping execution plan in the same mapping task.

9. The unified declarative mapping and transformation system for heterogeneous data formats and their internal structures according to claim 8, characterized in that, The parsing and semantic binding module generates a mapping execution plan that includes an input source table, a variable table, an output plan table, an access plan, an expression plan, a target write plan, and a predicate plan. The adapter execution module includes: The data access adapter unit includes a JSON data access adapter, an XML data access adapter, and a Binary data access adapter, which are used to read field values ​​from input sources of the corresponding formats, respectively. The data writing adapter unit includes a JSON data writing adapter, an XML data writing adapter, and a Binary data writing adapter, which are used to write the mapping results to the corresponding output targets. Both the data access adapter unit and the data write adapter unit are subject to unified scheduling by the mapping execution plan, and expression evaluation, predicate control, and variable processing follow unified semantics.

10. The unified declarative mapping and transformation system for heterogeneous data formats and their internal structures according to claim 8, characterized in that, The Binary data access adapter or the Binary data write adapter is bound to a binary structure description file, which includes a structure, field names, field types, field order, array length, byte length, and bit field information. The parsing and semantic binding module calculates the offset, field length, array step size, and bit position of the Binary field based on the binary structure description file, so that the Binary protocol data, as a data format with structured semantics, participates in the conversion with JSON and XML in the same declarative mapping semantics. The system supports both interpreted execution mode and code-generated execution mode. The interpreted execution mode reads the mapped execution plan and executes it dynamically at runtime. The code-generated execution mode pre-converts the mapped execution plan into C language source code, and directly calls the generated code at runtime to complete the mapping conversion.

Images