A hierarchical API response misalignment prevention mapping method based on array-associated scope recognition
By constructing an array appearance tree and identifying associated scopes, generating record groups and performing cardinality conservation checks, the problems of field misalignment and data distortion in API responses are solved, and an interpretable, traceable, and verifiable hierarchical API response mapping is achieved.
Patent Information
- Application Number
- CN202611603711.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot effectively identify the array-related scope of fields when parsing API responses, leading to silent field misalignment, illegal Cartesian expansion, parent field propagation errors, inability to express multi-entity outputs, and lack of verification of mapping correctness, resulting in data distortion and pollution.
By constructing an array appearance tree, the associated scope of fields is identified, record groups are generated, and cardinality conservation checks are performed to ensure that only provable fields are allowed to synthesize the same output record. The mapping results are verified using association keys and cardinality constraints to prevent misalignment and erroneous data writing.
It implements an interpretable, traceable, and verifiable hierarchical API response mapping, avoids silent misalignment errors, correctly expresses multi-level and multi-entity responses, and ensures the accuracy and traceability of write results.
Smart Images

Figure CN123115149A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of data integration, application programming interface data acquisition, semi-structured data parsing, data conversion, and data quality control, specifically to a hierarchical API response anti-misalignment mapping method, system, computer-readable storage medium, and electronic device based on array association scope identification. Background Technology
[0002] With the increasing demand for data exchange between enterprise business systems, government service platforms, industrial internet platforms, and internet services, Application Programming Interfaces (APIs) have become an important way to transmit data between systems. Many APIs return semi-structured response data in JSON or XML format, and the actual responses often contain multiple levels of nested objects and arrays, such as orders and their details, personnel and their organizations, equipment and their alarm lists, projects and their member lists, etc. Data synchronization frameworks, such as DataX, typically use a Reader component to read data from the source, a Channel component to transmit data in internal Record format, and a Writer component to write data to the target. They also have the ability to extend the types of data sources and target devices through a plugin mechanism.
[0003] The current ApiReader in the project supports HTTP / HTTPS requests, GET or POST request parameters, request headers, request bodies, interface authentication, call frequency control, failure retries, basic pagination, JSON path reading, and recursive retrieval of array paths. For simple responses involving a single array or where the path structures of each field are strictly consistent, the existing capability can convert the parsed field values into DataX Records and write them to the target end.
[0004] However, in responses with complex hierarchical structures, the array position of a field does not inherently indicate that it belongs to the same business entity. For example, each order element in the `orders` array in the response may simultaneously contain both `items` and `coupons` arrays. Although `items[i].sku` and `coupons[i].code` have the same array index, they belong to two independent multi-value entities and should not be concatenated into a single "product-coupon" record. Similarly, when the user array and the role array are sibling arrays, elements with the same index do not necessarily belong to the same user. Existing parsing implementations typically recursively locate the array independently for each field path and write each field into the same intermediate row set according to its array index. When multiple fields originate from different array scopes, the system may mistakenly identify array indices with the same value as evidence of association, leading to the following problems:
[0005] 1. Silent field misalignment. The i-th element from different sibling arrays is written to the same row. The task may complete normally in each stage of calling, parsing and writing, but the business meaning of the target data has been erroneous and is difficult to detect.
[0006] 2. Illegal Cartesian expansion or truncation. When forcibly merging two independent arrays, some implementations use index alignment to discard the tail data of the longer array, while others use cross-combination to generate a large number of false records. Both lead to data distortion.
[0007] 3. Parent field propagation error. Parent object fields should be broadcast to all elements of their child arrays; when there are multiple nested parent arrays, the existing column-by-column filling method cannot distinguish which parent object instance a field should inherit from, and may bring fields from adjacent parent objects into the current child object.
[0008] 4. Inability to express multiple entities in a single response. An API response often contains a main entity, child entities, and multiple detail entities. However, the existing processing method of "one Reader corresponding to one wide table row set" does not have the ability to split and output the main table, child table, or side table based on the associated scope.
[0009] 5. Lack of verifiable evidence of mapping correctness. Most existing tasks only verify HTTP status codes and business success codes, or only record the number of rows read and written; even if the number of rows is the same, it cannot prove that the fields concatenated to the same target row actually come from the same array instance.
[0010] 6. Lack of security degradation mechanism. For two parallel arrays that have a corresponding relationship in business but do not provide an association key in the response, the existing implementation may default to association by position; however, position association is an unprovable assumption, and the resulting erroneous results will be silently written to the production target, causing data pollution.
[0011] Therefore, there is an urgent need to develop a hierarchical API response anti-misalignment mapping scheme that identifies the array association scope of fields during the API response parsing phase, allows only fields with provable association relationships to be combined into the same output record, and verifies the mapping results with association keys and cardinality constraints. Summary of the Invention
[0012] To overcome the aforementioned deficiencies in existing technologies, this application proposes a novel hierarchical API response anti-misalignment mapping method and system based on array-related scope identification. More specifically, this invention proposes a method for identifying the related scope, generating record groups, and preventing misaligned writing of hierarchical API responses such as JSON and XML containing multi-level objects, nested arrays, parallel arrays, or objects within arrays. This aims to solve the technical problems in existing DataX API integration caused by independent parsing by field and concatenation by array index, resulting in hierarchical response field misalignment, parent-child attribution errors, illegal Cartesian expansion, long and short array truncation, and successful tasks containing erroneous data. Thus, while maintaining compatibility with existing Reader-Channel-Writer processing chains, it achieves an interpretable, traceable, and verifiable hierarchical API response mapping mechanism.
[0013] To achieve the above objectives, the present invention employs the following technical strategies:
[0014] 1. Propose an array occurrence tree: Separate the static array path from the actual array occurrence in each parent object, and accurately record the parent-child relationship and element instances of multi-level arrays.
[0015] 2. A field association scope identification mechanism based on array ancestor sequence, longest common prefix and nearest common array ancestor is proposed to determine whether fields can be safely combined into the same record from the parsing logic level.
[0016] 3. Explicitly exclude "same array index" as the default association criterion, and design splitting, isolation and blocking mechanisms for cases where sibling arrays have no natural association.
[0017] 4. Propose an association key composed of endpoint / request summary, entity, record group, and array occurrence vector, so that the target record has the ability to locate the hierarchical source traceably.
[0018] 5. A differentiated cardinality conservation check is proposed for same-element, parent-level broadcast, cross-level dependency, and explicit association key connections, which can identify silent truncation and illegal Cartesian expansion.
[0019] 6. In the DataX plugin architecture, the associated scope proof, record group, and verification status are passed along with the internal Record, taking into account both the feasibility of the project and the verifiability of the target end write.
[0020] Specifically, this application provides the following technical solutions:
[0021] The first aspect of this application provides a hierarchical API response misalignment prevention mapping method based on array-associated scope recognition, applied to the DataX data synchronization system, such as... Figure 1 As shown, it includes the following steps:
[0022] S1, read the interface configuration and mapping configuration, parse the API response into a node tree, and build an array occurrence tree for array definition nodes and their occurrence under the parent object instance;
[0023] S2, extract and compare the array ancestor sequence of each field to be mapped to determine the field association scope; generate a mapping plan and form a record group based on the field association scope, only include fields with provable association relationships in the same record group, and prohibit including fields in the same record group by array subscript;
[0024] S3, traverse the array occurrence tree according to the mapping plan to generate the output Record and the association key containing the array occurrence vector;
[0025] S4 performs a cardinality check on the record group; if the check passes, the output Record, its associated key, and the check status are written to the DataX Record extended field or task context and sent to the writing end via the transmission channel; if the check fails or the association cannot be proven, security measures are taken.
[0026] Furthermore, in the method of this application, the array occurrence node in the array occurrence tree carries at least the array canonical path, the parent array occurrence node identifier, the current element index, the parent object instance identifier, and the number of array elements; the same array definition node forms different array occurrence nodes under different parent object instances.
[0027] Furthermore, in the method of this application, in step S2, the field association scope is determined based on the longest common prefix and the nearest common array ancestor of the field array ancestor sequence, and includes at least the same element association scope, parent broadcast scope, cross-level dependent scope, explicit association key scope, sibling array no natural association scope, and association relationship uncertain scope;
[0028] When two fields share the same deepest common array node, and the fields are located within the same element instance of that common array node, they are considered to have the same element associative scope. When the first field is not located in any array or is only located in the parent object of the second field's array ancestor, it is considered to have the parent broadcast scope. When the first field is located in the array element pointed to by the proper prefix of the second field's array ancestor sequence, it is considered to have the cross-level dependent scope. When two fields are located in different array branches and have verifiable equality keys or verifiable mapping rules, they are considered to have the explicit key scope. When two fields are only associated based on the same array position, array length, or traversal order, they are considered to have the sibling array without natural association scope. Other cases that do not meet the preset proof conditions are considered to have uncertain association scope.
[0029] Furthermore, in the method of this application, in step S2, the mapping plan includes at least a record group, a root scope of the record group, the value scope of each field, field inheritance rules, output entities, association key generation rules, cardinality verification rules, and exception handling rules; fields in the same element association scope, parent broadcast scope, or the same cross-layer subordinate link are compiled into the same record group; fields in sibling arrays without natural association scope or with uncertain association relationships shall not be combined into the same record group solely based on array subscripts.
[0030] Furthermore, in the method of this application, in step S3, when generating the output Record, the array occurrence vector corresponding to the root scope of the record group is used as the positioning context, and the field values are extracted according to the rules of reading the same element, parent broadcasting, cross-level inheritance, or explicit association key matching.
[0031] Furthermore, in the method of this application, the association key includes at least one of the following: interface endpoint or interface identifier, normalized request condition summary, response entity path, record group identifier, and array occurrence vector; the array occurrence vector includes the element sequence number or instance identifier of the record group root array occurrence node and its ancestor array occurrence nodes; the association key also includes the source business primary key, the parent entity business primary key, or an explicit association key value; the array occurrence vector is not used alone as an idempotent key for cross-synchronization, but is used in combination with the response content normalized summary, request summary, or source version identifier.
[0032] Furthermore, in the method of this application, step S4, the cardinality conservation check includes at least one of the following: the number of output records of the same element group is consistent with the number of elements in the root array; the number of child records after parent broadcast is consistent with the number of elements in the child array; the number of output records of cross-level subordinate record groups is consistent with the sum of the number of elements in the child arrays under each parent object; the number of matches after explicit association key matching conforms to the preset one-to-one, one-to-many, or many-to-one constraints; the association keys of the target entity are not repeated or the repetition conforms to the preset aggregation rules.
[0033] Furthermore, in the method of this application, in step S4, the security handling includes at least one of the following: splitting into different record groups or different target entities, writing to an isolated temporary storage area, outputting unconfirmed mapping evidence, rejecting the current configuration release, skipping uncertain fields, or terminating the task; the security handling is triggered when it is detected that the sibling arrays have no natural association, the association key is not unique, the array cardinality is abnormal, the field type is incompatible, or the association relationship cannot be proven; wherein, the same array length, the same array element sequence number, similar field names, the same values by chance, or appearing in the same response page, shall not be used alone as proof that two array branches can be associated.
[0034] The second aspect of this application provides a hierarchical API response misalignment prevention mapping system based on array-associative scope recognition. The system, when running, implements the steps of the aforementioned hierarchical API response misalignment prevention mapping method based on array-associative scope recognition, such as... Figure 2 As shown, the system includes:
[0035] The interface configuration module is used to store the interface address, authentication method, request parameters, success conditions, response root path, and pagination parameters.
[0036] The mapping configuration module is used to store source field paths, target entities, target fields, association keys, array association strategies, target end write strategies, and mapping plan versions.
[0037] The response node tree building module is used to parse JSON, XML, or other hierarchical responses and form a node tree.
[0038] The array appearance tree building module is used to record the specific occurrence of each array definition under each parent object instance, the array element index, and the ancestor relationship;
[0039] The scope identification module is used to calculate the array ancestor sequence, longest common prefix, nearest common array ancestor, and field-related scope of a field.
[0040] The mapping plan generation module is used to organize record groups, field inheritance rules, association key rules, and exception policies according to the associated scope;
[0041] The record group execution module is used to generate the DataX internal Record based on the array occurrence vector;
[0042] The cardinality proof module is used to perform checks on the number of inputs and outputs of record groups, the uniqueness of association keys, the cardinality of connections, and parent-child attribution.
[0043] The security handling module is used to split, isolate, block, or manually confirm fields for which a relationship cannot be proven.
[0044] The Writer adapter module is used to write Records with record group identifiers and association keys to the target main table, sub-table, associated table, temporary storage area, or other target end, and to return the write status.
[0045] Furthermore, in the system of this application, the scope identification module is configured as follows: when the deepest common arrays of two fields have the same node and are located within the same element instance, it is determined to be an element-related scope; when the first field is not located in any array or is only located in the parent object of the ancestor of the second field array, it is determined to be a parent broadcast scope; when the first field is located in the array element pointed to by the proper prefix of the ancestor sequence of the second field array, it is determined to be a cross-level dependent scope; when the two fields are located in different arrays with branches and there is a verifiable equal association key or mapping rule, it is determined to be an explicit association key scope; when the only association criterion is the same array position, array length, or traversal order, it is determined to be a sibling array without natural association scope; and in other cases, it is determined to be an indeterminate association scope.
[0046] A third aspect of this application provides an electronic device, including: a memory and a processor;
[0047] Memory: Used to store computer programs;
[0048] Processor: Used to execute the computer program to implement the steps of the aforementioned hierarchical API response misalignment prevention mapping method based on array-associated scope recognition.
[0049] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned hierarchical API response anti-misalignment mapping method based on array-associated scope identification.
[0050] In summary, compared with the prior art, the solution of the present invention has the following advantages:
[0051] (1) By using the array appearance tree, we can distinguish the actual appearance of the "same array definition" under different parent objects, and avoid confusion between different parent object instances due to relying only on static JSON paths.
[0052] (2) Changing the field merging rule from "the same array index can be concatenated" to "the associated scope can be proven to allow concatenation" can directly suppress the most dangerous silent misalignment error.
[0053] (3) Different processing methods are adopted for parent broadcast, nested subordinate, explicit association key connection and sibling array unrelated, which can correctly express one-to-many, multi-level subordinate and multi-entity response, rather than forcibly flattening it into a single wide table.
[0054] (4) By using array occurrence vectors, association keys, record group identifiers and mapping plan versions, evidence of the source location of each target record is preserved, making the write results traceable to the parent object and array elements in the response.
[0055] (5) By using cardinality conservation check, problems such as index truncation, illegal Cartesian expansion, non-unique association key and missing parent-child relationship can be identified; even if the HTTP request is successful, it can prevent erroneous data from entering the formal target table.
[0056] (6) DataX’s Reader.Job, Reader.Task, RecordSender, Channel, Writer.Task and plugin configuration mechanism can be reused without requiring changes to DataX’s core transmission protocol, making it easy to implement in engineering.
[0057] (7) Unlike existing breakpoint resume, pagination snapshot consistency and fault recovery mechanisms that focus on “read location and read / write commit”, this invention focuses on “whether the internal fields of a single level response belong to the same business instance”, and the protected objects and technical effects are clearly distinguished. Attached Figure Description
[0058] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.
[0059] Figure 1 This is a schematic diagram illustrating the operation steps of the hierarchical API response anti-misalignment mapping method based on array association scope recognition according to the present invention.
[0060] Figure 2 This is a structural diagram of the hierarchical API response misalignment prevention mapping system based on array association scope recognition, as described in this invention.
[0061] Figure 3 This is a schematic diagram of the overall architecture for preventing misalignment mapping of hierarchical API responses based on DataX in an embodiment of the present invention.
[0062] Figure 4 This is a schematic diagram illustrating the construction of the response node tree and the array occurrence tree in an embodiment of the present invention.
[0063] Figure 5 This is a schematic diagram of the process for comparing ancestor sequences of field arrays and determining the associated scope in an embodiment of the present invention.
[0064] Figure 6 This is a schematic diagram illustrating the process of record group generation, array appearance vector, association key, and cardinality conservation verification in an embodiment of the present invention.
[0065] Figure 7 This is a schematic diagram illustrating the splitting, isolation, and secure writing process when sibling arrays have no natural association, as described in an embodiment of the present invention. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0067] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.
[0068] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."
[0069] like Figures 3-7 As shown, the present invention provides the following technical solution:
[0070] In a first aspect, the present invention provides a hierarchical API response misalignment prevention mapping method based on array-associated scope identification.
[0071] This method is applied to a data synchronization system that includes a DataX reader, a transmission channel, and a writer, and includes the following steps:
[0072] S101, Read the interface configuration and mapping configuration. The interface configuration includes at least the interface address, request method, response format, success condition, and response root path; the mapping configuration includes at least the target entity or target table, target field, source field path, optional entity primary key path, association key path, array association strategy, and target end write strategy.
[0073] S102, read the API response and parse it into a node tree. The node tree includes object nodes, array definition nodes, array occurrence nodes, scalar nodes, and the path, parent node, node type, element index, and primitive value or value reference of each node.
[0074] S103. For each array definition node and its actual occurrence, construct an array occurrence tree. The array occurrence node in the array occurrence tree carries at least the array canonical path, the parent array occurrence node identifier, the current element index, the parent object instance identifier, and the number of array elements; the same array definition node forms different array occurrence nodes under different parent object instances.
[0075] S104. For each field path to be mapped, locate its scalar node or object node, and extract the array ancestor sequence corresponding to the field path; the array ancestor sequence is composed of array occurrence nodes from the response root node to the node where the field is located in hierarchical order.
[0076] S105, for at least two fields to be mapped, compare the ancestor sequences of the arrays to determine the field association scope. The association scope includes at least: same-element association scope, parent broadcast scope, cross-level dependent scope, explicit association key scope, sibling arrays without natural association scope, and scope with uncertain association relationship.
[0077] Specifically, when two fields share the same deepest common array node and the fields are located within the same element instance of that common array node, they are classified as having the same element-related scope. When the first field is not located in any array or is only located in the parent object of the second field's array ancestor, it is classified as having the parent broadcast scope. When the first field is located in the array element pointed to by the proper prefix of the second field's array ancestor sequence, it is classified as having a cross-level dependent scope. When two fields are located in different array branches and have verifiable equality keys or verifiable mapping rules configured, they are classified as having explicit key scope. When two fields are only associated based on the same array position, array length, or traversal order, they are classified as having no natural association scope between sibling arrays. Other cases that do not meet the preset proof conditions are classified as having an uncertain association scope.
[0078] S106, Generate a mapping plan based on field association scope. The mapping plan includes at least a record group, the root scope of the record group, the value scope of each field, field inheritance rules, output entity, association key generation rules, cardinality check rules, and exception handling rules. Fields that are associated with the same element, broadcast from a parent, or belong to the same cross-level subordinate link can be grouped into the same record group; fields in sibling arrays without a natural association scope or with an uncertain association scope must not be grouped into the same record group solely based on array indices.
[0079] S107, traverse the array occurrence tree according to the mapping plan and generate an output Record for each record group. When generating the output Record, use the array occurrence vector corresponding to the root scope of the record group as the positioning context, and extract field values according to the rules of reading the same element, parent broadcasting, cross-level inheritance, or explicit association key matching.
[0080] S108, generate an association key for each output Record. The association key includes at least one of the following: interface endpoint or interface identifier, normalized request condition summary, response entity path, record group identifier, and array occurrence vector; the array occurrence vector includes the element index or instance identifier of the root array occurrence node of the record group and its ancestor array occurrence nodes. Optionally, the association key also includes the source business primary key, the parent entity business primary key, or an explicit association key value.
[0081] S109, Perform cardinality conservation checks on the record groups. Cardinality conservation checks include at least one of the following: the number of output records in the same element group is consistent with the number of elements in the root array; the number of child records after parent broadcasting is consistent with the number of elements in the child array; the number of output records in cross-level subordinate record groups is consistent with the sum of the number of elements in the child arrays under each parent object; the number of matches after explicit association key matching conforms to the preset one-to-one, one-to-many, or many-to-one constraints; the association keys of the target entity are not repeated or the repetition conforms to the preset aggregation rules.
[0082] S110: When the cardinality conservation check passes, the output Record, along with the association key, record group identifier, array occurrence vector summary, mapping plan version, and check status, is written to the DataX Record extended field or task context and sent to the Writer via DataXChannel. The Writer writes the data to the main table, sub-table, edge table, object storage, or other target end according to the target entity, association key, and writing strategy.
[0083] S111 When it is detected that the sibling arrays have no natural association, the association key is not unique, the array cardinality is abnormal, the field type is incompatible, or the association relationship cannot be proven, at least one security action shall be performed: split into different record groups or different target entities, write to the isolation temporary storage area, output the mapping evidence to be confirmed, reject the current configuration release, skip uncertain fields, or terminate the task; it is prohibited to write the field to the same business record only based on the same array index.
[0084] The rules for determining the scope of field associations are as follows:
[0085] To avoid ambiguity in implementation caused by using only natural language descriptions, this invention can be described using the following notation. For field f, its array ancestor sequence is denoted as A(f) = [a1, a2, ..., am], where each ai is an actual occurrence node in the array; for field g, A(g) = [b1, b2, ..., bn]. The longest common prefix of the two is denoted as LCP(A(f), A(g)).
[0086] 1. When A(f) = A(g), and f and g are recursively obtained from the same array element instance, f and g belong to the same element-associated scope.
[0087] 2. When A(f) is a prefix of A(g), or f is located in the ancestor object of the array element pointed to by A(g), f is subject to parent broadcast or cross-level subordination rules for g, and the value of f is inherited according to the array occurrence vector of g.
[0088] 3. When A(f) and A(g) have different first array occurrence nodes after the longest common prefix, and there is no configured verified association key, uniqueness constraint or explicitly defined connection rule between them, f and g belong to sibling arrays without natural association scope.
[0089] 4. When different array branches have configured leftKeyPath, rightKeyPath, joinType and cardinality constraints, and the runtime verifies that the values of leftKeyPath and rightKeyPath satisfy the connection rules and uniqueness conditions, f and g can form a connection record group according to the explicit association key scope; if the verification fails, it will proceed to isolation or splitting.
[0090] 5. Arrays with the same length, array element numbers, similar field names, coincidentally identical values, or appearing on the same response page cannot be used as proof that two array branches are related.
[0091] Secondly, this invention provides a hierarchical API response misalignment prevention mapping system based on array association scope recognition.
[0092] This system includes:
[0093] 1. Interface configuration module, used to store interface address, authentication method, request parameters, success conditions, response root path and pagination parameters;
[0094] 2. Mapping configuration module, used to store source field path, target entity, target field, association key, array association strategy, target end write strategy and mapping plan version;
[0095] 3. Response node tree building module, used to parse JSON, XML or other hierarchical responses and form a node tree;
[0096] 4. The array appearance tree construction module is used to record the specific appearance of each array definition under each parent object instance, the array element index, and the ancestor relationship;
[0097] 5. Scope identification module, used to calculate the array ancestor sequence of a field, the longest common prefix, the nearest common array ancestor, and the field's associated scope;
[0098] 6. Mapping plan generation module, used to organize record groups, field inheritance rules, association key rules and exception strategies according to the associated scope;
[0099] 7. Record group execution module, used to generate DataX internal Records based on the array occurrence vector;
[0100] 8. Cardinality verification module, used to verify the number of inputs and outputs of record groups, the uniqueness of association keys, connection cardinality, and parent-child affiliation;
[0101] 9. Security handling module, used to split, isolate, block or manually confirm fields for which the relationship cannot be proven;
[0102] 10. The Writer adapter module is used to write Records with record group identifiers and association keys to the target main table, sub-table, associated table, temporary storage area or other target end, and return the write status.
[0103] To more clearly illustrate the technical solution of this application, the present invention will be further described below in conjunction with the DataX framework and embodiments.
[0104] Example 1: Extended Architecture of DataX APIReader
[0105] The existing ApiReader includes Reader.Job and Reader.Task. Reader.Job is responsible for reading the address, request method, parameters, request headers, request body, pagination fields, response format, success conditions, and field configurations; Reader.Task initiates an HTTP request, parses the response, and sends the record to the DataX Channel via RecordSender. This embodiment extends the configuration as follows without changing the main Reader—Channel—Writer link:
[0106] associationMode: Association identification mode, including STRICT, KEYED, DECLARED_POSITIONAL and AUTO; among them, STRICT prohibits location association, KEYED only allows connection after verification by association key, DECLARED_POSITIONAL requires users to explicitly specify the business semantics of location association and set verification samples, and AUTO takes precedence over STRICT.
[0107] entityMappings: A list of target entity mappings. Each entity mapping includes entityName, targetTable, rootPath, identityPath, writeMode, and a list of fields.
[0108] fieldMappings: A list of field mappings. Each field mapping includes sourcePath, targetColumn, expectedType, nullPolicy, and valueTransform.
[0109] joinRules: Optional explicit join rules across arrays, including leftPath, leftKeyPath, rightPath, rightKeyPath, joinType, cardinality, and unmatchedPolicy.
[0110] cardinalityPolicy: Cardinality verification policy, including EXACT, AT_LEAST_ONCE, UNIQUE_KEY, DECLARED_RATIO, and OFF; in production environments, it should not be configured as OFF by default.
[0111] associationLedger: Used to store the response fingerprint, mapping plan version, record group, association key digest, array occurrence vector digest, and location of anomalous evidence.
[0112] Example configuration is as follows:
[0113] {
[0114] "name": "apireader",
[0115] "parameter": {
[0116] "protocol": "https",
[0117] "address": "https: / / api.example.com / v1 / orders / 20260921",
[0118] "method": "get",
[0119] "response": "json",
[0120] "responsefield": "code",
[0121] "successflag": "200",
[0122] "associationMode": "STRICT",
[0123] "associationLedger": "jdbc:postgresql: / / meta-db / api_association_ledger",
[0124] "entityMappings": [
[0125] {
[0126] "entityName": "order",
[0127] "targetTable": "ods_order",
[0128] "rootPath": "data.orders[]",
[0129] "identityPath": "id",
[0130] "writeMode": "UPSERT",
[0131] "fields": [
[0132] {"sourcePath": "id", "targetColumn": "order_id"},
[0133] {"sourcePath": "customer.name", "targetColumn": "customer_name"},
[0134] {"sourcePath": "totalAmount", "targetColumn": "total_amount"} ]
[0136] },
[0137] {
[0138] "entityName": "order_item",
[0139] "targetTable": "ods_order_item",
[0140] "rootPath": "data.orders[].items[]",
[0141] "identityPath": "sku",
[0142] "writeMode": "UPSERT",
[0143] "fields": [
[0144] {"sourcePath": "sku", "targetColumn": "sku"},
[0145] {"sourcePath": "quantity", "targetColumn": "quantity"},
[0146] {"sourcePath": "order.id", "targetColumn": "order_id"} ]
[0148] },
[0149] {
[0150] "entityName": "order_coupon",
[0151] "targetTable": "ods_order_coupon",
[0152] "rootPath": "data.orders[].coupons[]",
[0153] "identityPath": "code",
[0154] "writeMode": "UPSERT",
[0155] "fields":
[0156] {"sourcePath": "code", "targetColumn": "coupon_code"},
[0157] {"sourcePath": "order.id", "targetColumn": "order_id"}
[0159] }
[0161] }
[0162] }
[0163] Herein, order.id indicates inheriting order.id from the array element corresponding to the current rootPath to the scope of the specified ancestor entity. items[] and coupons[] are respectively declared as different entities, instead of being forcibly spliced into the same wide table row.
[0164] Second Embodiment: Array Occurrence Tree Construction
[0165] Assume that the interface returns the following simplified response:
[0166] {
[0167] "code": "200",
[0168] "data": {
[0169] "orders":
[0170] {
[0171] "id": "O100",
[0172] "customer": {"name": "Party A"},
[0173] "items":
[0174] {"sku": "A", "quantity": 2},
[0175] {"sku": "B", "quantity": 1}
[0176] ,
[0177] "coupons":
[0178] {"code": "C10"}
[0180] },
[0181] {
[0182] "id": "O200",
[0183] "customer": {"name": "Party B"},
[0184] "items":
[0185] {"sku": "D", "quantity": 3}
[0186] ,
[0187] "coupons":
[0188] {"code": "C20"},
[0189] {"code": "C30"}
[0191] }
[0193] }
[0194] }
[0195] The array definition paths in the response node tree include data.orders, data.orders.items, and data.orders.coupons. The array occurrence tree includes:
[0196] orders#0: path data.orders, element sequence number 0;
[0197] orders#0 / items#0: path data.orders.items, parent occurrence orders#0, element sequence number 0;
[0198] orders#0 / items#1: path data.orders.items, parent occurrence orders#0, element sequence number 1;
[0199] orders#0 / coupons#0: path data.orders.coupons, parent occurrence orders#0, element sequence number 0;
[0200] orders#1: path data.orders, element sequence number 1;
[0201] orders#1 / items#0: The path is data.orders.items, the parent element is orders#1, and the element's sequence number is 0.
[0202] orders#1 / coupons#0 and orders#1 / coupons#1: The path is data.orders.coupons, and the parent element is orders#1. The element numbers are 0 and 1 respectively.
[0203] As can be seen, item#0 exists under orders#0 and orders#1 respectively. Although they have the same static path, they are different actual occurrences in different parent entities. The array-based tree must distinguish this difference.
[0204] Example 3: Determining the Scope of Association and Mapping Scheme
[0205] For the `order_item` entity, the array ancestor sequence of the fields `sku` and `quantity` is `[orders#k,items#j]`, which belongs to the scope of the same element; the array ancestor sequence of the field `order.id` is `[orders#k]`, which is a prefix of the aforementioned sequence. Therefore, the order identifier is propagated to each `items` element according to the parent broadcasting rule.
[0206] For the misconfigured wide table entity `order_item_coupon`, the array ancestor sequence of its `sku` field is `[orders#k, items#j]`, and the array ancestor sequence of its `code` field is `[orders#k, coupons#t]`. Their longest common prefix is `[orders#k]`. The first array occurrence after this is different between `items#j` and `coupons#t`, and there are no `joinRules` providing a verifiable join key. Therefore, it is determined that the sibling arrays have no natural join scope. The system must not combine `sku` and `code` into a record using the format `j=t`.
[0207] In STRICT mode, the mapping plan generation module outputs: splitting the current wide table configuration into two record groups, order_item and order_coupon; if the target only allows one wide table, the task is marked as ASSOCIATION_UNPROVABLE, evidence is saved, and publishing is stopped. In KEYED mode, if both items[] and coupons[] contain promotionId, the user can configure promotionId as the association key and declare a one-to-one or one-to-many cardinality; the system first verifies the existence of the key value and that the cardinality conforms to the declaration after the join, and then generates the join record group.
[0208] Example 4: Association Key and Cardinality Conservation Verification
[0209] For the order_item record corresponding to orders#0 / items#1, the association key can be generated as follows:
[0210] associationKey = Hash(
[0211] endpointFingerprint + normalizedRequestFingerprint +
[0212] entityName + mappingPlanVersion +
[0213] orderBusinessId + "|orders:0|items:1" )
[0215] For responses without a business primary key, the vector appearing in the array can still be used as a traceable location basis within this response; this vector should not be used alone as an idempotent key for cross-synchronization, but should be combined with the normalized summary of the response content, the request summary, or the source version identifier.
[0216] In this example, the `order` entity should output 2 records; the `order_item` entity should output 3 records; and the `order_coupon` entity should output 3 records. For each order with node `r`, the following condition must be met:
[0217] count(order_item where parentOccurrence=r) = count(items under r)
[0218] count(order_coupon where parentOccurrence=r) = count(coupons under r)
[0219] If items and coupons are incorrectly merged according to the same index, order O100 will only generate 1 item and lose SKU=B, and order O200 will only generate 1 item and lose coupon=C30. If cross-combination is used, O100 will generate 2 items and O200 will generate 2 items, neither of which represents the actual business situation. This invention can detect these two anomalies through the above cardinality constraint.
[0220] Example 5: Explicit Association Key Connection
[0221] A device interface returns both `sensors[]` and `thresholds[]` arrays under the same device object, each containing a `sensorId`. User configuration:
[0222] leftPath=data.devices[].sensors[]; leftKeyPath=sensorId;
[0223] rightPath=data.devices[].thresholds[]; rightKeyPath=sensorId; joinType=LEFT; cardinality=ONE_TO_ONE; unmatchedPolicy=ISOLATE.
[0224] The scope identification module first confirms that the two arrays belong to the same device node, and then establishes the key-value index. If each sensorId appears at most once in thresholds[], then merging by sensorId to generate a sensor_threshold record group is allowed; if thresholds[] contains duplicate sensorIds, missing sensorIds, or the same key appears across different devices, the cardinality proof module outputs JOIN_KEY_NON_UNIQUE, JOIN_KEY_MISSING, or PARENT_SCOPE_VIOLATION, and isolates the abnormal records. This process differs from joining by array index.
[0225] Example 6: Integration with DataX Transport and Writer
[0226] After successfully parsing the response, Reader.Task sequentially calls ResponseNodeTreeBuilder, ArrayOccurrenceTreeBuilder, AssociationScopeResolver, and MappingPlanExecutor. MappingPlanExecutor creates a DataX Record for each record group and sends it via RecordSender. Optionally, the Record carries the following metadata as extended columns or within the task context:
[0227] api_mapping_plan_version;
[0228] api_entity_name;
[0229] api_record_group_id;
[0230] api_association_key;
[0231] api_occurrence_vector_digest;
[0232] api_parent_association_key;
[0233] api_cardinality_validation_state;
[0234] api_response_shape_fingerprint.
[0235] The Writer.Task selects the target table based on the api_entity_name and performs MERGE, UPSERT, unique key deduplication, or temporary write based on the api_association_key. For main-sub-table writes, a batch of responses can be written to the temporary storage area first, and then published after the main table, all sub-tables, and cardinality constraints have all passed. If the target Writer does not support multi-entity writes, the Reader can split the task into multiple DataX subtasks by record group or send them to a unified temporary storage table for distribution by the subsequent publishing component.
[0236] Example 7: Adapting XML Responses
[0237] For XML responses, repeating child elements with the same name, elements with the `maxOccurs` attribute, or collection elements identified according to configuration are used as array definition nodes; an array occurrence node is created for each actual repeating element. Field paths can be represented using XPath or normalized dot paths. The array ancestor sequence, associated scope, array occurrence vector, and cardinality conservation rules are the same as in the JSON implementation; therefore, this invention is not limited to the JSON format.
[0238] Example 8: Anomaly Classification and Security Strategy
[0239] Exceptions include at least: SIBLING_ARRAY_UNPROVABLE, POSITIONAL_JOIN_REJECTED, JOIN_KEY_MISSING, JOIN_KEY_NON_UNIQUE, CARDINALITY_MISMATCH, PARENT_SCOPE_VIOLATION, FIELD_SCOPE_NOT_FOUND and TYPE_CONVERSION_FAILED.
[0240] For SIBLING_ARRAY_UNPROVABLE, the default behavior is to split the entity; for POSITIONAL_JOIN_REJECTED, writing to the formal table is prohibited; for JOIN_KEY_NON_UNIQUE and CARDINALITY_MISMATCH, writing to the isolation area and saving the array occurrence vector and key-value digest; for FIELD_SCOPE_NOT_FOUND, filling, isolating, or terminating according to the nullPolicy; for TYPE_CONVERSION_FAILED, the DataX dirty data handling mechanism is used, but other record groups that have passed the scope check must not be changed.
[0241] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.
[0242] Embodiments of this application also disclose an electronic device, including: a processor, a communication interface, a memory for storing a processor-executable computer program, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor executes the executable computer program to implement the steps of the above-described hierarchical API response anti-misalignment mapping method based on array-associated scope identification.
[0243] The operations described in this application can be implemented by writing computer program code using one or more programming languages or a combination thereof.
[0244] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the hierarchical API response anti-misalignment mapping method based on array association scope identification disclosed in this application.
[0245] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Such computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.
[0246] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the hierarchical API response misalignment prevention mapping method based on array association scope identification disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.
[0247] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.
[0248] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A hierarchical API response error prevention and position mapping method based on array association scope recognition, applied to a DataX data synchronization system, characterized in that, include: S1, read the interface configuration and mapping configuration, parse the API response into a node tree, and build an array occurrence tree for array definition nodes and their occurrence under the parent object instance; S2, extract and compare the array ancestor sequence of each field to be mapped to determine the field association scope; Generate a mapping plan and assemble record groups based on the field association scope. Only fields whose association relationship can be proven should be included in the same record group. It is prohibited to include fields in the same record group by array subscript. S3, traverse the array occurrence tree according to the mapping plan to generate the output Record and the association key containing the array occurrence vector; S4 performs a cardinality check on the record group; if the check passes, the output Record, its associated key, and the check status are written to the DataX Record extended field or task context and sent to the writing end via the transmission channel; if the check fails or the association cannot be proven, security measures are taken.
2. The method of claim 1, wherein, The array occurrence node in the array occurrence tree carries at least the array canonical path, the parent array occurrence node identifier, the current element index, the parent object instance identifier, and the number of array elements; the same array definition node forms different array occurrence nodes under different parent object instances.
3. The method of claim 1, wherein, In step S2, the field association scope is determined based on the longest common prefix and the nearest common array ancestor of the field array ancestor sequence, and includes at least the same element association scope, parent broadcast scope, cross-level dependent scope, explicit association key scope, sibling array no natural association scope, and association relationship uncertain scope. When two fields share the same deepest common array node, and the fields are located within the same element instance of that common array node, they are considered to have the same element associative scope. When the first field is not located in any array or is only located in the parent object of the second field's array ancestor, it is considered to have the parent broadcast scope. When the first field is located in the array element pointed to by the proper prefix of the second field's array ancestor sequence, it is considered to have the cross-level dependent scope. When two fields are located in different array branches and have verifiable equality keys or verifiable mapping rules, they are considered to have the explicit key scope. When two fields are only associated based on the same array position, array length, or traversal order, they are considered to have the sibling array without natural association scope. Other cases that do not meet the preset proof conditions are considered to have uncertain association scope.
4. The method of claim 1, wherein, In step S2, the mapping plan includes at least a record group, a root scope of the record group, the value scope of each field, field inheritance rules, output entities, association key generation rules, cardinality verification rules, and exception handling rules; fields in the same element association scope, parent broadcast scope, or the same cross-layer subordinate link are compiled into the same record group; fields in sibling arrays without a natural association scope or with an uncertain association relationship must not be combined into the same record group solely based on array subscripts.
5. The method of claim 1, wherein, In step S3, when generating the output Record, the array occurrence vector corresponding to the root scope of the record group is used as the positioning context, and the field values are extracted according to the rules of reading the same element, parent broadcasting, cross-level inheritance, or explicit association key matching.
6. The method of claim 1, wherein, The association key includes at least one of the following: interface endpoint or interface identifier, normalized request condition summary, response entity path, record group identifier, and array occurrence vector; the array occurrence vector includes the element sequence number or instance identifier of the record group root array occurrence node and its ancestor array occurrence nodes; the association key also includes the source business primary key, the parent entity business primary key, or an explicit association key value; the array occurrence vector is not used alone as an idempotent key for cross-synchronization, but is used in combination with the response content normalized summary, request summary, or source version identifier.
7. The method of claim 1, wherein, In step S4, the cardinality check includes at least one of the following: the number of output records of the same element group is consistent with the number of elements in the root array; the number of child records after the parent broadcast is consistent with the number of elements in the child array; The number of outputs for cross-level subordinate record groups is consistent with the sum of the number of elements in the subarrays of each parent object; The number of matches after explicit association key matching conforms to the preset one-to-one, one-to-many, or many-to-one constraints; the association keys of the target entity are either not repeated or repeat in accordance with the preset aggregation rules.
8. The method of claim 1, wherein, In step S4, the security handling includes at least one of the following: splitting into different record groups or different target entities, writing to an isolated temporary storage area, outputting mapping evidence to be confirmed, rejecting the current configuration release, skipping uncertain fields, or terminating the task; the security handling is triggered when it is detected that the sibling arrays have no natural association, the association key is not unique, the array cardinality is abnormal, the field types are incompatible, or the association relationship cannot be proven; wherein, the same array length, the same array element sequence number, similar field names, the same values by chance, or appearing in the same response page, shall not be used as proof that the two array branches can be associated.
9. A hierarchical API response error prevention mapping system based on array associative scope recognition, characterized in that, The system runtime implements the hierarchical API response misalignment prevention mapping method based on array association scope identification as described in any one of claims 1 to 8, including: The interface configuration module is used to store the interface address, authentication method, request parameters, success conditions, response root path, and pagination parameters. The mapping configuration module is used to store source field paths, target entities, target fields, association keys, array association strategies, target end write strategies, and mapping plan versions. The response node tree building module is used to parse JSON, XML, or other hierarchical responses and form a node tree. The array appearance tree building module is used to record the specific occurrence of each array definition under each parent object instance, the array element index, and the ancestor relationship; The scope identification module is used to calculate the array ancestor sequence, longest common prefix, nearest common array ancestor, and field-related scope of a field. The mapping plan generation module is used to organize record groups, field inheritance rules, association key rules, and exception policies according to the associated scope; The record group execution module is used to generate the DataX internal Record based on the array occurrence vector; The cardinality proof module is used to perform checks on the number of inputs and outputs of record groups, the uniqueness of association keys, the cardinality of connections, and parent-child attribution. The security handling module is used to split, isolate, block, or manually confirm fields for which a relationship cannot be proven. The Writer adapter module is used to write Records with record group identifiers and association keys to the target main table, sub-table, associated table, temporary storage area, or other target end, and to return the write status.
10. The system of claim 9, wherein, The scope identification module is configured as follows: when the deepest common arrays of two fields have the same node and are located within the same element instance, it is determined to be an element-related scope; when the first field is not located in any array or is only located in the parent object of the ancestor of the second field array, it is determined to be a parent broadcast scope; when the first field is located in the array element pointed to by the proper prefix of the ancestor sequence of the second field array, it is determined to be a cross-level dependent scope; when the two fields are located in different arrays with branches and there is a verifiable equal association key or mapping rule, it is determined to be an explicit association key scope; when the only association criterion is the same array position, array length, or traversal order, it is determined to be a sibling array without natural association scope; and in other cases, it is determined to be an indeterminate association scope.
