Data concurrent transmission processing method and device, equipment and storage medium
By generating data identifiers and mapping fields when system A and system B connect, and by using a delayed consumption strategy and interface call mechanism, the problem of downstream data not being able to be associated with upstream data during asynchronous execution is solved, thus achieving high efficiency and reliability in data transmission.
Patent Information
- Application Number
- CN202511634994.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-01-23
AI Technical Summary
When system A and system B are interfacing with each other, the asynchronous upstream data and downstream data transmissions are triggered at the same time, resulting in the upstream data not being saved to system B and the downstream data being unable to be associated with the upstream data. Existing technologies cannot effectively solve this problem.
By generating identifiers for upstream and downstream data, mapping fields, and querying external identifiers in the target system, a delayed consumption strategy and interface call mechanism are adopted to ensure that downstream data is associated with upstream data. This includes delayed query, retry counting, anomaly tracking, and backtracking mechanisms to optimize data transmission strategies.
This effectively avoids downstream data association failures, improves data transmission efficiency, ensures that upstream and downstream data are correctly associated in the target system, and achieves reliability and consistency in data transmission.
Smart Images

Figure CN121388024A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data concurrent transmission processing method, apparatus, device and storage medium. Background Technology
[0002] An information application system typically provides its own interfaces for external calls. However, different systems have inconsistent interface specifications, and there's no function to call all external system interfaces. When an enterprise wants to interface its information application system's data with other external systems, it faces problems such as both systems having interfaces for external calls, but lacking the necessary functionality to call this data for integration, the enterprise lacking the necessary development personnel, and the workload of developing separate interfaces for each system being integrated. The existing product, Integration Vision, is a data transfer platform. Because the information application system lacks a function to notify external systems when new data is generated, Integration Vision proactively calls the information application system's interfaces periodically to check for new data generation, encapsulates it into the data structure required by the target system's interface, and transmits it to the target system.
[0003] When system A and system B connect their data, data from system A needs to be transferred to the relevant modules in system B. However, the data in each module has upstream and downstream relationships. Existing technical solutions have the following problems: When multiple interface data calls are executed asynchronously (data transmission between different module interfaces occurs simultaneously without interference), upstream data transmission and downstream data transmission may be triggered at the same time. As a result, when system B processes downstream data, the upstream data has not yet been saved from system A to system B, and thus the data cannot be correlated. Summary of the Invention
[0004] This application provides a data concurrent transmission processing method, apparatus, device, and storage medium to solve the problem of lost correlation between upstream and downstream data when data calls are executed asynchronously in related technologies.
[0005] The first aspect of this application provides a method for concurrent data transmission processing, the method comprising: The first upstream data and the first downstream data generated by the source system are identified to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data. During the process of pushing data from the source system to the target system, the data identifier and the parent identifier are mapped to fields respectively to generate corresponding external identifiers and external parent identifiers; In the target system, query whether the external identifier exists in the target system based on the external parent identifier; If no external identifier corresponding to the second upstream data is found, the second upstream data will be queried periodically using a delayed consumption strategy. When the number of queries exceeds a preset trigger threshold, the interface of the source system is called to obtain the second upstream data, and the second upstream dataset is associated with the second downstream data.
[0006] Optionally, in the first implementation of the first aspect of this application, the step of periodically querying the second upstream data using a delayed consumption strategy if no external identifier corresponding to the second upstream data is found includes: If no external identifier corresponding to the second upstream data is found, the target message is sent to the message queue. Execute tasks according to the messages corresponding to the target messages, and perform periodic queries on the second upstream data using a delayed consumption strategy; If an external identifier corresponding to the second upstream data is found, the target message is confirmed to be consumed and the information of the second upstream data is updated in the second downstream data of the target system. If no external identifier corresponding to the second upstream data is found, the retry count of the target message is incremented, and the target message is rewritten into the delay queue.
[0007] Optionally, in the second implementation of the first aspect of this application, the step of calling the interface of the source system to obtain the second upstream data when the number of queries exceeds a preset trigger threshold, and establishing an association between the second upstream dataset and the second downstream data, includes: The retry count value is compared with a preset trigger threshold; When the retry count reaches the trigger threshold, the interface of the source system is called to obtain the first upstream data, and the upstream dataset is obtained. The second downstream data is compared based on the upstream dataset; If a data item corresponding to the external parent identifier exists in the upstream dataset, the second upstream data is generated by mapping the fields of the data item, and a correspondence is established between the second upstream data and the second downstream data.
[0008] Optionally, in the third implementation of the first aspect of this application, after the step of incrementing the retry count of the target message and rewriting the target message into the delay queue if the external identifier corresponding to the second upstream data is not found, the method further includes: When the retry count reaches the preset maximum number of times, the historical execution records of the target message are aggregated and analyzed to obtain an abnormal feature vector. By performing feature hierarchical mapping on the abnormal feature vectors, a multidimensional anomaly tracking index structure is constructed. Based on the multidimensional anomaly tracking index structure, clustering calculations are performed on anomaly records belonging to the same parent task chain to generate anomaly task clusters. By comparing the time intervals and message retry distribution in the abnormal task cluster, suspicious interface blocking nodes are identified, and task tracing records with blocking identifiers are generated. Once the task tracking record is written to the exception log table, a backtracking task instance is automatically generated based on the node path corresponding to the blocking identifier, and the backtracking task instance is written to the backtracking queue.
[0009] Optionally, in the fourth implementation of the first aspect of this application, after the step of automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance into the backtracking queue after the task tracking record is written to the exception log table, the method further includes: Based on the completed data association records in the target system and the original data records in the source system, the task identifier, external identifier, and external parent identifier of each task are compared to generate a task correspondence matrix; By cross-validating the task correspondence matrix and the anomaly tracking index structure, the task status distribution within the same data transmission cycle is determined, and a task status mapping table is generated. Based on the number of task executions, delay intervals, and anomaly identifiers recorded in the task status mapping table, the confidence parameters of each task node are weighted and calculated to obtain the node confidence update set. By normalizing the node confidence update set, a task scheduling parameter set is generated, and the task scheduling strategy of the integrated vision is updated according to the task scheduling parameter set.
[0010] Optionally, in the fifth implementation of the first aspect of this application, after the step of automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance into the backtracking queue after the task tracking record is written to the exception log table, the method further includes: Based on the backtracking task instance, the bidirectional interface between the source system and the target system is invoked to perform data consistency verification on the external identifier and external parent identifier of the upstream and downstream data, and generate an integrity comparison table. By analyzing the integrity comparison table item by item, abnormal datasets in the target system are identified. Based on the abnormal dataset, an interface compensation call is initiated for the missing upstream data entries, and a manual verification task package is generated for the abnormal records that are still not complete. Write the task identifier, interface path, and comparison results of the manual verification task package into the manual processing database.
[0011] Optionally, in a sixth implementation of the first aspect of this application, the method further includes: The node confidence update sets of multiple parallel tasks are sorted according to the task scheduling parameter set to generate a task priority sequence. By performing hierarchical aggregation on the task priority sequence, tasks with confidence levels higher than a preset confidence threshold are assigned to the high-speed execution queue, tasks with confidence levels in the middle range are assigned to the standard execution queue, and tasks with confidence levels lower than the minimum threshold are assigned to the delayed execution queue. Based on the task distribution status of the high-speed execution queue, standard execution queue, and delayed execution queue, the data transmission thread pool of the integrated vision is dynamically allocated to generate a resource allocation matrix; By scheduling and controlling the resource allocation matrix, the task execution order table for the next transmission cycle is determined.
[0012] A second aspect of this application provides a data concurrent transmission processing apparatus, which is used to implement a data concurrent transmission processing method. The data concurrent transmission processing apparatus includes: The generation module is used to generate identifiers for the first upstream data and the first downstream data generated by the source system, so as to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data. The mapping module is used to perform field mapping between the data identifier and the parent identifier respectively during the process of pushing data from the source system to the target system, and generate corresponding external identifiers and external parent identifiers; The query module is used to query whether the external identifier exists in the target system based on the external parent identifier; The query module is also used to periodically query the second upstream data if no external identifier corresponding to the second upstream data is found, using a delayed consumption strategy. The association module is used to call the interface of the source system to obtain the second upstream data when the number of queries exceeds a preset trigger threshold, and to establish an association between the second upstream dataset and the second downstream data.
[0013] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the data concurrent transmission processing method provided in the first aspect of this application.
[0014] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the data concurrent transmission processing method provided in the first aspect of this application.
[0015] In summary, according to the data concurrent transmission processing method, apparatus, device, and storage medium provided in this application, the first upstream data and the first downstream data generated by the source system are identified to obtain a data identifier for the first upstream data and a parent identifier for the corresponding first downstream data. During the process of pushing the data from the source system to the target system, the data identifier and the parent identifier are mapped to fields to generate corresponding external identifiers and external parent identifiers. In the target system, the external parent identifier is used to query whether the external identifier exists. If no external identifier corresponding to the second upstream data is found, the second upstream data is periodically queried using a delayed consumption strategy. When the number of queries exceeds a preset trigger threshold, the interface of the source system is called to obtain the second upstream data, and the second upstream dataset is associated with the second downstream data. Through the implementation of this application, when the downstream data from the source system arrives at the target system before the upstream data, the corresponding external identifier can be queried in the target system using the external parent identifier to complete the association between the upstream and downstream data in the target system. This effectively avoids downstream data association failures and improves data transmission efficiency. Attached Figure Description
[0016] Figure 1 A flowchart illustrating the concurrent data transmission processing method provided in this application embodiment; Figure 2 A schematic diagram of the program modules of the data concurrent transmission processing apparatus provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0017] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] To address the issue of lost correlation between upstream and downstream data during asynchronous data retrieval in related technologies, embodiments of this application provide a method for concurrent data transmission processing, such as... Figure 1This is a flowchart illustrating the concurrent data transmission processing method provided in this embodiment. The concurrent data transmission processing method includes the following steps: Step 110: Generate identifiers for the first upstream data and the first downstream data generated by the source system to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data.
[0019] Specifically, during concurrent data transmission, the first step involves generating identifiers for the first upstream and downstream data generated by the source system. Each data entry is uniquely marked to ensure identifiability and traceability in subsequent data transmission and association. This identifier generation process is typically accomplished using an internal task numbering algorithm. A data identifier is generated for each upstream data entry, and a parent identifier is generated for the corresponding downstream data entry, thus forming a structured hierarchical relationship. Data identifiers can be hashed based on the primary key information, generation time, and data category of the source system. Parent identifiers represent the logical dependency relationship between downstream data and its upstream data. In this way, even during asynchronous transmission between different modules or interfaces, the system can still determine the logical link between upstream and downstream data based on the correspondence between data identifiers and parent identifiers, laying the foundation for subsequent field mapping and data matching.
[0020] Step 120: During the process of pushing data from the source system to the target system, perform field mapping on the data identifier and the parent identifier respectively to generate the corresponding external identifier and external parent identifier.
[0021] Specifically, after the data identifiers are generated, the data from the source system is pushed to the target system. During transmission, field mapping is required. Field mapping converts the data identifiers and parent identifiers from the source system into field forms that the target system can recognize, i.e., generating external identifiers and external parent identifiers. External identifiers are typically used to record the unique number of the original data in the source system, while external parent identifiers are used to store the parent number of the corresponding upstream data in the downstream data of the source system. This mapping process is implemented through data mapping rules in the interface protocol. When the system receives data packets, it parses the field names, types, and value ranges, and maps the source fields to the target field structure. In this way, the target system can retain the parent-child relationship information of the source system when saving downstream data. Even if the upstream data has not yet been synchronized, it can pre-record its corresponding external parent identifier, thus enabling automatic association when upstream data is supplemented in the future.
[0022] Step 130: In the target system, query whether the external identifier exists in the target system based on the external parent identifier.
[0023] Specifically, after data is transmitted to the target system, the system queries the target system to see if a matching external identifier exists within the target system, based on the generated external parent identifier. This process involves data matching and association detection. The target system searches its own database or cache for the external identifier entry corresponding to the external parent identifier using an index retrieval method. If the query result exists, it indicates that the upstream data has been transmitted to the system and has a referential relationship. At this point, the system can find the corresponding upstream data record based on the external parent identifier and update the association field in the downstream data. If the query result is empty, it means that the upstream data has not yet arrived or has not been processed by the system, and the data match has not been established. This query method effectively prevents the problem of downstream data blindly associating with incorrect upstream data under asynchronous concurrency conditions, ensuring the correctness and uniqueness of the association relationship.
[0024] Step 140: If no external identifier corresponding to the second upstream data is found, the second upstream data is queried periodically using a delayed consumption strategy.
[0025] Specifically, when the target system fails to find an external identifier corresponding to the second upstream data after querying the external parent identifier, the system will trigger a delayed consumption strategy based on an asynchronous transmission mechanism to periodically review the upstream record of that data. The delayed consumption strategy is implemented through a message queue. Its core principle is to set a delay time parameter in the message, causing the message to enter a dormant state before being consumed and automatically triggering the query after the time expires. The system generates a target message object based on the current query status. The target message contains information such as the external parent identifier, task identifier, query timestamp, and retry count, and writes it to the delayed queue. After the message reaches the set time, the consumer module automatically reads it and executes the query operation. Through the cyclical execution of the delayed consumption strategy, even if upstream data arrives in batches at different times, the system can capture its external identifier and complete the data association in subsequent queries, avoiding the loss of association due to differences in the order of transmission.
[0026] Step 150: When the number of queries exceeds the preset trigger threshold, call the interface of the source system to obtain the second upstream data, and establish a relationship between the second upstream dataset and the second downstream data.
[0027] Specifically, when the system detects that the number of queries for the target message has exceeded a preset trigger threshold, it indicates that the corresponding external identifier has still not been obtained after multiple delayed queries. The system determines that the upstream data may have failed to be transmitted normally due to interface blocking or synchronization anomalies. At this time, it will actively call the source system interface to obtain the second upstream data. The interface call process uses the external parent identifier as the query parameter and pulls the corresponding upstream dataset from the source system through a controlled API request. After receiving the dataset returned by the source system, the system performs structured parsing and field validation on the data to ensure that the data format, type, and integrity meet expectations. Then, it converts the fields in the source system to correspond to the target system structure through field mapping rules, generating a second upstream data object that conforms to the target system's standards. The system then compares the second downstream data with the external parent identifier as the key. If a match exists in the downstream data, an update operation is performed in the database to establish a relationship between the second upstream data and the second downstream data.
[0028] In one optional implementation of this embodiment, if no external identifier corresponding to the second upstream data is found, the step of periodically querying the second upstream data using a delayed consumption strategy includes: if no external identifier corresponding to the second upstream data is found, sending a target message to a message queue; executing a task according to the message corresponding to the target message, and periodically querying the second upstream data using a delayed consumption strategy; if an external identifier corresponding to the second upstream data is found, confirming the consumption of the target message and updating the information of the second upstream data to the second downstream data of the target system; if no external identifier corresponding to the second upstream data is found, incrementing the retry count value of the target message and rewriting the target message into the delayed queue.
[0029] In this embodiment, during the asynchronous data transmission process, when the target system fails to find an external identifier corresponding to the second upstream data after querying based on the external parent identifier, the system immediately triggers a write operation to the message queue. A message queue is a middleware with asynchronous communication characteristics used to transfer message objects between different systems or modules. The message object here is called the target message, which contains necessary information related to the data association process. For example, it queries upstream data X based on the external parent ID of downstream data Y, updates the information of upstream data X in downstream data Y, completes the upstream and downstream data association, and records the number of operations n. The message queue can temporarily store the target message, allowing the main thread of data transmission to remain unblocked, thus ensuring the continuity of concurrent transmission. After the target message is written, the system generates a corresponding execution task for the message and assigns it to the delayed consumption module. Delayed consumption is a mechanism built based on the time characteristics of message middleware. By setting a delay time parameter in the message header, the message remains dormant before being consumed, automatically triggering a query operation again after a specified time interval. This delay mechanism can be implemented through a dead-letter exchange in the message queue or a scheduled task, enabling the system to periodically query data that has not yet been successfully matched. Delayed consumption can be achieved through the TTL (Time To Live) parameter, which means that after the message's lifespan expires, it is automatically forwarded to the delay queue, received and reprocessed by the consumer module. In this way, the system can perform periodic checks on data that has not been associated without affecting the main data transmission task, thereby automatically restoring the association after the upstream data is synchronized later. When the delay time arrives, the consumer module begins to consume the target message, that is, it queries the target system again based on the external parent identifier carried in the target message to determine whether the external identifier of the second upstream data has been written. If a matching result is found, it means that the upstream data has been transmitted or supplemented in the target system, and the system will confirm the consumption of the target message. Confirming consumption means removing the message from the queue and marking it as processed to prevent duplicate consumption. Subsequently, the system will update the core fields of the second upstream data to the corresponding second downstream data record according to the association information recorded in the message to complete the data association supplementation operation. This update process can be implemented through foreign key mapping or update statements in the database. For example, it matches the external parent identifier in the downstream data with the external identifier in the upstream data and updates the associated fields. For instance, if the order master data is the upstream record and the order details are the downstream record, when the details data is written first and the master data is entered later, the system can automatically complete the logical reference between the master and child tables by matching the external identifier field. If, after delayed consumption, a second query still does not find an external identifier corresponding to the second upstream data, the system will not directly terminate processing but will increment the retry count for the target message. The retry count is an internal parameter of the message object used to record the number of times the message has been queried repeatedly.The system uses this parameter to determine if the message count has reached the maximum retry threshold to prevent infinite loops. After the count increments, the system recalculates the time interval for the next delayed consumption and rewrites the target message into the delayed queue for later execution.
[0030] In one optional implementation of this embodiment, the step of calling the interface of the source system to obtain second upstream data when the number of queries exceeds a preset trigger threshold, and establishing an association between the second upstream dataset and the second downstream data, includes: comparing the retry count value with the preset trigger threshold; when the retry count value reaches the trigger threshold, calling the interface of the source system to obtain first upstream data to obtain an upstream dataset; comparing the second downstream data according to the upstream dataset; if there is a data item in the upstream dataset corresponding to an external parent identifier, then generating second upstream data by mapping the fields of the data item, and establishing a correspondence between the second upstream data and the second downstream data.
[0031] In this embodiment, when the retry count of the target message is compared with a preset trigger threshold and it is determined that the threshold has been reached, Integrated Vision initiates an interface call to the source system using the external parent identifier carried in the message as a key query parameter. This could be an RPC (Remote Procedure Call). The interface call must carry authentication information and a unique request ID to support idempotency. The interface response is returned in structured JSON or XML format, and the upstream dataset is the data array returned by the interface that conforms to the agreed-upon pattern. Upon receiving the response, the returned data is first validated. Validation includes the response status code, message integrity verification (e.g., verifying field signatures or hashes), and consistency with the agreed-upon pattern. Once the validation passes, the upstream data is parsed and organized into a comparable dataset according to record entries. Subsequently, the obtained upstream dataset is compared with the second downstream data in the target system. The comparison uses the external parent identifier as the primary key for matching, and cross-validation can be performed based on other auxiliary keys such as timestamps, business order numbers, or unique identifiers generated by the source system to reduce the probability of false matches. The comparison rules are driven by a mapping rule table, which records the mapping relationship from source fields to target fields, data type conversion rules, and default value strategies in case of missing data. When a data item corresponding to an external parent identifier is found in the upstream dataset, field mapping and type conversion operations are performed on the fields of that data item according to the mapping rules. Field mapping includes direct mapping, combined mapping (obtaining the target field value through concatenation or calculation), and derived mapping (calculating the target field from multiple source fields using functions or rules). Formatting or time zone adjustment is performed on numeric, date, and other types of fields. After mapping, a second upstream data record is generated in the target system using transaction control. Transaction control must include atomic operations for creating the upstream record and updating the downstream record to avoid inconsistencies caused by concurrent writes. Version numbers or optimistic locking fields can be introduced into the transaction for concurrent conflict detection. The generated upstream record should include an external identifier field for subsequent comparison, and also record the data source identifier and API call metadata to support auditing. Subsequently, the second downstream data in the target system is associated and bound, that is, the associated fields of the downstream records are written into the primary key or external identifier of the upstream records, and the status identifier and timestamp of the downstream records are updated. After the database update is completed, the transaction is committed and a detailed record of this supplementary entry and association is written to the operation log. To ensure the traceability and idempotency of the operation, the interface call and data writing process should record the request ID, response summary, mapping rule version number and execution result. When it is found that the interface returns data containing multiple pages, pagination retrieval should be supported and each page of data should be compared and mapped separately. If data conflict or field validation failure is found during the mapping process, the record should be written to the exception handling queue and a manual verification work order should be generated. When the mapping and writing are successful, the corresponding target message should be consumed and confirmed or subsequent cleanup logic should be issued to avoid repeated retries.
[0032] In one optional implementation of this embodiment, after the steps of incrementing the retry count of the target message and rewriting the target message into the delay queue if no external identifier corresponding to the second upstream data is found, the method further includes: when the retry count reaches a preset maximum number of times, performing aggregate analysis on the historical execution records of the target message to obtain an abnormal feature vector; constructing a multi-dimensional anomaly tracking index structure by performing feature hierarchical mapping on the abnormal feature vector; performing clustering calculation on the abnormal records belonging to the same parent task chain according to the multi-dimensional anomaly tracking index structure to generate an abnormal task cluster; determining suspicious interface blocking nodes by comparing the time intervals and message retry distribution in the abnormal task cluster, and generating task tracking records with blocking identifiers; after the task tracking records are written to the anomaly log table, automatically generating backtracking task instances according to the node paths corresponding to the blocking identifiers, and writing the backtracking task instances into the backtracking queue.
[0033] In this embodiment, when the retry count reaches the preset maximum limit, the system determines that the message has not been associated after multiple delayed queries. Therefore, the anomaly analysis module is activated to aggregate and analyze the historical execution records of the target message. The historical records include the timestamp of each execution, response status, delay duration, retry interval, and interface identifier. By normalizing and statistically calculating these data, a vector reflecting the characteristics of abnormal behavior is generated. This anomaly feature vector describes the abnormal patterns of the task in the time, interface, and queue dimensions, and provides input for subsequent hierarchical index construction. The system then performs feature hierarchical mapping on the anomaly feature vector, dividing information of different dimensions into independent levels, such as the time layer, interface layer, and dependency layer, and assigning a unique index key to each layer. Through this mapping method, a multi-dimensional anomaly tracking index structure containing hierarchical relationships is constructed, enabling the system to quickly locate the source of the anomaly in subsequent analysis. This structure is stored in the database as a tree index, with nodes connected by task identifiers and interface paths, thus forming a traceable topology for anomaly information. Based on this index structure, the system performs clustering calculations on anomaly records within the same parent task chain. Clustering is based on the similarity between anomalous feature vectors, grouping records with similar features into anomalous task clusters. Each cluster corresponds to a set of tasks with common anomalous features, used to reveal potential systemic anomalies. The system further compares the time intervals and message retry distribution within the clusters. If multiple tasks are found to have a continuous failure or a concentrated trend in response delays on the same interface, that interface is identified as a suspicious blocking node. After identifying the blocking node, the system generates a task tracking record with a blocking identifier, including the interface number, time interval, and affected task identifier, and writes this record to the anomaly log table. Based on the node path corresponding to the blocking identifier, the system automatically generates a backtracking task instance. The backtracking task instance defines the interface path and call parameters that need to be retried, used to re-execute the blocked data transmission operation in subsequent scheduling. The generated backtracking task instance is written to the backtracking queue and executed asynchronously by the system's background scheduling module, thereby automatically restoring the blocked data link in subsequent cycles, achieving data association recompensation and link closure.
[0034] In an optional implementation of this embodiment, after the task tracking record is written to the anomaly log table, and after automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance to the backtracking queue, the method further includes: comparing the task identifier, external identifier, and external parent identifier of each task with the completed data association records in the target system and the original data records in the source system to generate a task correspondence matrix; determining the task status distribution within the same data transmission cycle by cross-validating the task correspondence matrix and the anomaly tracking index structure to generate a task status mapping table; calculating the confidence parameters of each task node by weighting the task execution count, delay interval, and anomaly identifier recorded in the task status mapping table to obtain a node confidence update set; generating a task scheduling parameter set by normalizing the node confidence update set, and updating the task scheduling strategy of the integrated vision according to the task scheduling parameter set.
[0035] In this embodiment, during data transmission and task management, a detailed comparison can be made between the completed data association records in the target system and the original data records in the source system. First, the task identifier, external identifier, and external parent identifier corresponding to each piece of data are arranged in order to form a task correspondence matrix. This matrix is a multi-dimensional structured table, where rows represent downstream data tasks, columns represent upstream data tasks or association attributes, and cell content records whether the correspondence has been established and the timestamp information of the association. The matrix can clearly reflect the upstream and downstream dependencies and completion status of each task, thereby providing a foundation for subsequent anomaly analysis and scheduling optimization. After the matrix is generated, it is cross-validated with the anomaly tracking index structure. The anomaly tracking index structure records the execution history and blocked nodes of each task in different interfaces, time intervals, and parent task chains in a tree or graph format. By comparing the status of each task in the matrix with the anomaly record of the corresponding task in the index structure, the completion status, latency distribution, and potential anomalies of each task within the same data transmission cycle can be determined, thus forming a task status mapping table. This mapping table not only records whether a task was successfully completed, but also includes the number of times the task was executed, the latency time for each instance, and anomaly identification information, which can accurately depict the dynamic state during data transmission. Based on the task status mapping table, the confidence parameters of each task node are weighted and calculated. The confidence parameters are used to quantify the reliability and success rate of each node in the data link. The weights can be determined comprehensively based on the number of task completions, latency intervals, and anomaly frequency. For example, nodes that have been successfully completed consecutively have higher confidence values, while nodes that have experienced delays or anomalies have their weights reduced accordingly, thus forming a dynamic reliability evaluation throughout the entire task chain. After weighted calculation, the node confidence update set is normalized, ensuring that the confidence values of each node are compared and scheduled under a unified dimension. The normalized data retains the differences in relative node reliability while facilitating the generation of scheduling parameters globally within the integrated view, ultimately forming a task scheduling parameter set. This parameter set is used to adjust task priorities, retry intervals, thread allocation, and message queue consumption strategies, enabling fine-grained control and optimization of asynchronous data transmission. For example, if the upstream task node corresponding to a downstream data point has experienced delays in the past several cycles, the normalized confidence value will decrease. This allows the system to prioritize allocating more resources for retries or actively triggering interface calls in the next transmission cycle, while avoiding resource waste on highly reliable nodes, thereby improving overall system throughput and data consistency. Based on the generated task scheduling parameter set, the integrated vision can dynamically update the scheduling strategy, including adjusting the task queue order, delayed consumption time interval, number of concurrent threads, and exception backtracking priority. This enables global optimization and adaptive scheduling of asynchronous concurrent task chains, while ensuring the accuracy of data association and task processing efficiency in multi-task, high-concurrency environments, forming a closed-loop control mechanism that makes data transmission and compensation processes predictable and self-healing.
[0036] In an optional implementation of this embodiment, after the task tracking record is written to the exception log table, and after automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance to the backtracking queue, the method further includes: calling the bidirectional interface between the source system and the target system based on the backtracking task instance to perform data consistency verification on the external identifiers and external parent identifiers of the upstream and downstream data, and generating an integrity comparison table; determining the abnormal dataset in the target system by analyzing the integrity comparison table item by item; initiating interface compensation calls for the missing upstream data entries based on the abnormal dataset, and generating a manual verification task package for the still incomplete abnormal records; and writing the task identifier, interface path, and comparison results of the manual verification task package into the manual processing database.
[0037] In this embodiment, after the backtracking task instance is triggered, the system calls the bidirectional interface between the source system and the target system based on the key identifiers recorded in the task. This performs a comprehensive data consistency check on the external identifiers and external parent identifiers of the upstream and downstream data. This process first obtains the upstream dataset and its corresponding downstream data's external parent identifiers from the source system via the interface. Simultaneously, it retrieves the downstream data and its associated upstream external identifiers from the target system, comparing the data on both sides one by one. The interface call must support idempotency, ensuring that the data state is not repeatedly modified during repeated calls. The interface response data is returned in a structured format, such as JSON or XML, containing the external identifier of the upstream record, the external parent identifier of the downstream record, the data generation time, the status identifier, and a unique request ID. The system parses the interface response to establish a mapping relationship between the corresponding data entries of the source system and the target system, and summarizes the mapping information to generate an integrity comparison table. Each row in this table corresponds to one downstream data record, and each column records the matching status of the external identifiers of the source system and the target system, the corresponding parent identifiers, and the verification time. The table clearly displays the consistency of each data item in the transmission link. By analyzing the integrity comparison table item by item, abnormal datasets in the target system can be identified. These abnormal datasets include downstream records with missing upstream associations, mismatched external parent identifiers, duplicate or conflicting association records, etc. Each abnormal data record contains an external identifier, an external parent identifier, a task identifier, and a verification result for subsequent processing. Based on the abnormal dataset, the system initiates interface compensation calls for missing upstream data entries. The compensation call requests the corresponding upstream data from the source system, obtains complete field information, and creates the upstream data record in the target system. Simultaneously, it updates the associated fields of the downstream data to ensure data integrity. For example, if the order master data corresponding to the order details does not exist, the master data is obtained through the interface and matched with the details. Abnormal records that are still not successfully completed after the compensation call are packaged into a manual verification task package. This task package contains the abnormal records to be manually verified, their corresponding task identifiers, interface paths, integrity comparison results, and previous compensation attempt records. The task package information is written to the manual processing database for further processing and verification by operations personnel or data administrators. The writing of manual verification task packages ensures that no abnormal data is missed, and provides traceable records for subsequent audits. For example, if the corresponding master data for a certain order detail is still not found after continuous compensation calls, its task identifier, interface path, comparison status, and reason for the anomaly are recorded in the manual processing database, making it easy for humans to confirm whether the record needs to be marked as invalid or handled specially. Throughout the process, bidirectional interface calls, integrity comparison table generation, abnormal dataset identification, interface compensation, and the writing of manual verification task packages form a closed loop, ensuring that upstream and downstream data in the data transmission link maintain a high degree of consistency and traceability in an asynchronous concurrent environment, while providing dual protection of system self-healing and manual intervention.
[0038] In one optional implementation of this embodiment, the node confidence update sets of multiple parallel tasks are sorted according to the task scheduling parameter set to generate a task priority sequence. By performing hierarchical aggregation on the task priority sequence, tasks with confidence levels higher than a preset confidence threshold are assigned to a high-speed execution queue, tasks with confidence levels in the middle range are assigned to a standard execution queue, and tasks with confidence levels lower than the minimum threshold are assigned to a delayed execution queue. Based on the task distribution status of the high-speed execution queue, the standard execution queue, and the delayed execution queue, the data transmission thread pool of the integrated vision is dynamically allocated to generate a resource allocation matrix. By scheduling and controlling the resource allocation matrix, the task execution order table for the next transmission cycle is determined.
[0039] In this embodiment, during data transmission scheduling, the node confidence update sets of multiple parallel tasks can be sorted according to the task scheduling parameter set. This sorting process first quantifies and evaluates the confidence value of each task node. Confidence represents the reliability and stability of a task node in successfully completing its associated operations in the data link. A higher value means a higher task completion success rate, shorter latency, and fewer anomaly records in the previous transmission cycle. The system sorts the node confidence update sets in descending order, placing high-confidence nodes at the top for priority scheduling in the next transmission cycle, thereby optimizing overall task throughput and data consistency. After sorting, the task priority sequence is aggregated hierarchically. Tasks with confidence levels above a preset confidence threshold are placed in a high-speed execution queue. This queue corresponds to the task with the highest resource allocation priority in the system's concurrent thread pool, ensuring that critical tasks are completed in the shortest possible time while reducing the impact of latency on downstream data association. Tasks with confidence levels in the middle range are placed in a standard execution queue, allocated appropriate thread resources to maintain overall system balance. Tasks with confidence levels below the minimum threshold are placed in a delayed execution queue. These tasks may have historical delays, anomalies, or interface blocking risks; delayed execution reduces the impact on the main transmission thread and provides a buffer time for anomaly compensation. For example, in an order data transmission scenario, if an order detail task has successfully associated the corresponding order master data multiple times in the past, its node confidence is high, so it is placed in the high-speed execution queue. Tasks that still have interface delays or occasional missing upstream data are placed in the delayed execution queue to reduce the risk of concurrency conflicts. Based on the task distribution of the high-speed execution queue, standard execution queue, and delayed execution queue, the system dynamically adjusts the data transmission thread pool allocation of the integrated vision, mapping the number of threads, concurrency, and priority resources to each queue to generate a resource allocation matrix. This matrix uses rows to represent task queue categories and columns to represent allocable thread resources. Each cell records the number of threads allocated and the task scheduling weight, thus forming a quantifiable resource scheduling structure. This allows the system to rationally utilize hardware resources and avoid thread starvation or resource waste in asynchronous concurrent transmission. By scheduling and controlling the resource allocation matrix, the system dynamically generates a task execution order table for the next transmission cycle according to the task priority sequence and queue distribution. This order table arranges each task identifier and its corresponding execution thread in a time series or priority sequence. When a task is triggered, the scheduling module starts data transmission and interface calls according to the order in the table, ensuring that high-confidence tasks are completed first, standard tasks are executed continuously, and low-confidence tasks are compensated or verified at set intervals in the delayed queue, thereby achieving global optimization and adaptive control of task execution.In this process, a closed-loop scheduling mechanism is formed through confidence ranking, hierarchical aggregation, resource matrix construction and sequence list generation. This enables the system to balance performance and reliability in multi-task, high-concurrency and asynchronous data transmission environments, achieve high efficiency, consistency and stability of data transmission, and at the same time ensure the flexibility of task priority adjustment and dynamic resource allocation under abnormal or delayed conditions.
[0040] According to the data concurrent transmission processing method provided in this application, the first upstream data and the first downstream data generated by the source system are identified to obtain a data identifier for the first upstream data and a parent identifier for the corresponding first downstream data. During the process of pushing data from the source system to the target system, field mapping is performed on the data identifier and the parent identifier to generate corresponding external identifiers and external parent identifiers. The external parent identifier is used to query whether the target system has an external identifier. If an external identifier corresponding to the second upstream data in the target system is found, the association between upstream and downstream data in the target system is completed. Through the implementation of this application, when downstream data from the source system arrives at the target system before upstream data, the corresponding external identifier can be queried in the target system using the external parent identifier, thus completing the association between upstream and downstream data in the target system and improving data transmission efficiency.
[0041] Figure 2 This application provides a data concurrent transmission processing apparatus, which can be used to implement the data concurrent transmission processing method described in the foregoing embodiments. For example... Figure 2 As shown, the concurrent data transmission processing device mainly includes: The generation module 10 is used to generate identifiers for the first upstream data and the first downstream data generated by the source system, so as to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data. Mapping module 20 is used to perform field mapping between data identifier and parent identifier during the process of pushing data from source system to target system, and generate corresponding external identifier and external parent identifier. Query module 30 is used to query whether an external identifier exists in the target system based on the external parent identifier. The query module 30 is also used to perform a timed query on the second upstream data if no external identifier corresponding to the second upstream data is found; The association module 40 is used to call the interface of the source system to obtain the second upstream data when the number of queries exceeds the preset trigger threshold, and to establish an association between the second upstream dataset and the second downstream data.
[0042] In an optional implementation of this embodiment, the query module is further configured to: if no external identifier corresponding to the second upstream data is found, send a target message to the message queue; execute a task according to the message corresponding to the target message, and perform a periodic query on the second upstream data through a delayed consumption strategy; if an external identifier corresponding to the second upstream data is found, confirm the consumption of the target message and update the information of the second upstream data to the second downstream data of the target system; if no external identifier corresponding to the second upstream data is found, increment the retry count value of the target message and rewrite the target message into the delayed queue.
[0043] In one optional implementation of this embodiment, the association module is used to: compare the retry count value with a preset trigger threshold; when the retry count value reaches the trigger threshold, call the interface of the source system to obtain the first upstream data and obtain the upstream dataset; compare the second downstream data according to the upstream dataset; if there is a data item in the upstream dataset that corresponds to the external parent identifier, then generate the second upstream data by mapping the fields of the data item, and establish a correspondence between the second upstream data and the second downstream data.
[0044] In an optional implementation of this embodiment, the generation module is further configured to: when the retry count reaches a preset maximum limit, aggregate and analyze the historical execution records of the target message to obtain an abnormal feature vector; construct a multi-dimensional anomaly tracking index structure by performing feature hierarchical mapping on the abnormal feature vector; perform clustering calculation on the abnormal records belonging to the same parent task chain according to the multi-dimensional anomaly tracking index structure to generate an abnormal task cluster; determine suspicious interface blocking nodes by performing pattern comparison on the time interval and message retry distribution in the abnormal task cluster, and generate task tracking records with blocking identifiers; after the task tracking records are written to the anomaly log table, automatically generate a backtracking task instance according to the node path corresponding to the blocking identifier, and write the backtracking task instance to the backtracking queue.
[0045] In an optional embodiment of this invention, the concurrent data transmission processing device further includes an update module. The update module is used to: compare the task identifier, external identifier, and external parent identifier of each task with the completed data association records in the target system and the original data records in the source system to generate a task correspondence matrix; determine the task status distribution within the same data transmission cycle by cross-validating the task correspondence matrix with the anomaly tracking index structure, and generate a task status mapping table; calculate the confidence parameters of each task node by weighting the task execution count, delay interval, and anomaly identifier recorded in the task status mapping table to obtain a node confidence update set; generate a task scheduling parameter set by normalizing the node confidence update set, and update the task scheduling strategy of the integrated horizon according to the task scheduling parameter set.
[0046] In an optional implementation of this embodiment, the concurrent data transmission processing device further includes a verification module. The verification module is used to: call the bidirectional interface between the source system and the target system based on the backtracking task instance to perform data consistency verification between the external identifiers and external parent identifiers of upstream and downstream data, generating an integrity comparison table; analyze each item in the integrity comparison table to determine abnormal datasets in the target system; initiate interface compensation calls for missing upstream data entries based on the abnormal datasets, and generate manual verification task packages for incomplete abnormal records; and write the task identifier, interface path, and comparison results of the manual verification task packages into the manual processing database.
[0047] In an optional embodiment of this invention, the concurrent data transmission processing device further includes a sorting module. The sorting module is used to: sort the node confidence update sets of multiple parallel tasks according to a task scheduling parameter set to generate a task priority sequence; by hierarchically aggregating the task priority sequence, assign task identifiers with confidence levels higher than a preset confidence threshold to a high-speed execution queue, task identifiers with confidence levels in the middle range to a standard execution queue, and task identifiers with confidence levels lower than a minimum threshold to a delayed execution queue; dynamically allocate the data transmission thread pool of the integrated vision according to the task distribution status of the high-speed execution queue, the standard execution queue, and the delayed execution queue to generate a resource allocation matrix; and determine the task execution order table for the next transmission cycle by scheduling control of the resource allocation matrix.
[0048] According to the data concurrent transmission processing apparatus provided in this application, first upstream data and first downstream data generated by the source system are identified to obtain a data identifier for the first upstream data and a parent identifier for the corresponding first downstream data. During the process of pushing data from the source system to the target system, field mapping is performed on the data identifier and the parent identifier to generate corresponding external identifiers and external parent identifiers. The external parent identifier is used to query whether the target system has an external identifier. If an external identifier corresponding to the second upstream data in the target system is found, the association between upstream and downstream data in the target system is completed. Through the implementation of this application, when downstream data from the source system arrives at the target system before upstream data, the corresponding external identifier can be queried in the target system using the external parent identifier, thus completing the association between upstream and downstream data in the target system and improving data transmission efficiency.
[0049] According to the scheme provided in this application Figure 3 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the data concurrent transmission processing method described in the foregoing embodiments, and mainly includes: The system includes a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are communicatively connected. When the processor 302 executes the computer program 303, it implements the concurrent data transmission processing method described in the foregoing embodiments. The number of processors can be one or more.
[0050] The memory 301 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.
[0051] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 3 The memory in the illustrated embodiment.
[0052] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data concurrent transmission processing method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, a portable hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, or any other medium capable of storing program code.
[0053] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0054] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0055] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for concurrent data transmission processing, characterized in that, include: The first upstream data and the first downstream data generated by the source system are identified to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data. During the process of pushing data from the source system to the target system, the data identifier and the parent identifier are mapped to fields respectively to generate corresponding external identifiers and external parent identifiers; In the target system, query whether the external identifier exists in the target system based on the external parent identifier; If no external identifier corresponding to the second upstream data is found, the second upstream data will be queried periodically using a delayed consumption strategy. When the number of queries exceeds a preset trigger threshold, the interface of the source system is called to obtain the second upstream data, and the second upstream dataset is associated with the second downstream data.
2. The data concurrent transmission processing method according to claim 1, characterized in that, The step of periodically querying the second upstream data using a delayed consumption strategy if no external identifier corresponding to the second upstream data is found includes: If no external identifier corresponding to the second upstream data is found, the target message is sent to the message queue. Execute tasks according to the messages corresponding to the target messages, and perform periodic queries on the second upstream data using a delayed consumption strategy; If an external identifier corresponding to the second upstream data is found, the target message is confirmed to be consumed and the information of the second upstream data is updated in the second downstream data of the target system. If no external identifier corresponding to the second upstream data is found, the retry count of the target message is incremented, and the target message is rewritten into the delay queue.
3. The data concurrent transmission processing method according to claim 2, characterized in that, The step of calling the interface of the source system to obtain the second upstream data when the number of queries exceeds a preset trigger threshold, and establishing an association between the second upstream dataset and the second downstream data, includes: The retry count value is compared with a preset trigger threshold; When the retry count reaches the trigger threshold, the interface of the source system is called to obtain the first upstream data, and the upstream dataset is obtained. The second downstream data is compared based on the upstream dataset; If a data item corresponding to the external parent identifier exists in the upstream dataset, the second upstream data is generated by mapping the fields of the data item, and a correspondence is established between the second upstream data and the second downstream data.
4. The data concurrent transmission processing method according to claim 2, characterized in that, After the step of incrementing the retry count of the target message and rewriting the target message into the delay queue if the external identifier corresponding to the second upstream data is not found, the method further includes: When the retry count reaches the preset maximum number of times, the historical execution records of the target message are aggregated and analyzed to obtain an abnormal feature vector. By performing feature hierarchical mapping on the abnormal feature vectors, a multidimensional anomaly tracking index structure is constructed. Based on the multidimensional anomaly tracking index structure, clustering calculations are performed on anomaly records belonging to the same parent task chain to generate anomaly task clusters. By comparing the time intervals and message retry distribution in the abnormal task cluster, suspicious interface blocking nodes are identified, and task tracing records with blocking identifiers are generated. Once the task tracking record is written to the exception log table, a backtracking task instance is automatically generated based on the node path corresponding to the blocking identifier, and the backtracking task instance is written to the backtracking queue.
5. The data concurrent transmission processing method according to claim 4, characterized in that, After the step of automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance into the backtracking queue after the task tracking record is written to the exception log table, the method further includes: Based on the completed data association records in the target system and the original data records in the source system, the task identifier, external identifier, and external parent identifier of each task are compared to generate a task correspondence matrix; By cross-validating the task correspondence matrix and the anomaly tracking index structure, the task status distribution within the same data transmission cycle is determined, and a task status mapping table is generated. Based on the number of task executions, delay intervals, and anomaly identifiers recorded in the task status mapping table, the confidence parameters of each task node are weighted and calculated to obtain the node confidence update set. By normalizing the node confidence update set, a task scheduling parameter set is generated, and the task scheduling strategy of the integrated vision is updated according to the task scheduling parameter set.
6. The data concurrent transmission processing method according to claim 4, characterized in that, After the step of automatically generating a backtracking task instance based on the node path corresponding to the blocking identifier and writing the backtracking task instance into the backtracking queue after the task tracking record is written to the exception log table, the method further includes: Based on the backtracking task instance, the bidirectional interface between the source system and the target system is invoked to perform data consistency verification on the external identifier and external parent identifier of the upstream and downstream data, and generate an integrity comparison table. By analyzing the integrity comparison table item by item, abnormal datasets in the target system are identified. Based on the abnormal dataset, an interface compensation call is initiated for the missing upstream data entries, and a manual verification task package is generated for the abnormal records that are still not complete. Write the task identifier, interface path, and comparison results of the manual verification task package into the manual processing database.
7. The data concurrent transmission processing method according to claim 5, characterized in that, The method further includes: The node confidence update sets of multiple parallel tasks are sorted according to the task scheduling parameter set to generate a task priority sequence. By performing hierarchical aggregation on the task priority sequence, tasks with confidence levels higher than a preset confidence threshold are assigned to the high-speed execution queue, tasks with confidence levels in the middle range are assigned to the standard execution queue, and tasks with confidence levels lower than the minimum threshold are assigned to the delayed execution queue. Based on the task distribution status of the high-speed execution queue, standard execution queue, and delayed execution queue, the data transmission thread pool of the integrated vision is dynamically allocated to generate a resource allocation matrix; By scheduling and controlling the resource allocation matrix, the task execution order table for the next transmission cycle is determined.
8. A data concurrent transmission processing device, characterized in that, The data concurrent transmission processing device is used to implement the data concurrent transmission processing method according to claim 1, and the data concurrent transmission processing device includes: The generation module is used to generate identifiers for the first upstream data and the first downstream data generated by the source system, so as to obtain the data identifier of the first upstream data and the parent identifier of the corresponding first downstream data. The mapping module is used to perform field mapping between the data identifier and the parent identifier respectively during the process of pushing data from the source system to the target system, and generate corresponding external identifiers and external parent identifiers; The query module is used to query whether the external identifier exists in the target system based on the external parent identifier; The query module is also used to periodically query the second upstream data if no external identifier corresponding to the second upstream data is found, using a delayed consumption strategy. The association module is used to call the interface of the source system to obtain the second upstream data when the number of queries exceeds a preset trigger threshold, and to establish an association between the second upstream dataset and the second downstream data.
9. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the data concurrent transmission processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the data concurrent transmission processing method according to any one of claims 1 to 7.