Device for unified data interaction with external system based on load balancing

Abstract

The invention provides a device for unified data interaction with an external system based on load balancing. The device comprises a data interaction service management module, a timer module, a service priority adjustment module, a data processing module and a data resetting module. Scanning frequency and task priority are dynamically adjusted through an intelligent load balancing algorithm, and efficient interaction, fault tolerance and management of data are realized in combination with a data processing and resetting module. The device improves the resource utilization rate, task processing efficiency and stability of the system, and is suitable for complex business scenes.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a device for unified data interaction with an external system based on load balancing. Background Art

[0002] In modern information systems, cross-system data interaction has become a key means to achieve business integration and data sharing. However, existing cross-system data interaction technologies generally have many problems and technical bottlenecks, which are difficult to meet the increasingly complex business needs and urgently need to be improved and optimized.

[0003] First, existing systems often deeply couple data interaction logic with business logic, resulting in poor system scalability and flexibility. Once the interface or data format of the external system changes, the internal business module needs to undergo tedious code modification and redeployment. This not only increases the maintenance difficulty and development cost of the system, but also poses a potential threat to the stability of the system. Such a design pattern is particularly fragile in a changing business environment and has become an important issue that restricts the efficient operation of the system.

[0004] Secondly, in terms of data interaction scheduling, traditional systems usually use a fixed-frequency timer scanning mechanism. This single task scheduling method lacks the ability to adapt to dynamic changes and is difficult to make corresponding adjustments based on real-time changes in system load. When the amount of data is small, fixed-frequency scheduling may lead to waste of resources; when the amount of data surges, it may cause task delays or even system crashes. This deficiency obviously makes it difficult to support the demand for efficient task scheduling in complex scenarios.

[0005] At the same time, in the actual data interaction process, different tasks usually have different importance. However, many existing technologies fail to effectively manage the priority of tasks and process all tasks with the same priority. This processing method easily causes key business tasks to be blocked by low-priority tasks, seriously affecting the timeliness and reliability of important businesses. The limitations of this priority processing are particularly evident in actual business scenarios.

[0006] In addition, when dealing with data transmission failure, most existing technologies only provide a simple retry mechanism. Although such a mechanism can improve the success rate to a certain extent, it lacks detailed management of failed tasks, especially the inability to classify, store or mark failed tasks. This lack of detailed management not only poses a major risk to the stability of the system, but also makes it difficult for operation and maintenance personnel to detect problems in a timely manner and take effective measures, which in turn affects the overall reliability and fault tolerance of the system.

[0007] In interactive scenarios with large amounts of data, traditional data interaction technologies still have inefficiencies. For example, many technologies lack intelligent scheduling capabilities and cannot dynamically adjust the frequency of data processing or resource allocation according to the real-time load of the system. This inefficient data processing mode leads to low system resource utilization and significantly increased data interaction delays, further affecting the overall performance of the system.

[0008] Finally, many existing systems lack a unified management mechanism when dealing with cross-system data interactions. The records of successful and failed tasks are often scattered in different modules or tables, making it difficult to track and analyze data. This decentralized management approach not only increases the risk of data loss and operational errors, but also greatly reduces the overall operating efficiency of the system. In actual operation and maintenance, this management model will greatly limit the flexibility and controllability of the system.

[0009] In summary, the existing cross-system data interaction technology has significant deficiencies in scalability, dynamic scheduling, priority management, fault tolerance and efficiency, and it is difficult to meet the actual needs of complex business scenarios. Therefore, there is an urgent need for a new technical solution that can improve the scalability and flexibility of the system while achieving effective management of dynamic loads, optimizing priority scheduling, improving fault tolerance, and greatly enhancing data interaction efficiency to meet the complex needs of modern information systems. Summary of the invention

[0010] In order to solve the above problems in the prior art, the present invention proposes a device for unified data interaction with an external system based on load balancing, comprising: a data interaction service management module, a timer module, a service priority adjustment module, a data processing module and a data reset module;

[0011] The data interaction service management module is used to receive the interaction data submitted by the business module through the interface, and store the interaction data in the to-be-sent table;

[0012] The timer module is used to periodically scan the table to be sent, assemble the data in the table to be sent, and send it to the corresponding external system according to the interface address; the timer module loads an intelligent load balancing algorithm, and when the data volume reaches a preset threshold, dynamically adjusts the frequency of data scanning and sending according to the current load of the system; and pre-allocates the data in the table to be sent through the task priority scheduling mechanism, reduces the scanning frequency of low-priority tasks, and gives priority to high-priority tasks; and dynamically adjusts the working cycle of the timer according to the real-time load of the system;

[0013] The service priority adjustment module is used to dynamically adjust the service priority of the data in the to-be-sent table according to the service importance, so as to ensure that the data with high service priority is sent first;

[0014] The data processing module includes: a success processing submodule, which is used to move the data from the waiting table to the success table after the data is successfully sent, and return the success result to the business module; a failure processing submodule, which is used to perform two automatic retries when the data fails to be sent; if the sending still fails, the data is moved from the waiting table to the failure table;

[0015] The data reset module includes: a failed data reset submodule, which is used to reset the data in the failed table, and the reset data re-enters the to-be-sent table; and a successful data reset submodule, which is used to reset the data in the successful table, and the reset data re-enters the to-be-sent table.

[0016] The data interaction service management module includes:

[0017] The interface receiving module is used to receive the interactive data submitted by the business module;

[0018] A data storage module, used for storing the interactive data in a table to be sent;

[0019] Data parsing module, used to parse and verify the legitimacy and integrity of interactive data;

[0020] Data management module, used to query, modify and delete data in the table to be sent;

[0021] Among them, the interactive data includes field information, interface address, priority and technical route; the table to be sent adopts a structured storage format, including a primary key field for uniquely identifying data, field information, interface address, priority, technical route and timestamp field.

[0022] The timer module comprises:

[0023] A scanning trigger module is used to periodically scan the table to be sent and trigger data processing according to a preset scanning cycle;

[0024] The data processing module is used to extract the data to be processed from the table to be sent, wherein the data to be processed includes field information, interface address, business priority and technical route; and to group the data according to the interface address, package and assemble the data, and send the data to the target external system;

[0025] The load balancing module is embedded in the scanning trigger module and is used to dynamically adjust the scanning frequency f according to the current load of the system. s and duty cycle T s ;

[0026] The priority scheduling module is used to calculate the task priority weight sum W p Prioritize the data in the to-be-sent table and optimize resource allocation;

[0027] System load monitoring module, used to calculate the real-time system load C r .

[0028] The scanning frequency f of the timer s The expression is:

[0029]

[0030] in:

[0031] f0 is the initial scanning frequency of the timer,

[0032] L d is the amount of data in the table to be sent,

[0033] W p is the sum of task priority weights,

[0034] C r It is the real-time system load, indicating the current occupancy of system resources.

[0035] α, β, γ, δ are control parameters.

[0036] The working period of the timer is T s The expression is:

[0037]

[0038] in:

[0039] T0 is the initial working period of the timer.

[0040] L d is the amount of data in the table to be sent,

[0041] W p is the sum of task priority weights,

[0042] k,λ,∈ is the control parameter.

[0043] The W p is the sum of the task priority weights, which is the comprehensive weight calculated based on the priorities of all tasks in the to-be-sent table. The calculation formula is:

[0044]

[0045] Among them, P i is the priority of the ith task. The smaller the priority value, the higher the importance of the task. n is the total number of tasks in the to-be-sent table.

[0046] C r is the real-time system load, indicating the current occupancy of system resources. The calculation formula is:

[0047]

[0048] C r The value range is 0 to 1, which is used to indicate the overall load status of system resources.

[0049] The service priority adjustment module includes:

[0050] An initial priority allocation module is used to allocate initial priorities according to the business importance of the task, wherein the business importance includes the urgency of the task, the criticality of the business module, the amount of data, and the interface dependency;

[0051] Dynamic priority adjustment module, used to reorder the tasks in the to-be-sent table according to real-time changes in business needs;

[0052] A sorting update module, used to prioritize the task queue based on a multi-factor weighted model, wherein the multi-factor weighted model uses task urgency, dependency, and business module importance as weight factors;

[0053] Resource allocation module, used to reserve resource channels for high-priority tasks;

[0054] The priority feedback module is used to synchronize the task processing status after priority adjustment to the relevant business modules.

[0055] The data processing module comprises:

[0056] The data success processing module is used to remove the corresponding record from the to-be-sent table after the data is successfully sent, and store it in the success table. The success table adopts a structured storage format and contains field information, interface address, task identifier, task status, successful sending timestamp and confirmation information returned by the external system;

[0057] The data failure processing module is used to start the automatic retry mechanism when data transmission fails, and retry the failed data twice. If it still fails after retry, the failed data will be removed from the waiting table and stored in the failure table;

[0058] A failure record storage module is used to record the data that is still unsuccessful after two retries into a failure table. The failure table adopts a structured storage format and contains field information, interface address, task identifier, number of failures, error code, failure timestamp and error information;

[0059] The result return module is used to return success status information to the business module after the data is sent successfully, including task identification, task status and confirmation information returned by the external system.

[0060] The data resetting module comprises:

[0061] The failure data reset module is used to move the specified records in the failure table to the waiting-to-send table and reset the status to "waiting to be sent". The failure data reset module includes:

[0062] The failure data screening and confirmation submodule is used to screen and confirm target failure records by task ID, timestamp or interface address;

[0063] The failed data migration and status update submodule is used to remove the specified records from the failed table and migrate them to the to-be-sent table;

[0064] The failed transaction guarantee submodule is used to ensure the atomicity of data removal and migration operations;

[0065] The failure log recording submodule is used to record the reset task identification, operation time, operator and failure reason information;

[0066] The success data reset module is used to move the specified record in the success table to the waiting-to-send table and reset the status to "waiting to send", and the module includes:

[0067] The success data screening and confirmation submodule is used to screen and confirm the target success records by task ID, timestamp or interface address;

[0068] The success data migration and status update submodule is used to remove the specified records from the success table and migrate them to the to-be-sent table;

[0069] The successful transaction guarantee submodule is used to ensure the atomicity of data removal and migration operations;

[0070] The success log recording submodule is used to record the reset task identifier, operation time, operator and detailed information of the success record.

[0071] Beneficial effects:

[0072] The present invention provides a device for unified data interaction with an external system based on load balancing, which realizes efficient and stable data interaction through modular design. The intelligent load balancing algorithm of the device is combined with a timer module to effectively and dynamically adjust the data scanning frequency and working cycle, thereby improving the system resource utilization and the response speed of key tasks. The business priority adjustment module dynamically optimizes task scheduling according to the importance of the business, ensures that high-priority tasks are processed first, and further enhances the flexibility and adaptability of the system. The data processing module ensures accurate storage and status feedback of tasks through success and failure processing submodules, improves the success rate of tasks and enhances the fault tolerance of the system. The data reset module supports flexible management of success and failure data, facilitates the resending and secondary processing of tasks, and provides strong support for operation and maintenance. Through the above design, the present invention can significantly improve the efficiency, stability and manageability of data interaction and meet complex and changing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0074] Figure 1 The diagram shows the composition of the device for unified data interaction with external systems based on load balancing of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0076] The present invention provides a device for unified data interaction with external systems based on load balancing, as shown in the attached Figure 1 As shown, including:

[0077] 1. A data interaction service management module, which is used to receive interaction data submitted by a business module through an interface and store the interaction data in a table to be sent. The interaction data includes field information, interface address, business priority and technical route;

[0078] 2. A timer module is used to periodically scan the table to be sent, assemble the data in the table to be sent, and send it to the corresponding external system according to the interface address; the timer module loads an intelligent load balancing algorithm, and when the data volume reaches a preset threshold, the frequency of data scanning and sending is dynamically adjusted according to the current load of the system; and the data in the table to be sent is pre-allocated through the task priority scheduling mechanism, and the scanning frequency of low-priority tasks is reduced to give priority to high-priority tasks; and the working cycle of the timer is dynamically adjusted according to the real-time load of the system;

[0079] 3. Priority adjustment module, used to dynamically adjust the business priority of the data in the to-be-sent table according to the business importance, so as to ensure that the data with high business priority is sent first;

[0080] 4. Data processing module, including:

[0081] a. Success processing submodule, which is used to move the data from the to-be-sent table to the success table after the data is sent successfully, and return the success result to the business module;

[0082] b. Failure processing submodule, used to perform two automatic retries when data transmission fails; if transmission still fails, the data is moved from the waiting table to the failure table;

[0083] 5.Data reset module, including:

[0084] a. Failure data reset submodule, used to reset the data in the failure table, and the reset data re-enters the to-be-sent table;

[0085] b. Success data reset submodule, used to reset the data in the success table, and the reset data re-enters the waiting to be sent table.

[0086] The implementation methods of each specific module are shown in the following examples:

[0087] Example 1 Data Interaction Service Management Module

[0088] This embodiment provides a specific technical implementation solution for building a data interaction service management system, aiming to provide efficient and stable data interaction support between business modules and external systems. The following is a detailed implementation process and technical details.

[0089] First, a data interaction service management system is constructed. The system is based on modular design and includes an interface receiving module, a data storage module, a data parsing module, and a data management module. The interface receiving module is used to receive the interaction data submitted by the business module, the data storage module is responsible for storing the data in the table to be sent, the data parsing module is used to parse and verify the legitimacy and integrity of the data, and the data management module provides the functions of querying, modifying, and deleting the data to be sent.

[0090] The business module submits interactive data through the standardized interface provided by the data interaction service management system. The submitted data includes field information, interface address, business priority and technical route. The field information refers to the specific data content that needs to be interacted with the external system, the interface address is used to specify the specific access path of the target external system, the business priority is an integer value indicating the importance of the task (the lower the value, the higher the priority), and the technical route is used to define the specific technical protocol or method of data interaction, such as HTTP, SOAP or message queue.

[0091] The to-be-sent table is one of the core components of the data interaction service management system, and is used to temporarily store all data to be sent. The to-be-sent table uses a structured storage format to support fast query and flexible expansion. The table contains the following fields: a primary key field that uniquely identifies the data, a field for storing field information, interface address, priority, and technical route of the interaction data, and a timestamp field for recording the creation and update time of the data. The field information is stored in a structured format, usually in JSON format, which can flexibly support the expansion of data fields for different business modules. The priority field sets the importance of the task and sets an index for it to speed up data query.

[0092] During the data storage process, the data parsing module first parses the data submitted by the business module to ensure the legitimacy and integrity of the field information, interface address, priority, and technical route. For example, when parsing field information, verify whether it conforms to the expected format, whether the interface address conforms to the specification, whether the priority is within a reasonable range, and whether the technical route is supported by the system. The data that passes the verification will be stored by the data storage module in the to-be-sent table, and the relevant timestamp will be recorded to support subsequent operations.

[0093] The data interaction service management system also designs reliability and scalability guarantee mechanisms for high-concurrency scenarios. For example, by creating indexes on priority fields, it supports fast sorting and query operations in large-scale data situations. The transaction management mechanism is used to ensure that when submitting data, if a step fails, the entire transaction is rolled back to ensure data consistency. In addition, the partition storage strategy is used to improve the data processing capabilities of the table and avoid storage bottlenecks in large data volume scenarios.

[0094] In order to verify the system functions, comprehensive tests were carried out during the actual deployment process, including submission tests on simulated interactive data to verify whether the interface receiving module can correctly receive and store data; correctness verification of the data parsing module to ensure that the field information is parsed correctly; and stress testing of large-scale data submission scenarios to confirm the stability and responsiveness of the system under high concurrency.

[0095] The detailed description of the above specific implementation methods ensures that the implementation method of the data interaction service management system can be accurately reproduced by technical personnel in this field, providing a reliable technical means and design basis for achieving efficient and stable data interaction.

[0096] Example 2 Timer module

[0097] This embodiment describes in detail a technical solution for realizing efficient data interaction by periodically scanning the table to be sent through a timer and combining it with intelligent load balancing technology, so as to optimize the operating efficiency and stability of the system. The following is the specific implementation process and technical details.

[0098] 1. Setting of timer module

[0099] The core function of the timer module is to implement periodic scanning of the table to be sent, and to complete data extraction and processing. In order to meet the needs of different business scenarios, the module is designed with flexible scanning cycle parameters, which can set the initial value when the system is deployed, and dynamically adjust it in the subsequent process according to the actual operation requirements. The trigger mechanism of the timer initially adopts a fixed time interval mode, and triggers data processing through a preset scanning cycle. As the system load changes, the timer can dynamically adjust the scanning frequency and working cycle through an intelligent load balancing algorithm to improve the efficiency of data interaction and resource utilization.

[0100] 2. Timer scanning process

[0101] Each time the timer is triggered, the module performs the following data scanning and processing operations:

[0102] First, the timer module extracts the qualified data to be processed from the table to be sent. These data include field information, interface address, business priority and technical route, etc. Field information refers to the specific interactive data content, and the interface address is the access path of the target external system.

[0103] Then, the extracted data is grouped according to the interface address. This process is designed to optimize assembly efficiency and reduce duplication. The grouping mechanism of the interface address is based on the internal rules of the system and the consistency of the external interface to ensure the efficiency and accuracy of data transmission.

[0104] Finally, the timer module packages and assembles the grouped data and sends it to the external system according to the target address corresponding to the group, completing the data interaction process.

[0105] 3. Embedding of intelligent load balancing algorithm

[0106] To cope with resource competition and task delay issues in high-load scenarios, the timer module is embedded with an intelligent load balancing algorithm, enabling it to run stably and process tasks efficiently under high-load conditions.

[0107] The intelligent load balancing formula mainly involves the scanning frequency f of the timer s and duty cycle T s .

[0108] The scanning frequency of the timer is f s and duty cycle T s The dynamic adjustment of is determined by the following formula:

[0109] Scan frequency formula:

[0110]

[0111] Duty cycle formula:

[0112]

[0113] Parameter definition and description:

[0114] f0 is the initial scanning frequency of the timer, representing the basic operating frequency of the system (unit: times / second).

[0115] T0 is the initial working period of the timer, indicating the basic time interval for scanning trigger (unit: seconds).

[0116] L d The amount of data in the table to be sent, indicating the number of tasks currently to be processed (unit: item).

[0117] W p is the sum of the task priority weights, which is the comprehensive weight calculated based on the priorities of all tasks in the to-be-sent table. The calculation formula is:

[0118]

[0119] Among them, P i is the priority of the ith task. The smaller the priority value, the higher the importance of the task. n is the total number of tasks in the to-be-sent table.

[0120] C r is the real-time system load, indicating the current occupancy of system resources. The calculation formula is:

[0121]

[0122] C r The value range is 0 to 1, which is used to indicate the overall load status of system resources.

[0123] α, β, γ, δ, κ, λ,∈ are control parameters in the formula, which are used to adjust the response sensitivity and adjustment amplitude of the formula, and are all non-zero positive real numbers.

[0124] When the amount of data in the to-be-sent table is large or the proportion of high-priority tasks is high, the positive adjustment term α·L in the formula d +β·W p Will drive the scanning frequency f s Improve, thereby speeding up task processing;

[0125] When the system load C r When it is higher, the negative adjustment term γ·C r +δ enhancement, reduce the scanning frequency f s , effectively alleviate the competition pressure of system resources;

[0126] Working cycle T s The adjustment design enables high-priority tasks to respond quickly when resources permit, while the processing of low-priority tasks is appropriately delayed until the resource idle stage, thereby achieving dynamic optimization scheduling of priorities.

[0127] The intelligent load balancing algorithm calculates the amount of data L in the table to be sent. d and task priority weight W p With the system real-time load C r The scanning frequency and working cycle of the timer are dynamically adjusted to improve resource utilization;

[0128] The average value of task priority weight W p The introduction of makes the formula more adaptable to complex scenarios of multi-task parallelism and avoids the limitations of a single priority mechanism;

[0129] Through flexible configuration of parameters, formulas can be adjusted according to different business scenarios, enhancing the adaptability and controllability of the system.

[0130] Through the design and embedding of the above-mentioned intelligent load balancing algorithm, the timer module can dynamically adjust the scanning frequency and working cycle according to the data volume of the table to be sent, the task priority and the real-time load of the system, and achieve the following technical effects: improve the processing speed of high-priority tasks to ensure that critical tasks receive priority response; effectively protect the stability of the system in high-load scenarios to avoid the risk of crash caused by overload; dynamically optimize the adjustment mechanism of scanning frequency and cycle to reduce resource waste and improve the efficiency of overall data interaction.

[0131] Through the above implementation, the scheme of the present invention can effectively realize the efficient scanning and task processing of the to-be-sent table by the timer module, and at the same time significantly improve the operating performance and stability of the system in combination with the intelligent load balancing algorithm, and is suitable for complex and changeable data interaction scenarios.

[0132] Example 3 Service priority adjustment module

[0133] This embodiment describes in detail a technical solution for dynamically adjusting the priority of data in a to-be-sent table according to business importance during data sending, so as to ensure that key tasks can be processed in a timely manner and improve the task scheduling efficiency and responsiveness of the system.

[0134] During the data interaction process, the system assigns an initial priority to each piece of data in the to-be-sent table according to its business importance. Data with high business importance is given a higher priority (smaller value) to ensure that it is responded to first during resource allocation and task processing. At the same time, the system has designed a dynamic priority adjustment mechanism that can reorder the data in the to-be-sent table according to real-time changes in business needs and dynamically optimize the task processing order.

[0135] 1. Initial priority allocation

[0136] Each data in the table to be sent will be assigned an initial priority value when it is entered. The initial priority is assigned according to the following criteria:

[0137] Urgency of the task: Assign higher priority to tasks that need to be processed in real time;

[0138] Criticality of business modules: Data tasks from core business modules are usually given higher priority;

[0139] Data volume: For processing small batches of data, the priority is relatively high to reduce waiting time;

[0140] Interface dependency: If some data tasks depend on the execution results of other tasks, higher priorities are assigned to tasks with strong dependencies.

[0141] 2. Dynamic priority adjustment mechanism

[0142] In order to adapt to the dynamic changes in business needs, the system has designed a priority adjustment strategy to continuously optimize the priority order in the to-be-sent table during timer scanning and data transmission. The adjustment mechanism includes the following technical means:

[0143] Real-time monitoring of business needs: The system dynamically adjusts the priority of data by monitoring the task queue status of the business module and changes in task importance.

[0144] Reordering logic: When business requirements change or new high-priority tasks are added, the system reorders all tasks in the waiting list according to the priority rules to ensure that high-priority tasks are always at the front of the task queue. The sorting algorithm uses a multi-factor weighted model to perform comprehensive sorting based on task urgency, dependency, and business module importance.

[0145] Reserve high-priority channels: Reserve specific resource channels for critical tasks to ensure that high-priority tasks can still be processed first when the system is under high load.

[0146] 3. Technical Implementation of Dynamic Priority Adjustment

[0147] Dynamic priority adjustment is achieved through the following steps:

[0148] Data filtering: Each time the system scans the table to be sent, it automatically filters the data tasks with higher priority in the current queue.

[0149] Sorting update: For newly added tasks or tasks with changed priorities, the system inserts them into the appropriate positions and adjusts the positions of other tasks to ensure that the entire task queue always maintains priority sorting.

[0150] Real-time feedback: After the adjustment is completed, the system will synchronize the priority change information to the relevant modules so that the business modules can grasp the processing status and priority update of the task in real time.

[0151] Example 4 Data Processing Module - Send Successfully

[0152] This embodiment describes in detail the processing method after the data is successfully sent. The system removes the data in the table to be sent and stores it in the success table, and returns the result to the business module to ensure the completion of the data interaction closed loop and support the operation and maintenance personnel to quickly query and manage the successful records. The following is the specific technical implementation and detailed operation process.

[0153] 1. Successful data processing flow

[0154] When data is sent to an external system and a confirmation response is received, the system performs the following operations on the successfully sent data:

[0155] Data removal: The system first removes the record from the waiting table to avoid repeated sending. The removal operation uses a transaction mechanism to ensure that the data is successfully written to the success table after being deleted from the waiting table to avoid data loss due to system failure.

[0156] Data storage: The system stores the successfully sent records in the success table to record the detailed information of the interaction. The success table is designed as structured storage to support subsequent fast retrieval and operation.

[0157] Result return: The system immediately returns a successful result to the business module that initiated the task, including the task ID, success status, and confirmation information returned by the external system to support the business module in executing subsequent processes.

[0158] 2. Design and implementation of success table

[0159] The success table is an important data storage component used to store all successful sending records. Its design should meet the integrity, efficiency and query requirements of data storage. The detailed design of the success table is as follows:

[0160] Field design:

[0161] The success table stores data in a structured format. The specific fields include:

[0162] id: primary key, uniquely identifies each successful record;

[0163] task_id: task ID, associated with the task in the original to-be-sent table;

[0164] field_info: field information of the data, recording the specific content of the interaction data;

[0165] interface_address: interface address of the external system;

[0166] status: task status, fixed as "success";

[0167] timestamp: The timestamp of successful sending, used to record the task completion time;

[0168] response_info: confirmation information returned by the external system, recording the response details (such as status code or return result).

[0169] The success table is designed as an index-optimized storage structure that supports efficient retrieval by time, task ID, or interface address. For example, an index is created for the time stamp field to quickly locate successful task records within a certain period of time.

[0170] In the process of removing data from the to-be-sent table and writing it to the success table, the system uses a transaction mechanism to ensure data consistency, ensure the atomicity of operations, and avoid data loss or duplicate records due to partial operation failures.

[0171] 3. Result return mechanism

[0172] In order to ensure that the business module can receive the successful status of data transmission in real time, the system has designed the following result return mechanism:

[0173] Return content: The data returned by the system includes the task ID, success status, confirmation information returned by the external system, and the record ID in the success table.

[0174] Return method: The system sends the results to the business module through asynchronous callback or active push to ensure that the business module can quickly receive the interaction results.

[0175] Result confirmation: After the business module receives the successful result, it matches the task ID with the local record to confirm that the data has been sent successfully to start the subsequent process.

[0176] 4. Operation and maintenance support functions

[0177] To facilitate operation and maintenance personnel to query and manage successfully sent data, the success table provides the following support functions:

[0178] Quick search: supports querying by task ID, timestamp or interface address. For example, operation and maintenance personnel can quickly retrieve all successfully sent task records within a certain period of time, or query the total number and details of successful tasks of a certain interface.

[0179] Statistical function: The success table supports statistics on key indicators such as task success rate and interface success rate by time period, providing data support for system optimization.

[0180] Data export: Allows operators to export query results to common file formats (such as CSV) for external analysis or report generation.

[0181] Through the above specific implementation methods, this solution achieves efficient processing after successful data transmission, ensuring the closed loop of data interaction, while providing comprehensive support for business modules and operation and maintenance personnel to meet the system's requirements for stability, real-time and manageability.

[0182] Example 5 Data Processing Module-Send Failure

[0183] This embodiment describes in detail the processing method after data transmission failure. The system ensures the reliability and fault tolerance of the data interaction process through an automatic retry mechanism and failure record management, and provides support for subsequent remedial measures.

[0184] 1. Processing flow after data sending failure

[0185] When the system detects that data transmission fails, the following processing flow will be initiated:

[0186] Automatic retry: The system will automatically retry the failed data twice. Before each retry, the data will be reassembled and sent to the target external system again.

[0187] Failure record storage: If the data fails after two retries, the system removes the data from the table to be sent and stores its detailed records in the failure table for subsequent operation and maintenance or system retry.

[0188] 2. Automatic retry mechanism

[0189] The automatic retry mechanism is the core response measure after data transmission failure, which aims to improve the success rate of the task through multiple attempts. The following are the specific implementation steps:

[0190] Retry limit: The system defaults to a maximum of 2 retries to avoid invalid repeated attempts that occupy system resources. The retry counter is associated with the task and is updated after each attempt.

[0191] Data reassembly: Before each retry, the system rereads the task information from the table to be sent and reassembles the interaction data according to the current status, including field information, interface address, and technical route. The reassembly process ensures data integrity and avoids inconsistencies caused by previous sending anomalies.

[0192] Retry interval: The system sets a configurable time interval between two retries, which is dynamically adjusted based on the task priority. The retry interval for high-priority tasks is shorter, and the interval for low-priority tasks is relatively longer to optimize system resource utilization.

[0193] Failure judgment conditions: The system judges the sending failure in the following ways:

[0194] Failure to establish a connection with the target external system;

[0195] The target system returns a non-successful status code;

[0196] A timeout or interruption occurred during data transmission.

[0197] 3. Failure record storage mechanism

[0198] For data that has not been successfully retried twice, the system removes it from the waiting table and stores it in the failure table. The design and storage process of the failure table are as follows:

[0199] Field design of failure table:

[0200] The failure table is used to store all task records that failed to be sent. Its fields include:

[0201] id: primary key, uniquely identifies each failed record;

[0202] task_id: task ID, associated with the task in the original to-be-sent table;

[0203] field_info: field information of the data, recording the specific content of the interaction data;

[0204] interface_address: interface address of the external system;

[0205] fai l_count: number of failures, recording the number of task retries;

[0206] error_code: error code, which stores the failure status returned by the external system;

[0207] timestamp: the timestamp of the last failure;

[0208] error_message: Error message describing the specific reason for the failure (such as connection timeout or system unavailable).

[0209] The failure table uses structured storage to support efficient retrieval by task ID, timestamp or interface address.

[0210] The system creates indexes for the fail_count and timestamp fields to quickly locate frequently failed tasks and recent failure records.

[0211] The system uses a transaction mechanism in the process of removing records from the table to be sent and writing them into the failure table to ensure the atomicity of the operation and avoid record loss due to failure of some operations.

[0212] The failure table provides operation and maintenance personnel with a wealth of management and query functions to ensure that failed tasks can be discovered and processed in a timely manner:

[0213] Failure record query: Operation and maintenance personnel can query the details of failed tasks by time period, task ID or interface address, including information such as failure reasons and number of retries.

[0214] Retry support: The system allows operation and maintenance personnel to manually retry failed tasks. When manually retrying, the task will be reloaded to the waiting table and queued according to priority.

[0215] Statistical analysis: The failure table supports the statistical functions of failure rate and failure cause, providing data support for system optimization. For example, the failure rate statistics by interface address are used to analyze the stability of the target system.

[0216] Embodiment 6 Data reset module

[0217] This embodiment describes in detail the operation and maintenance functions provided by the system, including the reset operation of the data in the failure table and the success table to support the resending of data and the reprocessing of tasks.

[0218] 1. Failed table data reset operation

[0219] When the operation and maintenance personnel reset the data in the failure table, the system will re-add the specified failed task to the waiting table and restore its status to "waiting to be sent" so that the system can process it again. The specific implementation steps are as follows:

[0220] 1.1 Data screening and confirmation

[0221] The operation and maintenance personnel can filter the target failure records by task ID, timestamp or interface address. The system provides a query interface and supports multi-condition combination filtering to ensure that the operation and maintenance personnel can accurately locate the records that need to be reset.

[0222] In the screening results, the operation and maintenance personnel select the failure record that needs to be reset, and the system will display the key information of the record (such as the reason for failure, number of failures and task content) for confirmation.

[0223] 1.2 Data Migration and Status Update

[0224] After confirmation, the system performs the following operations:

[0225] Remove specified records from the failure table: The system marks these records as "reset" and deletes them from the failure table to avoid repeated resets.

[0226] Re-add to the waiting table: Migrate the original data of the record, such as field information, interface address, priority, etc., to the waiting table and reset its status to "waiting to be sent". After the migration is completed, the record enters the system's waiting task queue according to its priority.

[0227] 1.3 Transaction Mechanism Guarantee

[0228] The data removal and migration operations use a transaction mechanism to ensure the atomicity of the operation. If any exception occurs during the migration process (such as database connection interruption), the system will roll back all operations to ensure data consistency.

[0229] 1.4 Record reset log

[0230] The system generates a log for each reset operation, including the following information:

[0231] Reset the task's identification and content;

[0232] The time and operator of the reset operation;

[0233] The original failure reason and the record information of the failure table.

[0234] Reset logs can be used for auditing and subsequent operation and maintenance analysis.

[0235] 2. Successful table data reset operation

[0236] When the operation and maintenance personnel reset the data in the success table, the system will re-add the specified successful task to the waiting table and restore its status to "waiting to be sent" so that it can be re-sent externally. The specific implementation steps are as follows:

[0237] 2.1 Data screening and confirmation

[0238] The operation and maintenance personnel can filter the target success records by task ID, timestamp or interface address. The system provides a query interface to support fast retrieval and precise positioning. The screening results include key information of each success record (such as task content, interface address and success time). The operation and maintenance personnel can view the detailed information to confirm whether a reset is required.

[0239] 2.2 Data Migration and Status Update

[0240] After confirmation, the system performs the following operations:

[0241] Remove specified records from the success table: The system marks these records as "reset" and deletes them from the success table;

[0242] Re-add to the waiting table: Migrate the original data of the record, such as field information, interface address, priority, etc., to the waiting table and reset its status to "waiting to be sent". After the migration is completed, the record enters the waiting table and is queued for processing.

[0243] 2.3 Transaction Mechanism Guarantee

[0244] Similar to the reset operation of the failed table, the reset process of the successful table also uses a transaction mechanism to ensure the atomicity of all operations and avoid incomplete or lost data due to failures in the middle.

[0245] 2.4 Record reset log

[0246] The system generates a log for each reset operation, including the following:

[0247] Reset the task's identification and content;

[0248] The time and operator of the reset operation;

[0249] Details of the original success record.

[0250] Log records make it easier for operation and maintenance personnel to track the execution of reset tasks.

[0251] Through the above specific implementation methods, the scheme of the present invention realizes flexible management and efficient resetting of data in the failure table and the success table, ensuring that the system has strong operation and maintenance capabilities, while meeting the needs of repeated processing and resending of tasks during data interaction.

[0252] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A device for unified data interaction with an external system based on load balancing, characterized in that: include: Data interaction service management module, timer module, service priority adjustment module, data processing module and data reset module; The data interaction service management module is used to receive the interaction data submitted by the business module through the interface, and store the interaction data in the to-be-sent table; The timer module is used to periodically scan the table to be sent, assemble the data in the table to be sent, and send it to the corresponding external system according to the interface address; the timer module loads an intelligent load balancing algorithm, and when the data volume reaches a preset threshold, dynamically adjusts the frequency of data scanning and sending according to the current load of the system; and pre-allocates the data in the table to be sent through the task priority scheduling mechanism, reduces the scanning frequency of low-priority tasks, and gives priority to high-priority tasks; and dynamically adjusts the working cycle of the timer according to the real-time load of the system; The service priority adjustment module is used to dynamically adjust the service priority of the data in the to-be-sent table according to the service importance, so as to ensure that the data with high service priority is sent first; The data processing module includes: a success processing submodule, which is used to move the data from the waiting table to the success table after the data is successfully sent, and return the success result to the business module; a failure processing submodule, which is used to perform two automatic retries when the data fails to be sent; if the sending still fails, the data is moved from the waiting table to the failure table; The data reset module includes: a failed data reset submodule, which is used to reset the data in the failed table, and the reset data re-enters the to-be-sent table; and a successful data reset submodule, which is used to reset the data in the successful table, and the reset data re-enters the to-be-sent table.

2. The device for unified data interaction with an external system based on load balancing according to claim 1, characterized in that: The data interaction service management module includes: The interface receiving module is used to receive the interactive data submitted by the business module; A data storage module, used for storing the interactive data in a table to be sent; Data parsing module, used to parse and verify the legitimacy and integrity of interactive data; Data management module, used to query, modify and delete data in the table to be sent; Among them, the interactive data includes field information, interface address, priority and technical route; the table to be sent adopts a structured storage format, including a primary key field for uniquely identifying data, field information, interface address, priority, technical route and timestamp field.

3. The device for unified data interaction with an external system based on load balancing according to claim 1, characterized in that: The timer module comprises: A scanning trigger module is used to periodically scan the table to be sent and trigger data processing according to a preset scanning cycle; The data processing module is used to extract the data to be processed from the table to be sent, wherein the data to be processed includes field information, interface address, business priority and technical route; and to group the data according to the interface address, package and assemble the data, and send the data to the target external system; The load balancing module is embedded in the scanning trigger module and is used to dynamically adjust the scanning frequency f according to the current load of the system. s and duty cycle T s ; The priority scheduling module is used to calculate the task priority weight sum W p Prioritize the data in the to-be-sent table and optimize resource allocation; System load monitoring module, used to calculate the real-time system load C r .

4. The device for unified data interaction with an external system based on load balancing according to claim 3, characterized in that: The scanning frequency f of the timer s The expression is: in: f0 is the initial scanning frequency of the timer, L d is the amount of data in the table to be sent, W p is the sum of task priority weights, C r It is the real-time system load, indicating the current occupancy of system resources. α, β, γ, δ are control parameters.

5. The device for unified data interaction with external systems based on load balancing according to claim 3, characterized in that: The working period of the timer is T s The expression is: in: T0 is the initial working period of the timer. L d is the amount of data in the table to be sent, W p is the sum of task priority weights, κ,λ,∈ are control parameters.

6. The device for unified data interaction with an external system based on load balancing according to claim 4 or 5, characterized in that: The W p is the sum of the task priority weights, which is the comprehensive weight calculated based on the priorities of all tasks in the to-be-sent table. The calculation formula is: Among them, P i is the priority of the i-th task. The smaller the priority value, the higher the importance of the task. n is the total number of tasks in the to-be-sent table. C r is the real-time system load, indicating the current occupancy of system resources. The calculation formula is: C r The value range is 0 to 1, which is used to indicate the overall load status of system resources.

7. The device for unified data interaction with external systems based on load balancing according to claim 1, characterized in that: The service priority adjustment module includes: An initial priority allocation module is used to allocate initial priorities according to the business importance of the task, wherein the business importance includes the urgency of the task, the criticality of the business module, the amount of data, and the interface dependency; Dynamic priority adjustment module, used to reorder the tasks in the to-be-sent table according to real-time changes in business needs; A sorting update module, used to prioritize the task queue based on a multi-factor weighted model, wherein the multi-factor weighted model uses task urgency, dependency, and business module importance as weight factors; Resource allocation module, used to reserve resource channels for high-priority tasks; The priority feedback module is used to synchronize the task processing status after priority adjustment to the relevant business modules.

8. The device for unified data interaction with an external system based on load balancing according to claim 1, characterized in that: The data processing module comprises: The data success processing module is used to remove the corresponding record from the to-be-sent table after the data is successfully sent, and store it in the success table. The success table adopts a structured storage format and contains field information, interface address, task identifier, task status, successful sending timestamp and confirmation information returned by the external system; The data failure processing module is used to start the automatic retry mechanism when data transmission fails, and retry the failed data twice. If it still fails after retry, the failed data will be removed from the waiting table and stored in the failure table; A failure record storage module is used to record the data that is still unsuccessful after two retries into a failure table. The failure table adopts a structured storage format and contains field information, interface address, task identifier, number of failures, error code, failure timestamp and error information; The result return module is used to return success status information to the business module after the data is sent successfully, including task identification, task status and confirmation information returned by the external system.

9. The device for unified data interaction with external systems based on load balancing according to claim 1, characterized in that: The data resetting module comprises: A failure data reset module is used to move the specified record in the failure table to the waiting-to-send table and reset the state to "waiting to send", and the failure data reset module includes: The failure data screening and confirmation submodule is used to screen and confirm target failure records by task ID, timestamp or interface address; The failed data migration and status update submodule is used to remove the specified records from the failed table and migrate them to the to-be-sent table; The failed transaction guarantee submodule is used to ensure the atomicity of data removal and migration operations; The failure log recording submodule is used to record the reset task identification, operation time, operator and failure reason information; The success data reset module is used to move the specified record in the success table to the waiting-to-send table and reset the status to "waiting to send", and the module includes: The success data screening and confirmation submodule is used to screen and confirm the target success records by task ID, timestamp or interface address; The success data migration and status update submodule is used to remove the specified records from the success table and migrate them to the to-be-sent table; The successful transaction guarantee submodule is used to ensure the atomicity of data removal and migration operations; The success log recording submodule is used to record the reset task identifier, operation time, operator and detailed information of the success record.

Images