Interface data transmission mode for data transmission, pushing and synchronization

Abstract

The invention provides an interface data transmission mode for data transmission, pushing and synchronization, and relates to the field of metal coiled material controllers. The interface data transmission mode based on data transmission, pushing and synchronization comprises the steps that a pluggable data adapter makes contact with an interface of a device, the data adapter is used as a bridge of the interface, a universal data model is defined through JSON Schema, data exchange and verification are allowed between different assemblies, the universal schema is designed, common fields such as i d, name and type are included, and the interface data transmission mode based on data transmission, pushing and synchronization is achieved. And custom fields are allowed to be expanded, a rule engine is used for realizing automatic conversion of data, and a data transmission channel is realized by adopting a factory mode. By simplifying the integration process among heterogeneous systems, developers do not need to write special conversion logic for each pair of systems, but utilize a uniform interface to perform data interaction, so that the complexity of system integration is remarkably reduced, and the development efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of metal coil controllers, in particular to a data transmission, push and synchronous interface data transmission method. Background Art

[0002] In the current wave of digital transformation, enterprise information systems are becoming increasingly complex and diversified. Data transmission, push and synchronization between different systems have become key requirements in enterprise IT architecture. With the development of Internet of Things (IoT), big data and artificial intelligence technologies, the complexity, real-time requirements and data volume of data interaction are increasing rapidly. This trend makes efficient and reliable data interaction solutions one of the core challenges of enterprise digital transformation.

[0003] At present, enterprises mainly use four data interaction methods: point-to-point direct integration, batch data synchronization, message queues and API calls. Point-to-point direct integration is suitable for small-scale systems and is simple and direct; batch data synchronization is often used for the migration and synchronization of large amounts of historical data; message queues (such as RocketMQ and RabbitMQ) improve system decoupling and are suitable for processing high-concurrency data streams; and API calls (such as RESTful API or HTTP / HTTPS) are widely used in microservice architectures and provide a high degree of flexibility. These methods have their own characteristics, and enterprises can choose the appropriate integration method based on their specific needs.

[0004] Defects and shortcomings of the existing technology:

[0005] 1. High integration complexity: When the number of systems increases, the integration complexity of point-to-point integration increases exponentially, resulting in high integration costs;

[0006] 2. Insufficient real-time performance: Batch data synchronization cannot meet the needs of real-time data interaction, affecting the timeliness of business decisions;

[0007] 3. Difficult configuration and maintenance: Although message queues improve system decoupling, they are complex to configure and increase maintenance difficulty;

[0008] 4. High development cost: API calling requires a lot of development work, which increases project cycle and labor costs;

[0009] 5. Data consistency is difficult to ensure: In a complex distributed environment, ensuring data consistency becomes a major challenge;

[0010] 6. Lack of unified management: Different integration methods lack a unified monitoring and management platform, which increases the difficulty of operation and maintenance;

[0011] 7. Limited scalability: Existing solutions often need to be redeveloped or significantly adjusted to deal with new systems or data sources, which lacks flexibility;

[0012] 8. Security issues: During data transmission, the lack of unified security policies and access control mechanisms increases the risk of data leakage;

[0013] The present invention aims to solve multiple technical challenges in system integration through innovative data transmission, push and synchronization interface solutions, reduce integration complexity, achieve high-performance real-time data processing, simplify configuration maintenance, reduce development workload, improve data consistency in distributed environments, provide a unified monitoring and management platform, design a flexible and scalable architecture, and enhance the security of data transmission. Summary of the invention

[0014] In view of the deficiencies in the prior art, the present invention provides a data transmission, push, and synchronous interface data transmission method.

[0015] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data transmission, push, and synchronous interface data transmission method, comprising the following steps:

[0016] S1: contacting the interface of the device through a pluggable data adapter, which will serve as a bridge for the interface;

[0017] S2: Use JSON Schema to define a common data model, allowing data exchange and validation between different components. Design a common schema that includes common fields such as id, name, type, and allows extension of custom fields.

[0018] S3: Use the rule engine to automate data transformation;

[0019] S4: Use factory mode to implement data transmission channel;

[0020] S5: A real-time data push service based on HTTP / HTTPS. The client establishes a connection through a specific URL. It supports the publish-subscribe model. The client subscribes to specific types of data updates. When new data arrives, the server automatically pushes it to the subscribed client.

[0021] S6: Supports multiple synchronization modes through a distributed scheduling system, uses distributed locks to ensure that tasks are not executed repeatedly in a cluster environment, implements job queues and work nodes, and dynamically allocates synchronization tasks;

[0022] S7: During the operation, use the exponential backoff algorithm to retry, increase the retry interval to avoid additional burden on the system, and set the maximum number of retries and the overall timeout;

[0023] S8: Define routing rules, including conditions and target systems. The rules are based on data content, metadata, or external conditions. The routing engine parses the incoming data, evaluates all rules, determines the matching target system, and forwards the data.

[0024] Preferably, the data adapter supports various data formats and transmission protocols (such as REST API, database, file system) for reading and writing.

[0025] Preferably, the rule engine defines various conversion rules, such as field mapping, data type conversion, and conditional judgment. The conversion rules are stored in a configuration file or a database and support dynamic updating.

[0026] Preferably, the factory mode supports a variety of message middleware (such as Kafka, RabbitMQ, ActiveMQ), the channel factory creates the corresponding transmission channel instance according to the configuration, supports dynamic selection of message middleware, and facilitates expansion and integration.

[0027] Preferably, the distributed scheduling system has multiple synchronization modes divided into timed synchronization, triggered synchronization and real-time synchronization.

[0028] Preferably, the exponential backoff algorithm calculates the waiting time before each retry, and its formula is:

[0029] Tn=Tbase×(2n)+Random_Factor

[0030] Tn is the waiting time for the nth retry;

[0031] Tbase is the initial waiting time, usually a fixed value in milliseconds or seconds;

[0032] n is the current number of retries, starting from 0 and increasing with each retry;

[0033] Random_Factor is a random jitter used to prevent congestion caused by multiple requests being retried simultaneously. It is usually a random value within a specified range. If random jitter is not required during the process, it can be deleted.

[0034] Preferably, the exponential backoff algorithm application sets a maximum waiting time to prevent the waiting time from being too long. Assuming that the maximum waiting time is Tmax, the formula is:

[0035] Tn=min(Tbase×(2n),Tmax)+Random_Factor

[0036] It is ensured that the waiting time will not exceed the Tmax limit.

[0037] Preferably, for tasks that fail continuously, they are transferred to a dead letter queue for subsequent manual intervention or special processing.

[0038] Preferably, a Web application is used as the management interface, providing visual tools for data flow configuration, allowing users to define data sources, transformation rules and target systems; real-time system monitoring function displays the status of each component, performance indicators and alarm information; viewing, creating and modifying scheduled tasks through the task management module; a centralized log system supports complex queries and analysis; and a role-based user management system is used to control access rights.

[0039] The present invention provides a data transmission, push, and synchronous interface data transmission method. It has the following beneficial effects:

[0040] 1. The present invention simplifies the integration process between heterogeneous systems through a unified data model and conversion mechanism. Developers no longer need to write special conversion logic for each pair of systems, but can use a unified interface for data interaction, which significantly reduces the complexity of system integration and improves development efficiency.

[0041] 2. The present invention enables the system to process a large number of concurrent messages through the data transmission channel of the high-performance message middleware. This design can meet the needs of high-concurrency scenarios, ensure fast and reliable data transmission between different systems, and effectively support the real-time business processing needs of enterprises.

[0042] 3. The present invention supports complex business logic processing by introducing a rule engine. The system can determine the data processing method according to predefined rules and adapt to various complex data processing scenarios. This flexibility enables the system to quickly respond to changes in business needs without large-scale code modifications.

[0043] 4. The present invention significantly improves the success rate of data transmission by introducing a multi-level retry mechanism and a dead letter queue. Even in the case of network instability or temporary system failure, data can be reliably transmitted and processed, which greatly reduces the risk of data loss and improves the reliability of the entire system.

[0044] 5. The present invention enables operation and maintenance personnel to quickly discover and locate problems through a comprehensive monitoring and alarm system. The system provides detailed logs and performance indicators, making problem diagnosis and resolution more efficient, which not only reduces the average failure time of the system, but also improves the overall operation and maintenance efficiency.

[0045] 6. Through the push mechanism of the present invention, the system can achieve millisecond-level data delay. This real-time performance can meet the needs of time-sensitive business scenarios such as real-time transactions and real-time monitoring, providing enterprises with faster decision-making support.

[0046] 7. The present invention enables the system to easily integrate new data sources or systems through plug-in design. Developers can develop new adapter plug-ins without modifying the core system. This design greatly reduces the time and effort required to add new systems or data sources, and improves the flexibility and scalability of the entire IT architecture.

[0047] 8. The present invention greatly simplifies the system configuration and maintenance process through visual configuration tools and a unified management interface, which not only reduces the system learning curve, but also reduces the time and human resources required for daily maintenance, thereby reducing the overall development and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example:

[0051] like Figure 1 As shown, the embodiment of the present invention provides a data transmission, push, and synchronous interface data transmission method, including the following steps:

[0052] S1: Contact the interface of the device through a pluggable data adapter. The data adapter will serve as a bridge for the interface. The data adapter supports various data formats and transmission protocols (such as REST API, database, file system, etc.) for reading and writing;

[0053] S2: Flow data into the model, use JSON Schema to define a common data model, allow data exchange and verification between different components, design a common schema, include common fields such as id, name, type, etc., and allow extension of custom fields;

[0054] S3: Use the rule engine to achieve automatic data conversion, define various conversion rules, such as field mapping, data type conversion, conditional judgment, etc., collect input data (facts) as the basis for rule judgment, and match the rules that meet the conditions according to the current facts. When multiple rules meet the conditions at the same time, the rule engine needs to use certain strategies (such as priority, timestamp, etc.) to decide which rules to execute, execute the selected rules, update the system status or perform other operations. After the rules are executed, new facts may be generated. At this time, the engine will match the rules again and repeat the above process until no new rules are triggered. The conversion rules can be stored in the configuration file or database and support dynamic update;

[0055] S4: The factory mode is used to implement data transmission channels, supporting a variety of message middleware (such as Kafka, RabbitMQ, and ActiveMQ). The channel factory creates corresponding transmission channel instances based on the configuration and supports dynamic selection of message middleware, which is easy to expand and integrate.

[0056] S5: A real-time data push service based on HTTP / HTTPS. The client establishes a connection through a specific URL and supports the publish-subscribe model. The client can subscribe to specific types of data updates. When new data arrives, the server will automatically push it to the subscribed clients.

[0057] S6: Through the distributed scheduling system, it supports multiple synchronization modes, including timed synchronization, triggered synchronization and real-time synchronization;

[0058] Scheduled synchronization: Scheduled synchronization is a mode that triggers task execution based on fixed time intervals. The implementation method can use the distributed task scheduling framework to define tasks such as "synchronize every 15 minutes". The scheduler triggers tasks regularly and puts them into the job queue. The execution of each task requires obtaining a distributed lock to ensure that only one working node executes it;

[0059] Triggered synchronization: Triggered synchronization starts tasks based on specific events, such as database changes, API calls, etc. The system monitors external events, such as file changes, database record insertions, etc. Once an event is triggered, a task is generated and added to the job queue, which is dynamically executed by the worker node, using message queues or subscription publishing mode (such as Kafka, RabbitMQ) to receive and distribute events. The worker node listens to the queue and executes the task;

[0060] Real-time synchronization: Real-time synchronization requires that tasks be executed immediately after being received, and that new tasks or data need to be synchronized be checked regularly. WebSocket, gRPC, etc. are used to push data or tasks in real time.

[0061] Use ZooKeeper's temporary nodes to implement distributed locks, ensure task coordination between multiple nodes, ensure that tasks are not repeated in a cluster environment, implement job queues and work nodes, and dynamically allocate synchronization tasks;

[0062] S7: During the operation, an exponential backoff algorithm is used for retrying, and the retry interval is increased to avoid additional burden on the system. The specific calculation formula is:

[0063] Tn=Tbase×(2n)+Random_Factor

[0064] Tn is the waiting time for the nth retry;

[0065] Tbase is the initial waiting time, which is usually a fixed value in milliseconds or seconds;

[0066] n is the current number of retries, starting from 0 and increasing with each retry;

[0067] Random_Factor is a random jitter, which is used to prevent congestion caused by multiple requests being retried at the same time. It is usually a random value within a specified range. If random jitter is not needed during the process, it can be deleted.

[0068] And the exponential backoff algorithm application will set a maximum waiting time to prevent the waiting time from being too long. Assuming the maximum waiting time is Tmax, the formula is:

[0069] Tn=min(Tbase×(2n),Tmax)+Random_Factor

[0070] Ensure that the waiting time does not exceed the Tmax limit. For tasks that fail continuously, they are transferred to the dead letter queue for subsequent manual intervention or special processing.

[0071] S8: Define routing rules, including conditions and target systems. The rules are based on data content, metadata, or external conditions. The routing engine parses the incoming data, evaluates all rules, determines the matching target system, and forwards the data.

[0072] Using a web application as the management interface, it provides visual tools for data flow configuration, allowing users to define data sources, transformation rules, and target systems; real-time system monitoring function displays the status of each component, performance indicators, and alarm information; view, create, and modify scheduled tasks through the task management module; centralized logging system supports complex queries and analysis; and a role-based user management system is used to control access rights.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data transmission, push, and synchronization interface data transmission method, characterized in that: The following steps are involved: S1: contacting the interface of the device through a pluggable data adapter, which will serve as a bridge for the interface; S2: Use JSON Schema to define a common data model, allowing data exchange and validation between different components. Design a common schema that includes common fields such as id, name, type, and allows extension of custom fields. S3: Use the rule engine to automate data transformation; S4: Use factory mode to implement data transmission channel; S5: A real-time data push service based on HTTP / HTTPS. The client establishes a connection through a specific URL. It supports the publish-subscribe model. The client subscribes to specific types of data updates. When new data arrives, the server automatically pushes it to the subscribed client. S6: Supports multiple synchronization modes through a distributed scheduling system, uses distributed locks to ensure that tasks are not executed repeatedly in a cluster environment, implements job queues and work nodes, and dynamically allocates synchronization tasks; S7: During the operation, use the exponential backoff algorithm to retry, increase the retry interval to avoid additional burden on the system, and set the maximum number of retries and the overall timeout; S8: Define routing rules, including conditions and target systems. The rules are based on data content, metadata, or external conditions. The routing engine parses the incoming data, evaluates all rules, determines the matching target system, and forwards the data.

2. A data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: Data adapters support reading and writing from various data formats and transport protocols (such as REST API, database, file system).

3. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: The rule engine defines various conversion rules, such as field mapping, data type conversion, and conditional judgment. The conversion rules are stored in configuration files or databases and support dynamic updates.

4. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: The factory mode supports a variety of message middleware (such as Kafka, RabbitMQ, ActiveMQ). The channel factory creates corresponding transmission channel instances based on the configuration and supports dynamic selection of message middleware for easy expansion and integration.

5. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: The distributed scheduling system has various synchronization modes: timed synchronization, triggered synchronization and real-time synchronization.

6. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: The exponential backoff algorithm calculates the waiting time before each retry, and its formula is: T n =T basc ×(2 n )+Random_Factor Tn is the waiting time for the nth retry; Tbase is the initial waiting time, usually a fixed value in milliseconds or seconds; n is the current number of retries, starting from 0 and increasing with each retry; Random_Factor is a random jitter used to prevent congestion caused by multiple requests being retried simultaneously. It is usually a random value within a specified range. If random jitter is not required during the process, it can be deleted.

7. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: The exponential backoff algorithm application sets a maximum waiting time to prevent the waiting time from being too long. Assuming the maximum waiting time is Tmax, the formula is: T n =min(T base ×(2 n ),T max )+Random_Factor It is ensured that the waiting time will not exceed the Tmax limit.

8. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: For tasks that continue to fail, they are transferred to the dead letter queue for subsequent manual intervention or special processing.

9. The data transmission, push, and synchronous interface data transmission method according to claim 1, characterized in that: Using a web application as a management interface, it provides a visual tool for data flow configuration, allowing users to define data sources, transformation rules, and target systems; Real-time system monitoring function displays the status of each component, performance indicators and alarm information; View, create and modify scheduling tasks through the task management module; The centralized log system supports complex query and analysis; and a role-based user management system to control access rights.

Images