A multi-source multi-target dynamic configuration data transmission platform

Abstract

The application discloses a kind of multi-source multi-target dynamic configuration data transmission platform, it is related to industrial data transmission technical field. Including: through data transmission platform, the information input in configuration interface is collected and handled, obtains the configuration information of data transmission task. Through configuration interface, the configuration information of various data transmission tasks is uniformly collected, different data sources and target end are adapted respectively relying on adapter, connection is established after completing code value matching and effective period double check in combination with security check code when task starts, then in turn, multi-source data extraction, field mapping format conversion are completed, and finally, according to scheduling strategy, write the data after processing into corresponding target end;But traditional transmission scheme mostly relies on hard coding development to connect logic, permission control only uses fixed account and has no time constraint, new system, change business rules all need to modify code and redeploy, and the cost of adaptation and adjustment is high.

Description

Technical Field

[0001] This invention relates to the field of industrial data transmission technology, and in particular to a multi-source, multi-target dynamically configured data transmission platform. Background Technology

[0002] In the process of digital transformation and industrial informatization construction in manufacturing enterprises, multi-source heterogeneous data transmission platforms, as core support facilities for cross-system data flow, play a crucial role in synchronizing production data, facilitating business collaboration, and integrating information between manufacturing execution systems and upstream and downstream business systems. Based on adapter adaptation and configurable data processing principles, these platforms achieve efficient and accurate data flow between multiple heterogeneous systems through dynamic configuration of data source connections, data extraction rules, field mapping transformations, and target writing strategies, ensuring the consistency of manufacturing business data and the efficiency of system collaborative operation. Data transmission platforms are widely used in data interfacing between MES and ERP, PLM, WMS, SCADA, and other systems in scenarios such as semiconductor manufacturing, electronic assembly, and flexible circuit board production. They are a key foundational component supporting manufacturing enterprises in breaking down information silos and achieving digital operations.

[0003] Currently, while traditional cross-system data transmission solutions possess basic data synchronization capabilities, significant technical bottlenecks remain. Most solutions employ a customized, hard-coded development model, with transmission rules and business logic embedded in the program code. This lacks the ability to dynamically adjust the entire process through configuration, requiring code modification and redeployment when business requirements change, resulting in low development iteration efficiency and slow response times. Some solutions rely on general-purpose ETL tools for data transfer, offering limited adaptability to heterogeneous data sources and target endpoints. Adding new data types requires separate development of adaptation programs, leading to high expansion costs and insufficient flexibility. Furthermore, some solutions employ rudimentary access control mechanisms, using fixed account authorization models and lacking time-sensitive security verification mechanisms bound to the data source. Access control adjustments are cumbersome and lack automatic expiration and recycling, failing to meet the requirements of hierarchical authorization and secure, controllable transmission management for production data in manufacturing scenarios, thus impacting the overall efficiency and security of cross-system data transfer. Summary of the Invention

[0004] The purpose of this invention is to provide a data transmission platform with dynamic configuration of multiple sources and multiple targets, which solves the technical problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-source, multi-target dynamically configured data transmission platform, comprising: The data transmission platform collects and processes the information input from the configuration interface to obtain the configuration information of the data transmission task; wherein, the configuration information includes data source connection information, target end connection information, data extraction rules, data conversion rules, transmission scheduling strategy, and security verification code bound to the data source; Through the data transmission platform, adapter matching is performed based on the data source connection information, and the corresponding data source adapter is loaded; adapter matching is also performed based on the target end connection information, and the corresponding target end adapter is loaded. The data transmission platform performs permission verification based on the security check code when the transmission task is started, in order to match the security code pre-configured by the data source and verify whether the current time is within the validity period. If both conditions are met, the data source connection is established. Through the data transmission platform, data is extracted and processed by the data source adapter according to the data extraction rules to obtain the data to be transmitted from the data source; Through the data transmission platform, field mapping and format conversion processing are performed on the data to be transmitted according to the data conversion rules to obtain the converted data. Through the data transmission platform, data is written to the target end adapter according to the transmission scheduling strategy, and the converted data is synchronized to the target end. The data source and target end can be any combination of database, file system, mail server and file transfer server. All transmission rules are set and take effect through configuration, without the need to write dedicated transmission program code.

[0006] Preferably, the following steps are also included: The configuration information of multiple data transmission tasks is stored and processed independently through the data transmission platform to obtain the independent configuration parameters corresponding to each task. The data transmission platform triggers the corresponding data transmission tasks according to the transmission scheduling strategy for each data transmission task, and starts the corresponding data transmission tasks accordingly. The transmission scheduling strategy includes a timer-based periodic triggering strategy and an event-based triggering strategy, and the data source connection information and target connection information of different data transmission tasks are independent of each other.

[0007] Preferably, the step of obtaining the data to be transmitted from the data source and synchronizing it to the target end specifically includes: Through the data transmission platform, the data extraction thread pool is invoked to extract data from multiple data sources in parallel, and the data to be transmitted from each data source is obtained synchronously. The data transmission platform calls the data conversion thread pool to perform format conversion and cleaning of the data to be transmitted, resulting in standardized transmission data. Through the data transmission platform, the data loading thread pool is invoked to perform parallel writing processing on multiple target ends, and the standardized transmission data is synchronized to each target end. The thread pools are connected sequentially in the order of data flow via a memory queue.

[0008] Preferably, the step of obtaining the data to be transmitted from the data source specifically includes: When the data source is a database, the data transmission platform performs query processing based on the structured query language query statements in the data extraction rules to extract the data to be transmitted from the database. When the data source is a file system, the data transmission platform reads and processes the data according to the file path and file format in the data extraction rules to obtain the data to be transmitted from the file system. When the data source is a mail server, the data transmission platform performs attachment extraction processing according to the email filtering conditions in the data extraction rules, and obtains the email attachments from the mail server as the data to be transmitted.

[0009] Preferably, the following steps are also included: The execution status of data transmission tasks is collected and processed in real time through the data transmission platform to obtain status operation data. Through the data transmission platform, statistical analysis and processing of the status operation data are performed to generate a transmission log that includes the amount of data transmitted, transmission time, number of successful records, and number of failed records. Through the data transmission platform, the failed tasks are marked with breakpoints and the breakpoint locations are recorded so that when the task is retried, the transmission can continue from the breakpoint location and a preset number of retries can be performed.

[0010] Preferably, the following steps are also included: When a data transmission task fails, an alarm is generated and processed according to a pre-configured alarm policy through the data transmission platform, and an alarm notification is received. The alarm notification is sent via email or SMS through a data transmission platform; the alarm notification includes the failed task identifier, the reason for failure, and the time of failure.

[0011] Preferably, when the target is a database, the step of synchronizing the transformed data to the target specifically includes: A database transaction is initiated before data is written through the data transmission platform; The converted data is written to the target database via the target adapter through the data transmission platform. The data transmission platform verifies the write results of all data. If all data is written successfully, the database transaction is committed; if any data write fails, the database transaction is rolled back.

[0012] Preferably, the following steps are also included: Through the data transmission platform, third-party adapters that conform to the unified interface specification can be connected via the adapter extension interface; The third-party adapter is registered through the data transmission platform. After registration, configuration options for the corresponding data source type or target type are added in the configuration interface.

[0013] Preferably, the following steps are also included: Through the data transmission platform, interface configuration is performed with the universal interface platform of the manufacturing execution system to establish a unified interface key system for sharing authentication rules and permission configurations. The business data of the manufacturing execution system is obtained through the data transmission platform and the universal interface platform. The data is synchronized to the target heterogeneous system through the data transmission platform according to the pre-configured transmission rules.

[0014] Preferably, the heterogeneous system includes at least one of an ERP system, a PLM system, a WMS system, and a SCADA system; The business data includes at least one of production work order data, process parameter data, quality inspection data, and equipment status data.

[0015] Compared with related technologies, the multi-source, multi-target dynamically configured data transmission platform provided by this invention has the following advantages: 1. This invention provides a multi-source, multi-target dynamically configurable data transmission platform. It collects configuration information for various data transmission tasks through a unified configuration interface, adapts to different data sources and target terminals using adapters, and establishes a connection upon task startup by combining security check codes to perform dual verification of code value matching and validity period. Then, it sequentially completes multi-source data extraction, field mapping format conversion, and finally writes the processed data to the corresponding target terminal according to the scheduling strategy. However, traditional transmission solutions mostly rely on hard-coded development of interface logic, and access control uses only fixed accounts with no time constraints. Adding new systems or changing business rules requires code modification and redeployment, resulting in high adaptation and adjustment costs. Therefore, this invention uses visual configuration to implement all transmission logic without writing program code, and combines it with a time-sensitive data source-specific security verification mechanism to complete access control. This approach not only significantly improves the efficiency of data interface adjustment for multiple heterogeneous manufacturing systems but also enables hierarchical control of production data access permissions, effectively reducing the security risks of data leakage and unauthorized access. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating steps S100-S400 of a multi-source, multi-target dynamically configured data transmission platform according to the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 As shown, a multi-source, multi-target dynamically configurable data transmission platform includes: S100. The information input into the configuration interface is collected and processed through the data transmission platform to obtain the configuration information of the data transmission task; wherein, the configuration information includes data source connection information, target end connection information, data extraction rules, data conversion rules, transmission scheduling strategy, and security verification code bound to the data source; S200: Through the data transmission platform, perform adapter matching processing based on the data source connection information and load the corresponding data source adapter; perform adapter matching processing based on the target end connection information and load the corresponding target end adapter. S300. Through the data transmission platform, when the transmission task is started, permission verification is performed based on the security check code to match the security code pre-configured by the data source and verify whether the current time is within the validity period. If both conditions are met, the data source connection is established. S400. Through the data transmission platform, data extraction processing is performed through the data source adapter according to the data extraction rules to obtain the data to be transmitted from the data source; S500: Through the data transmission platform, perform field mapping and format conversion processing on the data to be transmitted according to the data conversion rules to obtain the converted data; S600: Through the data transmission platform, according to the transmission scheduling strategy, data is written through the target adapter, and the converted data is synchronized to the target end. Those skilled in the art will understand that manufacturing enterprises typically have multiple heterogeneous systems such as ERP, PLM, MES, and SCADA. These systems differ in data formats, database types, and permission systems. Traditional data transmission solutions require developing separate interface programs and writing ETL code for each set of interfaces, resulting in long development cycles and poor flexibility in adjustments. This solution defines all transmission rules in a configurable manner. It first obtains complete task configuration and security verification rules, then adapts to different types of data sources and target terminals based on the adapter pattern. Combined with a time-sensitive security verification mechanism to ensure data access permissions, it ultimately completes data extraction, transformation, and end-to-end transmission. No dedicated transmission program code needs to be written throughout the process. This solves the pain point of large workload in heterogeneous system integration, ensures data transmission security and configurability, and can quickly respond to the ever-changing data flow needs in manufacturing scenarios.

[0019] Example: Step S100: Collect and process the information input from the configuration interface through the data transmission platform to obtain the configuration information of the data transmission task. This specifically includes the following steps: S101. Through the data transmission platform, perform information extraction and processing on the data transmission task to be created, and determine the task identifier and business scenario type of the data transmission task. S102. Through the data transmission platform, the configuration database is retrieved and processed according to the task identifier and business scenario type to obtain the default configuration template and historical reference configuration parameters for the corresponding business scenario. S103. Using the data transmission platform, the input custom parameters are matched and filled based on the default configuration template to generate complete data transmission task configuration information; wherein, step S103, using the data transmission platform, the input custom parameters are matched and filled based on the default configuration template to generate complete data transmission task configuration information specifically includes the following steps: S1031. Through the data transmission platform, perform format verification processing on the input data source connection information and the target connection information to determine the legality of the connection address and authentication information format; It is understandable that the data source and target types cover a variety of categories such as databases, file systems, mail servers, and file transfer servers. The connection parameter formats and authentication methods differ for different categories. Format verification can identify parameter errors in advance and avoid connection failures when subsequent transmission tasks are started. S1032. Through the data transmission platform, perform syntax verification on the input data extraction rules and data transformation rules to determine the syntax correctness and executability of the rule statements; S1033. Through the data transmission platform, generate a security verification code that is bound to the corresponding data source for the current data transmission task, and configure the access permission scope and validity period of the corresponding security code. S1034. Through the data transmission platform, all verified parameters and security verification codes are integrated into the default configuration template to generate complete and directly effective data transmission task configuration information. In this embodiment, data transmission requirements in manufacturing scenarios are often frequently adjusted according to production plans and system integration needs. Traditional solutions require developers to modify code and redeploy to complete the adjustments, resulting in long response cycles. By using preset scenario-specific default configuration templates combined with automated parameter verification, business personnel do not need to write configuration rules from scratch. They only need to modify a few custom parameters to complete the construction of the transmission task. At the same time, the built-in format and syntax verification can significantly reduce the probability of configuration errors. Combined with the security verification code mechanism that is bound to the data source one by one, configuration efficiency is greatly improved while ensuring the security of data access permissions, effectively reducing the configuration threshold and daily operation and maintenance costs of data transmission tasks.

[0020] Example: Step S200: Through the data transmission platform, adapter matching processing is performed based on the data source connection information, and the corresponding data source adapter is loaded; adapter matching processing is performed based on the target end connection information, and the corresponding target end adapter is loaded, specifically including the following steps: S201. Through the data transmission platform, perform type identification processing on the data source connection information and target connection information in the configuration information to determine the data source type and target type; S202. Through the data transmission platform, perform matching and retrieval processing on the adapter registration library according to the data source type, and retrieve the data source adapter of the corresponding type; perform matching and retrieval processing on the adapter registration library according to the target end type, and retrieve the target end adapter of the corresponding type. S203. Through the data transmission platform, initialize the data source adapter and the target adapter, and establish a connection channel between the adapter and the corresponding data terminal. Specifically, step S203, which involves initializing the data source adapter and target adapter through the data transmission platform and establishing a connection channel between the adapter and the corresponding data terminal, includes the following steps: S2031. Through the data transmission platform, the corresponding connection parameters and security verification code are passed to the data source adapter to start the data source connection initialization; It is understandable that different types of data sources correspond to different connection protocols and drivers. The adapter has built-in standard drivers for the corresponding types, so no additional installation or deployment is required. S2032. Through the data transmission platform, the corresponding connection parameters and write permission configuration are transmitted to the target adapter to start the target connection initialization; S2033. Through the data transmission platform, perform connectivity testing on the connection status between the data source and the target end to determine the availability of the connection channel; S2034. If the connectivity test passes, the adapter loading and channel establishment are completed; if the test fails, a parameter error message is returned and the subsequent process is terminated. In this embodiment, the manufacturing enterprise has multiple heterogeneous systems and data storage methods. Traditional integration solutions require developing a separate adapter for each data type, which is labor-intensive and has poor scalability. Through a plug-in adapter registration library and an automatic matching and loading mechanism, the platform can automatically identify the data source type and call the corresponding adapter to complete the connection. There is no need to modify the core engine code for new data source types; simply registering new adapters is sufficient for expansion, significantly improving the platform's adaptability to heterogeneous data sources and its scalability.

[0021] Example: Step S300: Through the data transmission platform, when the transmission task starts, permission verification is performed based on the security verification code to match the pre-configured security code of the data source and verify whether the current time is within the validity period. If both conditions are met, a data source connection is established. This specifically includes the following steps: S301. Through the data transmission platform, the security verification code carried by the data transmission task is parsed and processed to extract the security code identifier and timestamp information; S302. Through the data transmission platform, the extracted security code identifier is matched and verified with the security code pre-configured in the data source to determine the permission matching status of the security code. S303. Through the data transmission platform, the timestamp in the security verification code is compared and verified with the current system time to determine whether the security code is within its validity period. S304. If the permissions match and the validity period is valid, the verification passes, and a formal data source connection channel is established; if any verification fails, a permission verification failure message is returned, and the transmission task is terminated. Specifically, step S302, which involves matching the extracted security code identifier with the pre-configured security code of the data source through the data transmission platform to determine the permission matching status of the security code, includes the following steps: S3021. Retrieve the pre-configured security code baseline value and the corresponding data operation permission range from the data source's permission configuration table through the data transmission platform; It is understandable that different security codes correspond to different data access permissions. Some security codes only support data reading operations, while others support full data reading and writing operations. The scope of permissions is bound to business needs. S3022. Through the data transmission platform, the parsed security code identifier and the security code base value are matched to obtain the matching result; S3023. Through the data transmission platform, determine the data operation permission range corresponding to the current security code based on the matching result, and use it as a permission constraint for subsequent data operations; S3024. If the match is consistent, the permission verification passes; if the match is inconsistent, the permission verification fails. In this embodiment, access permissions in traditional data transmission schemes are usually hard-coded in the interface program. Adjusting permissions requires modifying the code and redeploying, and there is a lack of timeliness control mechanism, posing a security risk of expired permissions not being revoked. Through a two-layer verification mechanism of security code matching and validity period verification, all permission rules can be dynamically adjusted through configuration without modifying the program code. At the same time, the validity period verification can automatically revoke expired permissions, which not only improves the flexibility of permission management, but also ensures the security of access to production data in manufacturing scenarios.

[0022] Example: Step S400: Through the data transmission platform, data extraction processing is performed through the data source adapter according to the data extraction rules. The specific steps to obtain the data to be transmitted from the data source include the following: S401. Through the data transmission platform, the data extraction rules are sent to the data source adapter, and the query conditions, field ranges and data volume constraints in the extraction rules are parsed. S402. Through the data transmission platform, a data reading request is initiated to the data source through the data source adapter, and the corresponding data reading operation is executed according to the extraction rules; S403. Through the data transmission platform, the raw data returned by the data source is formatted to obtain standardized data to be transmitted. S404. Through the data transmission platform, the data to be transmitted is written into the memory queue corresponding to the data extraction thread pool, waiting for subsequent conversion processing.

[0023] Specifically, step S402, which involves initiating a data read request to the data source through the data transmission platform and the data source adapter, and performing the corresponding data read operation according to the extraction rules, includes the following steps: S4021. When the data source is a database, the data source adapter executes a query operation based on the structured query language query statement in the data extraction rules to read the data to be transmitted that meets the conditions from the database. It is understandable that different types of databases correspond to different SQL dialects. The adapter has built-in dialect conversion logic and can automatically adapt to the syntax differences of mainstream databases such as MSSQL, MySQL, and Oracle. S4022. When the data source is a file system, the data source adapter performs a read operation based on the file path and file format in the data extraction rules, and parses the structured data to be transmitted from the file system. S4023. When the data source is a mail server, the data source adapter performs a filtering operation according to the mail filtering conditions in the data extraction rules and extracts the corresponding email attachments from the mail server as data to be transmitted. S4024. After the data reading is completed, the data source adapter will encapsulate the reading results into a standard data format and return them.

[0024] In this embodiment, the manufacturing enterprise has diverse data source types, and different systems have different data storage formats and access protocols. Traditional solutions require developing separate reading programs for each type of data source, resulting in a large development workload and poor scalability when adding new data sources. Through a plug-in adapter architecture and unified data extraction rules, the platform can automatically adapt to different types of data sources. Business personnel only need to configure extraction rules to complete the unified extraction of multi-source data, eliminating the need for separate development for each data source type and significantly improving the access efficiency of heterogeneous data sources.

[0025] Example: Step S500: Through the data transmission platform, field mapping and format conversion processing are performed on the data to be transmitted according to the data conversion rules to obtain the converted data. Specifically, this includes the following steps: S501: Through the data transmission platform, pull the data to be transmitted from the memory queue and load the corresponding data conversion rules; S502. Through the data transmission platform, according to the field mapping relationship in the data conversion rules, perform source field to target field mapping matching processing on the data to be transmitted to obtain the field-aligned data; S503. Through the data transmission platform, according to the format conversion requirements in the data conversion rules, perform data type conversion, numerical calculation and data cleaning processing on the data after field alignment; S504. Through the data transmission platform, the processed data is encapsulated into a format compatible with the target end, and the converted data is written into the memory queue corresponding to the data loading thread pool.

[0026] Specifically, step S503, which involves performing data type conversion, numerical calculation, and data cleaning on the aligned data through the data transmission platform according to the format conversion requirements in the data conversion rules, includes the following steps: S5031. Through the data transmission platform, perform data type conversion processing to convert the source data field type into the field type required by the target end; It is understandable that the data types of fields with the same meaning often differ between heterogeneous systems, such as different date formats and numerical precision. Unified conversion can avoid format errors when writing to the target end. S5032. Through the data transmission platform, perform data cleaning processing to remove invalid data, complete missing fields, and filter duplicate records to ensure data integrity and consistency. S5033. Through the data transmission platform, numerical calculation processing is performed according to the pre-configured calculation rules to generate the derived field data required by the target end; S5034. After all conversion processing is completed, output standardized writable data.

[0027] In this embodiment, traditional ETL tools require writing dedicated transformation scripts for data transformation logic. When business rules change, these scripts need to be modified and re-debugged, resulting in high maintenance costs. With configurable data transformation rules, field mapping, format conversion, and data cleaning can all be achieved through parameter configuration, eliminating the need to write transformation code. Business personnel can quickly adjust the transformation logic according to their needs, significantly improving the efficiency and flexibility of adjusting data flow rules.

[0028] Example: Step S600: Through the data transmission platform, data is written to the target end adapter according to the transmission scheduling strategy, and the converted data is synchronized to the target end. This specifically includes the following steps: S601. Through the data transmission platform, determine whether the triggering conditions are met according to the transmission scheduling strategy. If they are met, start the data writing process. S602. Through the data transmission platform, the converted data is pulled from the memory queue and sent to the corresponding target adapter; S603. Through the data transmission platform, the converted data is written to the corresponding target end through the target end adapter; S604. Through the data transmission platform, perform statistical verification processing on the writing results, record the number of successfully written records and the number of failed records, and generate the execution result of a single transmission. Specifically, step S603, writing the converted data to the corresponding target end through the target end adapter via the data transmission platform, includes the following steps: S6031. When the target is a database, the target adapter starts a database transaction before writing. Understandably, database transactions can guarantee the atomicity of data writes, avoiding data inconsistency caused by partial success and partial failure of writes, and are especially suitable for the integrity requirements of production data in manufacturing scenarios; S6032, The target adapter writes the converted data into the target database in batches; S6033. When the target is a file system, mail server or file transfer server, the target adapter performs data writing or push operations according to the corresponding target protocol specifications. S6034. After all data is written, if it is a database target and all data is successfully written, the transaction is committed; if any data writing fails, the transaction is rolled back. In this embodiment, production data in a manufacturing scenario has extremely high consistency requirements. Partial write failures can lead to data gaps, affecting quality traceability and production statistics. Through the database transaction mechanism and multi-type target adaptation architecture, both data atomicity in database writing scenarios are guaranteed, and unified writing to multiple target terminals such as files, emails, and FTP is supported. With configurable scheduling strategies, automatic data synchronization can be achieved without developing timer programs, balancing data reliability and scenario adaptability.

[0029] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, the phrase "comprising an element defined as..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-source, multi-target dynamically configurable data transmission platform, characterized in that: include: The data transmission platform collects and processes the information input from the configuration interface to obtain the configuration information of the data transmission task; wherein, the configuration information includes data source connection information, target end connection information, data extraction rules, data conversion rules, transmission scheduling strategy, and security verification code bound to the data source; Through the data transmission platform, adapter matching is performed based on the data source connection information, and the corresponding data source adapter is loaded; adapter matching is also performed based on the target end connection information, and the corresponding target end adapter is loaded. The data transmission platform performs permission verification based on the security check code when the transmission task is started, in order to match the security code pre-configured by the data source and verify whether the current time is within the validity period. If both conditions are met, the data source connection is established. Through the data transmission platform, data is extracted and processed by the data source adapter according to the data extraction rules to obtain the data to be transmitted from the data source; Through the data transmission platform, field mapping and format conversion processing are performed on the data to be transmitted according to the data conversion rules to obtain the converted data. Through the data transmission platform, data is written to the target end adapter according to the transmission scheduling strategy, and the converted data is synchronized to the target end. The data source and target end can be any combination of database, file system, mail server and file transfer server. All transmission rules are set and take effect through configuration, without the need to write dedicated transmission program code.

2. The data transmission platform with multi-source and multi-target dynamic configuration according to claim 1, characterized in that: It also includes the following steps: The configuration information of multiple data transmission tasks is stored and processed independently through the data transmission platform to obtain the independent configuration parameters corresponding to each task. The data transmission platform triggers the corresponding data transmission tasks according to the transmission scheduling strategy for each data transmission task, and starts the corresponding data transmission tasks accordingly. The transmission scheduling strategy includes a timer-based periodic triggering strategy and an event-based triggering strategy, and the data source connection information and target connection information of different data transmission tasks are independent of each other.

3. The data transmission platform with multi-source and multi-target dynamic configuration according to claim 1, characterized in that: The steps of obtaining data to be transmitted from the data source and synchronizing it to the target end specifically include: Through the data transmission platform, the data extraction thread pool is invoked to extract data from multiple data sources in parallel, and the data to be transmitted from each data source is obtained synchronously. The data transmission platform calls the data conversion thread pool to perform format conversion and cleaning of the data to be transmitted, resulting in standardized transmission data. Through the data transmission platform, the data loading thread pool is invoked to perform parallel writing processing on multiple target ends, and the standardized transmission data is synchronized to each target end. The thread pools are connected sequentially in the order of data flow via a memory queue.

4. The data transmission platform with multi-source and multi-target dynamic configuration according to claim 1, characterized in that: The steps for obtaining the data to be transferred from the data source specifically include: When the data source is a database, the data transmission platform performs query processing based on the structured query language query statements in the data extraction rules to extract the data to be transmitted from the database. When the data source is a file system, the data transmission platform reads and processes the data according to the file path and file format in the data extraction rules to obtain the data to be transmitted from the file system. When the data source is a mail server, the data transmission platform performs attachment extraction processing according to the email filtering conditions in the data extraction rules, and obtains the email attachments from the mail server as the data to be transmitted.

5. A multi-source, multi-target dynamically configured data transmission platform according to claim 1, characterized in that: It also includes the following steps: The execution status of data transmission tasks is collected and processed in real time through the data transmission platform to obtain status operation data. Through the data transmission platform, statistical analysis and processing of the status operation data are performed to generate a transmission log that includes the amount of data transmitted, transmission time, number of successful records, and number of failed records. Through the data transmission platform, the failed tasks are marked with breakpoints and the breakpoint locations are recorded so that when the task is retried, the transmission can continue from the breakpoint location and a preset number of retries can be performed.

6. A multi-source, multi-target dynamically configured data transmission platform according to claim 5, characterized in that: It also includes the following steps: When a data transmission task fails, an alarm is generated and processed according to a pre-configured alarm policy through the data transmission platform, and an alarm notification is received. The alarm notification is sent via email or SMS through a data transmission platform; the alarm notification includes the failed task identifier, the reason for failure, and the time of failure.

7. The data transmission platform with multi-source and multi-target dynamic configuration according to claim 1, characterized in that: When the target is a database, the steps to synchronize the transformed data to the target include: A database transaction is initiated before data is written through the data transmission platform; The converted data is written to the target database via the target adapter through the data transmission platform. The data transmission platform verifies the write results of all data. If all data is written successfully, the database transaction is committed; if any data write fails, the database transaction is rolled back.

8. A multi-source, multi-target dynamically configured data transmission platform according to claim 1, characterized in that: It also includes the following steps: Through the data transmission platform, third-party adapters that conform to the unified interface specification can be connected via the adapter extension interface; The third-party adapter is registered through the data transmission platform. After registration, configuration options for the corresponding data source type or target type are added in the configuration interface.

9. A multi-source, multi-target dynamically configured data transmission platform according to claim 1, characterized in that: It also includes the following steps: Through the data transmission platform, interface configuration is performed with the universal interface platform of the manufacturing execution system to establish a unified interface key system for sharing authentication rules and permission configurations. The business data of the manufacturing execution system is obtained through the data transmission platform and the universal interface platform. The data is synchronized to the target heterogeneous system through the data transmission platform according to the pre-configured transmission rules.

10. A multi-source, multi-target dynamically configured data transmission platform according to claim 9, characterized in that: The heterogeneous system includes at least one of ERP system, PLM system, WMS system and SCADA system; The business data includes at least one of production work order data, process parameter data, quality inspection data, and equipment status data.

Images