Production information integration system based on visual configuration

Abstract

The invention provides a production information integration system based on visual configuration. Comprising a component management module used for providing a visual configuration interface; the data source management module is used for managing a data source component and configuring and verifying a data source according to input data source connection parameters; the data processing module is used for processing the data acquired by the data source management module according to the operation logic configured by the component management module; the target data management module is used for receiving and storing the data processed by the data processing module; the job scheduling module is used for scheduling the jobs configured by the component management module according to a preset scheduling rule; and the monitoring and diagnosis module is used for monitoring the operation state of each module in real time, recording the operation execution log of each module, detecting the abnormal condition in the operation of the system and generating a diagnosis report and a troubleshooting suggestion. The system development cost can be reduced, the system development period can be shortened, and the expandability and compatibility of the system can be improved.

Description

Technical Field

[0001] The present application relates to the field of industrial automation technology, and in particular to a production information integration system based on visualization configuration. Background Art

[0002] Currently, industrial automation and production management software have become an important part of the field of modern intelligent manufacturing. Its core goal is to optimize production processes, improve resource utilization efficiency and support management decisions through information and automation technologies.

[0003] Existing production management and information system integration solutions are usually based on fixed software architecture and modular development methods. Developers need to manually code to achieve interconnection and functional expansion between systems. Although some systems have certain integration functions, they are not flexible and user-friendly in complex manufacturing scenarios, and it is difficult to meet personalized needs and rapidly changing production environments.

[0004] The existing technology still has the following shortcomings in information system integration:

[0005] First, traditional integration methods rely on a lot of code writing, especially system modification and secondary development for customized needs, which not only has a long development cycle but also high maintenance costs.

[0006] Secondly, due to the tight coupling of module design, when faced with production equipment or technology updates, expanding new functions or integrating new equipment requires a lot of time and resources.

[0007] In addition, the lack of an intuitive visual configuration interface makes it difficult for non-technical users to quickly get started, and the operational flexibility is limited, which restricts the popularity and application of the system among non-professionals.

[0008] Finally, equipment and software vendors follow different standards and protocols, resulting in poor compatibility of data transmission between modules, which can easily cause data loss or incompleteness, affecting system stability and reliability. Summary of the invention

[0009] In view of this, an embodiment of the present application provides a production information integration system based on visual configuration to solve the problems of high development cost, long development cycle, poor system scalability, complex user operation and poor compatibility existing in the prior art.

[0010] The embodiment of the present application provides a production information integration system based on visual configuration, including: a component management module, which is used to provide a visual configuration interface so that a user can select and configure multiple functional components by dragging and dropping through the visual configuration interface; a data source management module, which is used to manage data source components, configure and verify the data source according to the input data source connection parameters, and generate a list of available data sources for the component management module to call; a data processing module, which is used to process the data obtained by the data source management module according to the job logic configured by the component management module; a target data management module, which is used to receive and store the data processed by the data processing module, and the target data management module includes an interface component, which is used to write the processed data to an external target system; a job scheduling module, which is used to schedule the jobs configured by the component management module according to preset scheduling rules, wherein the scheduling rules include timed scheduling, periodic scheduling and event-driven scheduling; a monitoring and diagnosis module, which is used to monitor the operating status of each module in real time, record the job execution log of each module, detect abnormal conditions in system operation, and generate diagnostic reports and troubleshooting suggestions.

[0011] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0012] The component management module is used to provide a visual configuration interface so that users can select and configure multiple functional components by dragging and dropping through the visual configuration interface; the data source management module is used to manage the data source components, configure and verify the data source according to the input data source connection parameters, and generate a list of available data sources for the component management module to call; the data processing module is used to process the data obtained by the data source management module according to the job logic configured by the component management module; the target data management module is used to receive and store the data processed by the data processing module, and the target data management module includes an interface component, which is used to write the processed data to the external target system; the job scheduling module is used to schedule the jobs configured by the component management module according to the preset scheduling rules, wherein the scheduling rules include timed scheduling, periodic scheduling and event-driven scheduling; the monitoring and diagnosis module is used to monitor the operating status of each module in real time, record the job execution log of each module, detect abnormal conditions in the system operation, and generate diagnostic reports and troubleshooting suggestions. This application can reduce the system development cost and cycle, and improve the scalability and compatibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 It is a schematic diagram of the overall architecture of the production information integration system in the actual application scenario provided by the embodiment of the present application;

[0015] Figure 2 It is a schematic diagram of the overall structure of a production information integration system based on visual configuration provided in an embodiment of the present application;

[0016] Figure 3 It is a flowchart of job configuration management provided by an embodiment of the present application;

[0017] Figure 4 It is a schematic diagram of the data source management process provided by an embodiment of the present application;

[0018] Figure 5 It is a schematic diagram of the conversion and mapping component design process provided in the embodiment of the present application;

[0019] Figure 6 It is a flowchart of the job scheduling method and triggering conditions provided in the embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] The technical fields involved in this application mainly include industrial automation and production management software. Industrial automation refers to the automation and intelligence of production processes through the use of control technology and information systems. It covers everything from automated control of production equipment to the integration and optimization of overall factory operations, aiming to improve production efficiency and reduce human intervention. Production management software is software designed specifically for industrial manufacturing processes, with the goal of improving resource allocation efficiency, optimizing operating processes, providing data analysis support, and helping management make more accurate decisions by monitoring and managing production links.

[0022] In the field of intelligent manufacturing, existing production management and information system integration solutions usually rely on fixed software architectures and complex programming processes to achieve interconnection and data interaction between systems. Traditional information system integration methods are mostly based on modular development. Developers need to have high programming skills and manually write a lot of code to complete system integration and function expansion. This approach not only has a long development cycle and high maintenance costs, but also makes it difficult to adapt to the changing production needs and personalized customization requirements in the manufacturing industry.

[0023] Although existing production management software has certain integration functions, most systems lack intuitive user interfaces and cannot provide flexible operation methods. For non-professional technicians, it is difficult to quickly deploy and modify production processes, which limits the application scope of the system. In addition, when facing new technologies or equipment updates, existing integrated software usually requires re-adjustment of the system and secondary development, resulting in additional time and resource consumption.

[0024] The technical defects or problems of the existing technical solutions are mainly manifested in the following aspects:

[0025] 1) High development cost and long cycle: System integration involves a lot of manual coding, especially in the customized production process, where requirements change frequently and secondary development and maintenance of the system are difficult.

[0026] 2) Poor scalability: The modular design of traditional integration methods is difficult to quickly adapt to changes in production equipment or technology, resulting in a lot of time required for adaptation when expanding new functions or integrating new equipment.

[0027] 3) Lack of user-friendliness: The operating interface of most integrated software is not intuitive enough, the usage threshold is high, and it is difficult for non-technical users to get started, which limits the flexibility of operation and the popularity of the system.

[0028] The reason for these problems is that existing technologies rely on traditional code development methods, lack flexible visual configuration methods, and cannot quickly respond to the increasingly complex and changing needs of the manufacturing industry.

[0029] Further, the causes of the existing technical problems are analyzed as follows:

[0030] 1) Limitations of software architecture design:

[0031] Most existing information system integrations are based on traditional closed architectures, where modules are tightly coupled and cannot achieve flexible combination and configuration of components. This architecture limits the system's customization capabilities, resulting in a large amount of manual coding and secondary development when faced with changing production needs or new function expansions, which cannot meet the requirements of rapid response.

[0032] 2) Diversity of data integration standards:

[0033] Since the equipment and software in smart manufacturing systems usually come from different suppliers and follow different standards and protocols, data transmission and compatibility issues between modules frequently occur during system integration. The differences between different standards make it difficult for modules to work seamlessly together, increasing the complexity of system integration.

[0034] 3) Limitations of technical implementation capabilities and resources:

[0035] When deploying and maintaining complex system integration solutions, some companies may find it difficult to cope with complex customization requirements due to insufficient technical capabilities or limited resources. When faced with system optimization, upgrades or new function integration, these companies often need to rely on external development teams, resulting in high project costs, long cycles, and difficulty in ensuring the sustainability and scalability of the system.

[0036] It can be seen that the existing factory production management systems mostly adopt a closed architecture, with fixed functions between modules, which makes it difficult to flexibly adapt to the personalized needs of different factories. When factories need to customize certain specific functions or adjust existing processes, a lot of custom development is often required, which increases the cost and time of implementation. In addition, software modules from different suppliers usually use different data formats or communication protocols, which makes compatibility issues prone to occur during system integration. This complexity may cause data to be lost or incomplete during transmission, which in turn affects the stability and reliability of the system.

[0037] In view of the problems existing in the prior art, the purpose of this application is to propose an integrated software solution based on configuration. The main technical problem solved by this application is how to achieve seamless integration and functional expansion between different system components through visual and flexible drag-and-drop operations, thereby reducing the development complexity of the system and improving scalability and user-friendliness. This application aims to provide flexible configuration tools so that non-technical users can also quickly customize and modify system functions, solve the reliance of prior art on professional skills and coding requirements, shorten the development cycle and improve production efficiency.

[0038] Before describing the contents of the technical solution of the present application in detail, first, the overall architecture involved in the technical solution of the present application in an actual scenario is explained with reference to the accompanying drawings. Figure 1 is a schematic diagram of the overall architecture of the production information integration system in the actual application scenario provided by the embodiment of the present application, such as Figure 1 As shown, the production information integration system can include the following:

[0039] 1. Core modules and functions of the system

[0040] 1. Data Integration Tools

[0041] The data integration tool is the core functional unit of the system, responsible for data extraction, conversion and loading, and realizing the seamless flow of data between different systems. It includes the following components:

[0042] Data source component: used to establish connections with multiple data sources (files, SQL, HTTP, etc.), extract raw data and provide it to subsequent modules.

[0043] Data conversion processor: performs conversion operations on the extracted raw data, including field mapping, format conversion, unit conversion, and data cleaning to meet the requirements of the target system.

[0044] Data loading component: responsible for writing the processed data to the target storage device or external system, such as a database (HSM-DB) or a buffer service.

[0045] Data cache: Temporarily store data during data flow to improve the real-time performance of data processing and system response speed.

[0046] 2. Job Management Module

[0047] This module is the core operating engine of data integration, which realizes the creation, management and execution of job tasks through the configuration interface:

[0048] Job execution engine: responsible for scheduling job tasks and executing data extraction, processing and storage operations in sequence according to preset rules (such as timing, cycle or event-driven).

[0049] Job management module: used to define and monitor job processes, including starting, stopping, pausing and resuming tasks.

[0050] Plug-in management module: supports module expansion and function upgrades, and implements new functions or supports new data processing requirements by introducing plug-ins.

[0051] 3. Log and exception management module

[0052] Used to improve system reliability and operation and maintenance capabilities:

[0053] Log management: records the operating status of each module and the results of job execution to facilitate user tracing and analysis.

[0054] Exception handling module: detects and captures exceptions that occur during operation, generates diagnostic reports, and notifies the monitoring module to take corresponding measures.

[0055] 4. User interface module

[0056] Provides an interactive interface for operation configuration and operation monitoring:

[0057] Configuration / Preview Mode UI: used for visual configuration and workflow design, supports users to define the connection relationship between modules by dragging and dropping and preview the data flow in real time.

[0058] Monitor mode UI: Provides visual feedback of real-time job status, showing the running status, log information and exception reports of the current task.

[0059] 2. Core advantages of the system

[0060] 1. Visual configuration and customization

[0061] The system introduces intuitive visual design tools, allowing users to configure the entities, fields, pages and interaction logic of each module by dragging and dropping, without the need for programming experience.

[0062] 2. Data mapping, format conversion and logic definition between modules are completed by dragging and dropping component configuration, which significantly reduces the complexity of system integration.

[0063] 3. Support modular expansion and automated deployment mechanism to reduce the risk of manual operation and configuration errors. The technical architecture design of the plug-in mechanism enables rapid expansion and dynamic loading of modules. The system's self-diagnosis and automatic update mechanism are used to quickly troubleshoot and complete module upgrades.

[0064] 3. Practical application scenarios of the system

[0065] The system is applicable to a variety of production information management and integration scenarios, and can achieve:

[0066] Cross-platform data integration: supports efficient compatibility between multiple data sources (such as SQL, files, HTTP protocol, etc.) and target systems to meet the needs of complex factory environments.

[0067] Flexible task scheduling: Efficiently schedule tasks through timing, cycle or event-driven methods to meet the requirements of different production links.

[0068] Real-time operation monitoring: Through the monitoring module and log management module, real-time feedback of task operation status is provided to support abnormality investigation and diagnosis.

[0069] Rapid expansion capability: Through plug-in mechanism and modular design, new devices or data processing logic can be quickly adapted to reduce expansion costs.

[0070] 4. Workflow of the overall system architecture

[0071] Users design the operation process through the configuration / preview mode UI and configure the parameters and connection relationships of each component.

[0072] The data source management module extracts the original data from the data source and converts and formats the data through the data conversion processor.

[0073] The job management module executes job tasks according to preset scheduling rules and coordinates the order of data extraction, processing and storage through the job execution engine.

[0074] The converted data is stored in the local database (HSM-DB) through the data loading component or transmitted to the external target system through the interface component.

[0075] The system records the task running status in real time and displays it to the user through the monitoring mode UI. Exception information is captured by the exception handling module and fed back to the monitoring interface.

[0076] Through the above-mentioned system architecture design, the production information integration system of this application has been greatly improved in flexibility, compatibility and ease of operation, solving the technical problems of long development cycle, high maintenance cost and high user operation threshold in the existing technology, and providing an efficient information integration solution for intelligent manufacturing and industrial automation.

[0077] The following is an explanation of the technical terms involved in the technical solution of this application, which may specifically include the following technical terms:

[0078] APS: Advanced Planning and Scheduling System, advanced planning and scheduling system.

[0079] ERP: Enterprise Resource Planning.

[0080] MES: Manufacturing Execution System.

[0081] WMS: Warehouse Management System, warehouse management system.

[0082] WCS: Warehouse Control System, warehouse control system.

[0083] ETL: Extract-Transform-Load, is the process of extracting (extract), transforming (transform), and loading (load) a large amount of raw data into the target storage data warehouse.

[0084] Job: describes the entire data flow processing process from the source system to the target system, and defines how to perform data conversion processing in this process. A job consists of a data extraction component, one or more data processing components, and one or more target data components.

[0085] Data extraction component: also known as data source component, which obtains data from the source system of the job.

[0086] Data processing component: A component that processes and handles data during the data flow.

[0087] Target data component: that is, the data loading component, which indicates the target of the data and writes the data into the final management and storage system.

[0088] External system: refers to the system integrated or interconnected with the HSM-OS system, including PLC, DCS, ERP, etc. These external systems are autonomous and exchange data with the system through the interface subsystem HSM-IO.

[0089] The structure of the production information integration system based on visualization configuration provided in the embodiment of the present application is described below in conjunction with the accompanying drawings and specific embodiments. Figure 2 is a schematic diagram of the overall structure of a production information integration system based on visual configuration provided in an embodiment of the present application, such as Figure 2 As shown, the production information integration system based on visualization configuration can specifically include the following contents:

[0090] The component management module 201 is used to provide a visual configuration interface so that the user can select and configure multiple functional components by dragging and dropping through the visual configuration interface;

[0091] The data source management module 202 is used to manage the data source components, configure and verify the data source according to the input data source connection parameters, and generate a list of available data sources for the component management module to call;

[0092] The data processing module 203 is used to process the data obtained by the data source management module according to the operation logic configured by the component management module;

[0093] The target data management module 204 is used to receive and store the data processed by the data processing module. The target data management module includes an interface component, and the interface component is used to write the processed data into an external target system;

[0094] The job scheduling module 205 is used to schedule the jobs configured by the component management module according to preset scheduling rules, wherein the scheduling rules include timed scheduling, periodic scheduling and event-driven scheduling;

[0095] The monitoring and diagnosis module 206 is used to monitor the operating status of each module in real time, record the job execution log of each module, detect abnormal conditions in system operation, and generate diagnostic reports and troubleshooting suggestions.

[0096] In some embodiments, the component management module is used to:

[0097] Generate a visual configuration interface, wherein a component toolbar is provided in the visual configuration interface, and the component toolbar includes multiple functional components;

[0098] In response to the user's selection and dragging operation on a functional component in the component toolbar, the functional component is added to the design canvas.

[0099] Specifically, the component management module generates an intuitive visual configuration interface, which may include a component toolbar. The component toolbar is located on the left side of the interface and is used to display multiple predefined functional components, which cover common system functions, such as: data source components, data processing components, target data components, design canvas, property setting panel, etc.

[0100] Furthermore, the user interaction and operation process includes the following:

[0101] 1. Selection and dragging of components:

[0102] The user selects a functional component from the component toolbar, such as a "database connection component", and adds it to a specified position in the design canvas by dragging the mouse.

[0103] The system responds to user operations by rendering the corresponding component icon and name in real time in the design canvas, and displays the default configuration parameters of the component in the property setting panel.

[0104] 2. Component parameter configuration:

[0105] When the user clicks the component icon in the canvas, the system automatically activates the property setting panel, and the user can enter relevant parameters in the panel, such as the database connection address, user name and password, etc.

[0106] The system performs real-time verification of the parameters entered by the user, such as checking whether the format of the database address is correct, and provides feedback on the verification results in the form of a prompt box on the interface.

[0107] 3. Connections between components:

[0108] Users can establish a connection between two functional components in the design canvas by dragging and dropping. For example, when connecting the "Database Connection Component" to the "Field Mapping Component", the system will automatically generate a connection line to indicate the data flow.

[0109] Through the design of the above embodiment, the system provides upstream and downstream logic verification for each connection to ensure that the connection between components complies with the data processing logic. For example, if the data type does not match (such as text data connected to a numerical processing component), the system will prompt the user to adjust in real time.

[0110] Through the visual configuration interface, users can complete the configuration of system modules by dragging, configuring and connecting without programming experience, avoiding the complex code writing in traditional systems.

[0111] The system provides instant error prompts and logic verification functions during the user configuration process, avoiding subsequent job failures caused by incorrect configuration and improving the accuracy and efficiency of operations.

[0112] The visual interface is clear and intuitive, supporting the free combination and flexible expansion of multiple functional components, which greatly improves the efficiency of system customization while reducing the complexity of user operations.

[0113] The component management module of this application can effectively lower the technical threshold for users to customize the production information system, and significantly improve the simplicity and flexibility of configuration, providing a new implementation method for the visual operation of the production management system.

[0114] In combination with the component management module of the above-mentioned embodiment, the process of using the component management module in the present application to perform job configuration management is described in detail below. Figure 3 is a flowchart of job configuration management provided by an embodiment of the present application, such as Figure 3 As shown, the process of job configuration management may specifically include the following contents:

[0115] This embodiment utilizes the component management module to implement job configuration management for information system integration. Figure 3 The job configuration management process shown in the figure supports flexible configuration operations of jobs in offline and online states, and can specifically include the following steps:

[0116] 1. Loading and displaying the job list

[0117] After the system is started, all job configuration records in the current system are loaded first, a job list is generated, and it is displayed in a table in the interface. Each job record includes basic information such as job name, status (such as "to be run", "running" or "completed"), creation time, update time, etc. Users can quickly find the target job through filtering and sorting functions.

[0118] 2. Operation selection

[0119] The user selects the operation to be performed on the job through the interface, including the following three situations:

[0120] 1) Create a new job: Users can choose to create a new job flow.

[0121] 2) Edit job: Users can choose to modify an existing job process. When the selected job is not in operation, the system allows the user to enter the editing mode.

[0122] 3) View jobs: Users can view the running status and execution log of jobs, and can choose to start, pause, stop or resume running jobs.

[0123] 3. New job flow configuration

[0124] 3.1 Initial configuration of the workflow

[0125] When the user selects "New Job", the system enters the job flow configuration interface. First, to create a job flow configuration, the user needs to enter the basic information of the job, such as the job name, job description, and run trigger type (such as timing, cycle, or event-driven). Secondly, draw the job flow configuration canvas, and the system automatically generates a blank canvas for the user to design the job flow through visual drag and drop operations.

[0126] 3.2 Loading component list

[0127] The system loads a list of available components from the component management module, including: data source components (such as "database connection components"); data processing components (such as "field mapping components" and "data conversion components"); target data components (such as "target storage components" and "interface transmission components"). Users drag and drop the required components from the toolbar to the configuration canvas, and define the connection relationship between components according to requirements to build the data flow of the job.

[0128] 4. Detailed configuration of job steps

[0129] 4.1 Data Object Configuration

[0130] The user can select a component in the canvas, and the system will pop up a "property setting panel" to allow the user to set parameters for the component. For example, the data source component configures connection parameters (such as database address, user name and password). The data processing component configures conversion rules (such as field mapping rules, data verification conditions). The target data component configures the storage path or the interface address of the target system.

[0131] 4.2 Data Preview

[0132] The system provides a "preview mode" where users can preview the format and results of data flowing through the component after configuration is complete. For example, if a user sets a renaming rule for a specific field in the field mapping component, the preview mode will display the data structure before and after the mapping in real time to help users verify whether the configuration is correct.

[0133] 5. Check and save the validity of the job

[0134] Logical verification: The system verifies the logical coherence of the workflow. For example, it checks whether the connection relationship between components is correct, whether the data format matches, and whether there are components with unconfigured parameters. If errors are found, the system will highlight the parts that do not meet the rules in the canvas and display a detailed error description on the right side of the interface to guide users to make corrections.

[0135] Save configuration: After the user confirms that the job flow configuration is correct, he can choose to save the job. The system compiles the job configuration, generates the operation logic of the job task and stores it in the system database for the job scheduling module to call.

[0136] 6. Editing workflow

[0137] When the user chooses to edit an existing job process, the system loads the configuration content of the job and presents the original process structure and parameter configuration in the configuration canvas.

[0138] Users can modify parameters of existing components, add or delete components, and adjust the connection relationship between components. After editing is completed, the system will re-check the validity of the job and save it.

[0139] 7.Job execution and control

[0140] Users can choose to start a job in the job viewing interface. The system executes job tasks according to the configured scheduling rules through the job scheduling module, including timed triggering, periodic triggering or event-driven triggering.

[0141] The system monitors the execution process of the job in real time, and displays the current status, data flow and execution log of the job through the monitoring mode UI.

[0142] Users can pause, resume, or terminate a running job. For a paused job, the system retains the current running status and users can choose to resume the job later.

[0143] Through the method of the above-mentioned embodiment, the present application supports flexible configuration and management of jobs both offline and online. Combining the visual operation interface with the logic verification mechanism, the complexity of job configuration is greatly reduced, and the efficiency and accuracy of user operations are improved.

[0144] In some embodiments, the component management module is further configured to:

[0145] Setting parameters for the functional components added to the design canvas, wherein the parameters include connection information of the data source component, conversion rules of the data processing component, and storage path or interface information of the target data component;

[0146] In response to the user's selection and dragging operations, the connection relationship between the functional components is defined in the design canvas, and the connection relationship is used to represent the path of data flow;

[0147] According to the connection relationship between functional components, the data flow logic is verified based on the predetermined rules to prompt the user to adjust the connection that does not meet the predetermined rules;

[0148] The verified functional components and connection relationships are converted into the operation logic of the job task for the job scheduling module to call and execute.

[0149] Specifically, the user selects the added functional component through the design canvas interface, and the system automatically pops up the "Parameter Configuration Window" to guide the user to set the following parameters:

[0150] 1) Connection information of the data source component: including the type of data source, connection address, port number, authentication information (user name, password), etc. For example, the user can enter the IP address and account information of the MySQL database for the "database connection component".

[0151] 2) Conversion rules of data processing components: including field mapping rules, format conversion methods (such as date format or data type conversion), and data cleaning logic (such as deduplication or filtering).

[0152] 3) Storage path or interface information of the target data component: including the file path of the target data storage, the storage address of the local or cloud database, or the URL and authentication information of the external system interface. For example, users can configure the "target storage component" to write the processed data to a specified shared storage path or send it to an external system through an HTTP interface.

[0153] Furthermore, the user defines the connection relationship between components by dragging in the design canvas, indicating the path of data flow from one functional component to the next.

[0154] For example, the user drags the "Database Connection Component" and the "Field Mapping Component" in the canvas to establish a connection. The system generates a line representing the data flow path, automatically marking the starting point and end point of the data. The user can continue to connect the "Field Mapping Component" with the "Target Storage Component" to form a complete data flow link.

[0155] The system supports users to adjust the connection relationship, such as deleting unnecessary connections through the right-click menu, or inserting new functional components (such as data verification components) between two components.

[0156] Furthermore, the component management module verifies the data flow logic according to the connection relationship between the components to ensure the correctness and logic of the data flow path. In some examples, the specific verification process is as follows:

[0157] First, the system checks whether the data flow is logical according to the predetermined rules. For example, whether the input and output data types match (such as numerical data flow cannot be directly input into the text processing component). Whether all components are configured and whether there are unconnected isolated components.

[0158] Secondly, if a logical error is detected, the system highlights the erroneous component or connection in the design canvas and displays an error description on the right side of the interface, such as "The input field of the field mapping component does not match the output field of the data source component."

[0159] Finally, the system provides adjustment suggestions to prompt users to modify connections that do not meet the rules. For example, the system prompts users to adjust field mapping rules or check whether component parameter settings are missing.

[0160] Furthermore, after verification, the component management module converts the functional components and connection relationships in the design canvas into executable job task logic. In some examples, the specific implementation is as follows:

[0161] The system compiles component parameters and connection relationships into a job description file (such as JSON or XML format), which represents the complete job logic, including: the starting node of the data (such as the database connection component) and the ending node (such as the target storage component); intermediate processing logic (such as field mapping and data verification rules); data flow path and field mapping relationship.

[0162] After the job description file is generated, the system automatically stores it in the job configuration database for the job scheduling module to call and execute.

[0163] For example, in some examples, the user wants to design a job task to process the order data of the ERP system and write it to the sales data repository of the target system. The following are the specific implementation steps of the process:

[0164] 1) Add components: Users add "ERP data source component", "field mapping component" and "target storage component" to the design canvas by dragging and dropping.

[0165] 2) Configuration parameters: Configure the connection address, user name and password of the ERP database in the ERP data source component. Configure the mapping rules in the field mapping component, and configure the storage path or external interface information of the target repository in the target storage component.

[0166] 3) Define connection relationships: Users use the canvas to define the path of data from the ERP data source to the field mapping component and then to the target storage component.

[0167] 4) Verify logic: The system prompts the user that the input fields of the field mapping component match the output fields of the ERP data source successfully, and verifies that the path logic is correct.

[0168] 5) Generate job logic: The system stores the job logic in the job database for the scheduling module to call on demand.

[0169] Through the design of the above embodiment, users can intuitively and efficiently complete the entire process from functional component configuration to job logic generation. Compared with the traditional method that relies on manual coding, the visual configuration tool of this application greatly simplifies the complexity of data integration and logic definition, while improving the flexibility of the system and the user operation experience.

[0170] In the following, in conjunction with the configuration process of the functional components of the above-mentioned embodiment, the data source management process and the conversion and mapping component design process of the present application are described respectively. Figure 4 is a schematic diagram of the data source management process provided by the embodiment of the present application, such as Figure 4 As shown, the data source management process may specifically include the following contents:

[0171] The data source management module is used to add, configure and verify data sources to ensure that the data source is connected correctly and can be used for subsequent job tasks. Figure 4 The data source management process shown in the figure includes data source selection, creation, parameter configuration, connection test and data source list update. The following are the specific implementation steps of the process:

[0172] 1. Selection of data source components

[0173] The user selects the "data source component" from the component toolbar and drags it to the design canvas through the visual configuration interface in the component management module. The system responds to the user's operation and displays a list of available data source types, such as database data source, file data source, interface data source, real-time stream data source, etc. The user selects the target data source type from the list.

[0174] 2. Create a new data source

[0175] After the user selects the target data source type, the system enters the new data source configuration interface and guides the user to enter the basic information of the data source, such as data source name: the user specifies an easily recognizable name for the data source, such as "ERP order database". Data source description: an optional field used to describe the purpose and characteristics of the data source, such as "used to synchronize ERP system order data". The system automatically loads the corresponding data source configuration template based on the user's selection to simplify the configuration process.

[0176] 3.Configuration of data source parameters

[0177] The system pops up a parameter input window based on the data source type selected by the user, prompting the user to configure the connection parameters of the data source. For example, for a database data source, the following information needs to be entered: database type, server address, port number, user name, and password.

[0178] Furthermore, for a file data source or an interface data source, parameter configuration may include a file path, an API address, request header information, and the like.

[0179] The system performs preliminary verification on the format of the parameters entered by the user, for example, checking whether the format of the server address is correct; detecting whether the port number is within the valid range; and verifying whether the necessary fields (such as user name and password) are entered.

[0180] 4. Data source connection test

[0181] After the user completes the parameter configuration, click the "Connection Test" button, the system attempts to establish a connection with the target data source and returns the test results, for example:

[0182] If the connection is successful, the system will pop up a prompt box to indicate that the connection is successful, and automatically detect the metadata of the target data source (such as database table structure and field information). The metadata can be used as a reference for subsequent job task configurations, for example, users can choose to extract specific tables or fields from them.

[0183] If the connection fails, the system will pop up an error prompt box and display specific error information, such as "Authentication failed" or "Server address unreachable". Users can recheck and modify parameter configuration according to the prompts and test again.

[0184] 5. Update and save data source status

[0185] The system marks the data source that successfully passes the connection test as "available" and generates corresponding status information, including data source type, connection status, and metadata summary.

[0186] The status information of available data sources is added to the "Available Data Source List". This list is called by the component management module and is available for user selection in subsequent job task configuration.

[0187] Each data source is displayed in the data source list with structured information, including the data source name, type, connection status, and last updated time.

[0188] 6. End the operation

[0189] After the user confirms that the data source has been correctly configured, click the "Save" button, and the system will store the data source configuration in the database for subsequent component calls. The process ends.

[0190] The following is an explanation of the conversion and mapping component design process of the present application in conjunction with the accompanying drawings and embodiments. Figure 5 is a schematic diagram of the conversion and mapping component design process provided by the embodiment of the present application, such as Figure 5 As shown, the transformation and mapping component design process may specifically include the following contents:

[0191] The conversion and mapping components are used to implement data format conversion, field mapping, and data rule processing to meet the target system's requirements for data format and structure. Figure 5 The transformation and mapping component design process shown includes structural design, property configuration, preview testing, and logic verification of the transformation component.

[0192] The process can specifically include the following implementation steps:

[0193] 1. Design structure of transformation and mapping components

[0194] First, through the component management module, the user selects the "Transformation and Mapping Component" from the toolbar and drags it to the design canvas. The system automatically loads the default data structure of the component, and the user can customize the structure according to actual needs:

[0195] Input data structure: defines the data fields and data types received by the component. For example, the user sets the input data structure to "Order_ID (integer)", "Order_Date (date)" and "Customer_Name (string)".

[0196] Output data structure: defines the fields and types after data conversion and mapping. For example, the user defines the output structure as "Sales_ID (integer)", "Sale_Date (date)" and "Client_Name (string)".

[0197] The system provides a drag-and-drop operation to drag the input field to the corresponding output field, and connect them with arrows in the interface to indicate the mapping relationship between the fields.

[0198] Furthermore, the transformation and mapping components are displayed as icons in the design canvas, including input pins and output pins, which correspond to the input and output nodes of the data respectively. Among them, the input pin: receives input data from the data source component or other processing component. The output pin: passes the transformed and mapped data to the downstream component, such as the target data component or the cache component. Users can define the data flow path by connecting pins.

[0199] 2. Property panel configuration

[0200] When the user selects the conversion and mapping components, the property setting panel automatically pops up, and the user can make detailed configurations in the panel, for example, including the following configuration rules:

[0201] 1) Field mapping rules:

[0202] The user defines the mapping relationship between the input field and the output field, for example, mapping the "Order_ID" field to "Sales_ID". In addition, the system also supports one-to-one, many-to-one, and one-to-many mapping relationships.

[0203] 2) Data conversion rules:

[0204] Users can set conversion rules for data formats, such as converting the date format from "YYYY-MM-DD" to "MM / DD / YYYY". Set unit conversion rules for numeric fields, such as converting "order amount" from RMB to USD.

[0205] 3) Verification rules:

[0206] Users can set data validation conditions, such as checking whether a field is empty, or whether the "Order_Date" field is earlier than the current date.

[0207] Furthermore, after the user configuration is completed, the system allows data preview and testing at the component level. For example, the user can manually enter or load sample data from a data source, such as a CSV file containing order information. The system executes the component logic on the sample data according to the mapping and transformation rules configured by the user. The system displays the mapped and transformed data results in real time, and the user can verify whether the data meets expectations. If the preview results do not meet expectations, the user can adjust the mapping rules or transformation rules until the preview results are correct.

[0208] Furthermore, the system performs logical verification on the configured conversion and mapping components to ensure that the data flow complies with the rules. For example, check whether the mapping relationship is complete and whether there are unmapped input fields; verify whether the field type matches, for example, a string field cannot be mapped to a numeric field; and verify whether the rule logic is correct, for example, check whether an illegal date format is set.

[0209] After verification, the component management module converts the logic of the transformation and mapping components into an executable component configuration file (such as JSON or XML format) and stores it in the system for subsequent calls.

[0210] Through the above-mentioned method of this embodiment, the user can efficiently complete the design and configuration of data conversion and mapping rules, which greatly simplifies the traditional manual coding process while ensuring the accuracy and logic of data processing.

[0211] In some embodiments, the data source management module is used to:

[0212] Receive data source connection parameters input by the user, the data source connection parameters include data source type, address, authentication information and connection information;

[0213] Establish a connection for the data source according to the data source connection parameters, and verify the data source and data source connection;

[0214] Generate data source status information based on the verification result. If the verification succeeds, the data source is marked as available. If the verification fails, an error message is returned.

[0215] The data source status information corresponding to the successfully verified data sources is summarized, and the summarized data source status information is added to the available data source list, and the available data source list is provided to the component management module for calling.

[0216] Specifically, when a user needs to add or configure a data source, the system receives the data source connection parameters input by the user through the interactive interface. For example, these parameters may include:

[0217] Data source type: The user selects the type of data source, such as database, file, or API interface.

[0218] Address: The user enters the connection address of the data source, such as the database IP address, file path, or API URL.

[0219] Authentication information: Depending on the type of data source, the user enters information such as username, password or token.

[0220] Other connection information: including port number, database name, request header information, etc.

[0221] The system performs format verification based on the parameters entered by the user, for example, confirming whether the format of the IP address is legal; checking whether the port number is a valid value; and confirming whether the authentication information is complete.

[0222] Furthermore, the system attempts to establish a connection with the data source based on the parsed connection parameters. For example, for a database data source, the system uses the input address, port number, username, and password to establish a connection through the database driver; for a file data source, the system attempts to access the file through the specified file path; for an API data source, the system sends a test request to the specified API URL and receives a response.

[0223] Furthermore, the system verifies the established connection, which may include the following verifications:

[0224] 1) Data source status verification: Check whether the data source is in normal operation. For example, for a database, the system verifies whether the table structure or data in the database can be successfully accessed; for a file, the system verifies whether the file is readable; for an API, the system verifies whether the response conforms to the expected format.

[0225] 2) Connection parameter verification: Check whether the parameters entered by the user match the actual configuration of the data source, such as whether the database name is correct.

[0226] Furthermore, if the verification is successful, the system returns a prompt message of "Connection successful" and records the relevant metadata of the data source (such as database table structure, field information). If the verification fails, the system returns an error message, prompting the user to check the connection parameters, and displays the specific error cause in the interface (such as "user name or password error" or "API address invalid").

[0227] Furthermore, after the verification is completed, the system generates status information of the data source according to the result, for example, the following status information may be included: data source name, data source type, connection status, and metadata information.

[0228] Furthermore, the system summarizes the status information of all successfully verified data sources and generates a "list of available data sources". The list is stored in a structured manner, including the name, type, status and metadata summary of each data source.

[0229] Furthermore, the system stores the summarized list of available data sources in a database accessible to the component management module, and marks the latest connection status of each data source.

[0230] When the component management module needs to configure a job task, the user can select the target data source from the list of available data sources. For example, if the user selects "ERP Order Database" as the input data source, the system will automatically load the connection parameters and metadata information of the data source for use in job process design.

[0231] Through the method of the present embodiment described above, the data source management module can efficiently complete the configuration, verification and status management of the data source, ensure the correctness and availability of the data source, and provide reliable data input support for the design and execution of subsequent job tasks.

[0232] In some embodiments, the data processing module is used to:

[0233] According to the job logic configured by the component management module, select one or more available data sources from the data source management module and obtain the corresponding data;

[0234] Perform preprocessing operations on the acquired data, and perform format conversion and field mapping on the preprocessed data;

[0235] The converted data is verified based on predetermined rules, and the verified data is passed to the target data management module and the data processing status is marked so that the monitoring and diagnosis module can record and track it.

[0236] Specifically, the data processing module loads the list of available data sources from the data source management module according to the job logic configured by the component management module. The user selects the target data source through the interface. The system loads its related metadata (such as table name and field information) according to the target data source selected by the user, and provides visualization options for the user to further specify the data extraction conditions.

[0237] Furthermore, the data processing module performs preprocessing operations on the acquired raw data to ensure data quality and meet subsequent processing requirements. Specifically, the following steps may be included:

[0238] 1) Data cleaning: Remove invalid or empty records. For example, for orders with an empty "Customer_Name" field, the system removes them from the data set and deletes duplicate records.

[0239] 2) Data normalization: Format data fields and normalize the units of numerical fields, such as converting the "order amount" field from RMB to USD.

[0240] 3) Data completion: fill in default values ​​for missing fields according to pre-defined rules.

[0241] Furthermore, the user configures data verification rules through the component management module, and the data processing module verifies the data according to these rules. For example, check whether the field value is empty; verify whether the numerical field is within a reasonable range, such as "order amount" should be greater than 0; check whether the date field is logical, such as "order date" must not be later than the current date.

[0242] Furthermore, for records that pass the verification, they are marked as "valid data"; for records that fail the verification, they are marked as "invalid data" and an error message (such as "the order amount is negative") is generated. The verified data and error message will be stored in the log file for the monitoring and diagnosis module to record and track.

[0243] Furthermore, the data processing module transfers the verified data to the target data management module for storage or transmission. In addition, the data processing module marks the processing status of each record, for example, "preprocessing completed", "verification passed", "mapping completed". The marking information will be attached to the data record and recorded and tracked by the monitoring and diagnosis module.

[0244] Through the above-mentioned method of this embodiment, the data processing module can efficiently complete data acquisition, preprocessing, format conversion, field mapping and verification operations, ensure that the quality of processed data meets the requirements of the target system, and provide detailed tracking information for the monitoring and diagnosis module.

[0245] In some embodiments, the target data management module is further configured to:

[0246] Manage the target data component and store or transmit the data output by the data processing module to the specified external target system according to the parameter configuration of the target data component.

[0247] Specifically, the target data management module completes parameter initialization of storage and transmission operations according to the configuration of the target data component by the user in the component management module.

[0248] For example, users can set the local storage path in the target data component. If the target data needs to be transmitted to an external system, users need to configure the connection information of the external system, including the system address (such as the API URL), transmission protocol (such as HTTP, MQTT) and authentication information (such as user name and password). In addition, users can also set the transmission trigger conditions of data, such as real-time transmission or batch transmission;

[0249] Furthermore, when the target data component is configured for local storage, the target data management module stores the processed data in the specified path in the form of files according to the storage path and file format set by the user. The system supports the generation of storage files by date or batch to ensure that the data files are organized reasonably. For example, the system generates daily files according to the date to avoid files that are too large; the system automatically verifies whether the file path is valid before storing data. If the path does not exist, the system will automatically create the required directory.

[0250] Furthermore, when the target data component is configured to be transmitted to an external target system, the target data management module establishes a connection according to the connection information configured by the user. The system establishes a secure connection with the target system through the configured authentication information.

[0251] Furthermore, the target data management module packages the data according to a predetermined rule and sends it to the target system. For example, the system formats the processed data into the format required by the target system, and requests the API interface of the target system to send the packaged data, and receives the response information. If the transmission is successful, the system records a success log, and if the transmission fails, the system records an error log.

[0252] Through the above-mentioned method of this embodiment, the target data management module can efficiently complete the data storage and transmission operations, ensure that the data is safely and reliably stored locally or transmitted to the external system in real time, and provide accurate status information support for the monitoring and diagnosis module.

[0253] In some embodiments, the target data management module is specifically used to:

[0254] Read the parameter configuration of the target data component, including storage path, external target system address, data transmission protocol and authentication information;

[0255] Determine a data processing method based on parameter configuration, the processing method including storing the data in a local system or transmitting the data to an external target system through an interface component;

[0256] When storing in a local system, the data is stored in a predetermined storage path according to parameter configuration;

[0257] In case of transmission to an external target system, the data is sent to the external target system via the interface component according to the external target system address and the data transmission protocol.

[0258] Specifically, after receiving the task request, the target data management module first reads the parameter configuration set by the user in the target data component. For example, if the user chooses to store locally, the module reads the configured file path. If the user chooses data transmission, the module reads the connection address of the target system. Read the transmission protocol configuration, such as HTTP, HTTPS, FTP or other protocols; parse the authentication method of the external system, such as username, password or API key.

[0259] Furthermore, the system can also perform format verification on the parameter configuration read, for example, check whether the local storage path is valid; verify whether the target system address meets the protocol requirements; and check the integrity of the authentication information.

[0260] Furthermore, the target data management module determines the data processing method according to the parameter configuration.

[0261] In the case of local storage, when the storage path set by the user is valid and there is no need to connect to an external system, the module stores the data in the local file system;

[0262] In the case of external transmission, when the user configures the target system address and transmission protocol, the module sends the data to the external target system through the interface component.

[0263] Furthermore, the module supports dynamic switching of processing modes. For example, if the external system connection fails, the module will temporarily store the data in a local path and complete the transmission operation after the connection is restored.

[0264] Furthermore, when the local storage mode is determined, the target data management module generates a storage file according to the parameter configuration, for example, creates a target file according to the storage path and file format configured by the user, and writes the data into the file according to the field order set by the user.

[0265] Furthermore, the module supports batch storage to avoid a single file being too large; the module automatically checks the availability of the target path before storage. If the path does not exist, the system automatically creates the relevant directory.

[0266] Furthermore, when the external transmission mode is determined, the target data management module establishes a connection with the external system according to the parameter configuration. For example, the connection is initialized using the configured target system address; and a request header is generated according to the authentication information set by the user (such as user name and password, API key).

[0267] Furthermore, the module packages the data according to the format requirements of the external system, for example, converts it into JSON format, sends the data to the external target system through the configured transmission protocol (such as HTTPS), and receives the system response. For example, if the response status is 200 (success), the module records the transmission success log; if the response status is not 200 (failure), the module records the error information and triggers the retry mechanism.

[0268] Furthermore, if an error occurs during the external system connection or transmission, the module can take the following actions based on the error type:

[0269] Short-term failure: If the network fluctuation causes transmission timeout, the module triggers a retry operation;

[0270] Configuration error: If the authentication information is wrong, the module notifies the user to update the configuration and try to connect again;

[0271] System unreachable: The module temporarily stores the data locally and generates a warning log for the user to check.

[0272] Furthermore, the module adds a storage or transmission status to each data record, for example, stored: the data has been successfully stored locally; transmitted: the data has been successfully sent to the external target system; transmission failed: the data transmission failed and will be retried.

[0273] Finally, the status information is stored in log files or databases and provided for real-time tracking and analysis by the monitoring and diagnosis module.

[0274] Through the above-mentioned method of this embodiment, the target data management module can flexibly respond to various data storage and transmission requirements, ensure efficient management, reliable storage and secure transmission of data, and provide comprehensive status tracking support for the monitoring and diagnosis module.

[0275] In some embodiments, scheduling the job configured by the component management module according to a preset scheduling rule includes:

[0276] Load the preset scheduling rules according to the job logic configured by the user through the component management module;

[0277] Add jobs to the scheduling queue according to their priority, monitor the triggering conditions of the scheduling rules, and execute the scheduling logic of the scheduling rules when the scheduling rules are triggered;

[0278] Among them, timed scheduling is used to activate the corresponding job when the preset trigger time arrives, periodic scheduling is used to activate the job cyclically at a preset time interval, and event-driven scheduling is used to activate the job when a preset event is detected.

[0279] Specifically, users define the operation logic of the job through the component management module, including the input data source, processing logic and output target of the job. For example, the job name is: daily sales data synchronization; input data source: "order table" in the ERP database; processing logic: cleaning and formatting order data; output target: "sales table" in the CRM system.

[0280] The scheduling method and triggering conditions of the job tasks of the present application are explained below in conjunction with the drawings and embodiments. Figure 6 is a flowchart of the job scheduling method and triggering conditions provided in the embodiment of the present application, such as Figure 6 As shown, the job scheduling method and triggering conditions may specifically include the following:

[0281] Users configure scheduling rules for jobs in the component management module. Scheduling rules can include the following three types:

[0282] Scheduled scheduling: The user specifies the trigger time, such as "execute at 0:00 every day".

[0283] Periodic scheduling: The user specifies a time interval, such as "execute every 2 hours".

[0284] Event-driven scheduling: Users configure event trigger conditions, such as "trigger execution when new order data files arrive in the specified directory."

[0285] The job scheduling module receives the scheduling rules configured by the user and loads them into the scheduling rule list.

[0286] Furthermore, the job scheduling module assigns priorities to jobs based on the job logic and importance configured by the user. For example, high priority: jobs involving key business processes (such as real-time order processing); low priority: non-real-time tasks (such as nightly log archiving).

[0287] Furthermore, the module adds the loaded jobs to the scheduling queue in order of priority to ensure that high-priority jobs are executed first. The scheduling queue stores key information of the job, for example, including: job name, scheduling rule type (timing, periodic or event-driven), priority, current status (waiting for scheduling, executing or completed).

[0288] Furthermore, the job scheduling module monitors the triggering conditions of each scheduling rule in real time, which may include the following triggering conditions:

[0289] Scheduled scheduling: The module checks the system clock to see if the current time matches the job's trigger time. For example, if the user configures the trigger time to be 0:00 every day, the module will activate the job when the system time reaches "00:00:00".

[0290] Periodic scheduling: The module calculates the interval between the last job execution time and the current time. If the interval time set by the user (such as 2 hours) is reached, the job is activated.

[0291] Event-driven scheduling: The module monitors user-configured events, such as the arrival of a new file in a directory or the addition of a new record in a database table. When an event is detected, the corresponding job is immediately triggered.

[0292] Furthermore, when the scheduling rule is triggered, the module executes the job according to the configured scheduling logic. For example, the triggered job is taken out of the scheduling queue; the job logic is loaded and the job execution process is started; and the job status is marked as "executing".

[0293] Finally, the job scheduling module tracks the status in real time during job execution and generates a scheduling log.

[0294] Through the above-mentioned method of this embodiment, the job scheduling module can efficiently manage and execute job scheduling rules based on timing, cycle and event-driven, ensure the flexibility, accuracy and reliability of the job, and provide complete status tracking and log support for system operation.

[0295] In some embodiments, the system further comprises:

[0296] The data conversion and compatibility module is used to provide automatic adaptation of protocols and data formats between the data source management module and the target data management module, and supports compatible conversion of multiple standard protocols and custom data formats.

[0297] Specifically, the system has a built-in intelligent data conversion engine that can automatically identify and parse multiple data formats, for example, including but not limited to: standard formats (such as JSON, XML), custom formats (such as user-defined proprietary data structures or protocols).

[0298] The intelligent data conversion engine dynamically generates conversion rules based on the input data format and the target data format, eliminating the need for users to manually write data mapping logic.

[0299] The module supports a variety of standard protocols and transmission methods. Users can select the protocol type of the data source and target system through the interface, and the system automatically adapts the relevant connection logic.

[0300] Furthermore, the data conversion and compatibility module can automatically detect the data format according to the data source input and adjust the output format according to the requirements of the target system. For example, receive data in JSON format from the data source; detect that the target system requires XML format; and automatically convert the JSON format data into XML format that meets the requirements of the target system.

[0301] Furthermore, the module supports field mapping and automatic conversion of data types, for example, converting the "Order_Date" field (string type) in the data source to the date format (date type) of the target system; converting the numeric field from floating point type (float) to integer type (integer).

[0302] Furthermore, the module performs preprocessing on the received data, including data cleaning, structural normalization and other operations to ensure that the source data adapts to the requirements of the target system. The system dynamically generates data conversion rules based on the intelligent data conversion engine, allowing users to define complex data mapping logic through the interface without manual coding, thus lowering the threshold for use.

[0303] In some embodiments, the data conversion and compatibility module is specifically used to:

[0304] Use the intelligent data conversion engine to perform format analysis on the original data received by the data source management module, and automatically identify the format type and encoding standard of the original data;

[0305] Parse the transmission protocol of the original data and extract the protocol header information and data content;

[0306] Determine the data format and protocol type required by the external target system based on the parameter configuration of the target data management module;

[0307] According to the data format and protocol type, the format conversion operation is performed in the intelligent data conversion engine;

[0308] According to the protocol requirements of the external target system, the protocol content of the format-converted data is repackaged to generate a protocol packet that complies with the transmission rules of the external target system.

[0309] Specifically, after receiving the original data from the data source management module, the data conversion and compatibility module uses the intelligent data conversion engine to analyze the data format and encoding standard.

[0310] Furthermore, the intelligent data conversion engine parses the structured or unstructured content of the data source and automatically identifies the data format type, such as JSON, XML, CSV or binary data. The system extracts data features according to preset rules.

[0311] Furthermore, the module detects the character encoding of the original data and performs standardized encoding conversion according to the requirements of the target system.

[0312] Furthermore, the module parses the original data transmission protocol and extracts the protocol header information and data content. For example, protocol header information extraction: extracts the protocol type from the transmission data packet; parses the transmission metadata, such as content length, data type, and data identifier. Data content separation: the module separates the effective data load from the protocol packet.

[0313] Furthermore, according to the parameter configuration of the target data management module, the module dynamically determines the requirements of the external target system for data format and protocol. For example, the requirements may include:

[0314] 1) Data format requirements: field structure and format type required by the target system, such as tag names and nesting rules of XML fields; data precision and unit requirements.

[0315] 2) Protocol type requirements: Determine the transmission protocol used by the target system, such as HTTP or MQTT; extract the authentication method (such as Token authentication) and protocol header template of the target system.

[0316] Furthermore, the module uses the intelligent data conversion engine to perform format conversion operations according to the determined target data format and protocol type. For example, the module converts the format of the original data according to the field mapping rules, converts the date field from a string format to a timestamp format, etc.

[0317] Furthermore, according to the protocol requirements of the target system, the module repackages the protocol content of the format-converted data, for example, adds authentication information according to the authentication method of the target system, embeds the format-converted data into the protocol payload, and generates a complete data transmission packet.

[0318] Through the method of the present embodiment described above, the data conversion and compatibility module effectively solves the adaptation problem of data formats and protocols between heterogeneous systems, improves the compatibility and transmission efficiency of system integration, and reduces the risk of manual intervention and data errors.

[0319] The technical solution of this application has at least the following advantages:

[0320] 1. Users can use a simple and intuitive drag-and-drop method to perform field mapping and data flow design between source data and target data, avoiding in-depth understanding of the underlying programming language and complex integration logic, greatly reducing the difficulty of integration configuration.

[0321] 2. With the help of standardized and simplified drag-and-drop operation processes, data mapping and conversion work becomes more intuitive and efficient, which greatly shortens the development time and reduces the overall development cost.

[0322] 3. The system has built-in data verification and conversion functions, allowing users to directly set data verification rules, format conversion, and data cleaning logic during the configuration process, improving the flexibility of data quality management.

[0323] 4. Provide real-time feedback and debugging support. Users can instantly view the effects of integrated configuration and data flow, which helps to quickly discover and solve problems during the configuration process, significantly improving the accuracy and efficiency of configuration.

[0324] 5. Standardized component and interface design enhances the reusability and maintenance convenience of the integrated solution, which helps the long-term stable operation of the system and future upgrades and maintenance.

[0325] 6. Through the fully automatic deployment mechanism, the system modules can be quickly updated and configured, which greatly reduces the complexity of manual operations and the risk of configuration errors.

[0326] 7. The system provides a flexible modular plug-in mechanism, and users can quickly integrate new functional modules or replace existing modules according to actual needs to adapt to the ever-changing factory management needs and ensure that the system maintains efficient operation during continuous optimization.

[0327] The technical solution of this application also reduces the complexity of system development and integration, reduces deployment and maintenance costs, and helps enterprises obtain higher production management efficiency and quality control levels with lower investment. It also improves the intelligence level of manufacturing information system integration, promotes the modernization and upgrading of the industry, provides enterprises with more flexible and reliable management tools, and supports more sustainable production and operation models.

[0328] In real-world application scenarios, the selection criteria for the technical solution of this application are as follows:

[0329] 1. Complexity of requirements and degree of customization: If a high degree of customization and support for complex business processes is required, you may need to choose custom development or an integrated development environment.

[0330] 2. Rapid deployment and low cost: If you focus on rapid deployment and low cost, you can consider application tools and visual integration tools because they can achieve system integration and customized development in a shorter time.

[0331] 3. Technical team capabilities: When selecting a technical solution, the team’s technical capabilities and experience need to be considered to ensure that the selected solution can be effectively implemented and maintained.

[0332] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A production information integration system based on visual configuration, characterized in that: include: A component management module is used to provide a visual configuration interface so that users can select and configure multiple functional components by dragging and dropping through the visual configuration interface; The data source management module is used to manage data source components, configure and verify data sources according to input data source connection parameters, and generate a list of available data sources for the component management module to call; A data processing module, used to process the data acquired by the data source management module according to the operation logic configured by the component management module; A target data management module, used for receiving and storing the data processed by the data processing module, the target data management module includes an interface component, and the interface component is used for writing the processed data into an external target system; A job scheduling module, used to schedule the jobs configured by the component management module according to preset scheduling rules, wherein the scheduling rules include timed scheduling, periodic scheduling and event-driven scheduling; The monitoring and diagnosis module is used to monitor the operating status of each module in real time, record the job execution log of each module, detect abnormal conditions in system operation, and generate diagnostic reports and troubleshooting suggestions.

2. The system according to claim 1, characterized in that The component management module is used for: Generate a visual configuration interface, wherein the visual configuration interface is provided with a component toolbar, and the component toolbar includes a plurality of functional components; In response to the user's selection and dragging operation on the functional component in the component toolbar, the functional component is added to the design canvas.

3. The system according to claim 2, characterized in that The component management module is also used for: Setting parameters for the functional components added to the design canvas, wherein the parameters include connection information of the data source component, conversion rules of the data processing component, and storage path or interface information of the target data component; In response to the user's selection and dragging operation, a connection relationship between the functional components is defined in the design canvas, wherein the connection relationship is used to characterize a path for data flow; According to the connection relationship between the functional components, the data flow logic is verified based on the predetermined rules to prompt the user to adjust the connection that does not meet the predetermined rules; The verified functional components and connection relationships are converted into the operation logic of the job task for the job scheduling module to call and execute.

4. The system according to claim 1, characterized in that The data source management module is used for: Receive data source connection parameters input by a user, wherein the data source connection parameters include data source type, address, authentication information, and connection information; Establishing a connection for a data source according to the data source connection parameters, and verifying the data source and the data source connection; Generate data source status information based on the verification result. If the verification succeeds, the data source is marked as available. If the verification fails, an error message is returned. The data source status information corresponding to the successfully verified data sources is summarized, and the summarized data source status information is added to the available data source list, and the available data source list is provided to the component management module for calling.

5. The system according to claim 1, characterized in that The data processing module is used for: According to the job logic configured by the component management module, one or more available data sources are selected from the data source management module, and corresponding data are obtained; Perform preprocessing operations on the acquired data, and perform format conversion and field mapping on the preprocessed data; The converted data is verified based on a predetermined rule, and the verified data is transferred to the target data management module and the data processing status is marked so that the monitoring and diagnosis module can record and track it.

6. The system according to claim 1, characterized in that The target data management module is also used for: Manage the target data component, and store or transmit the data output by the data processing module to a specified external target system according to the parameter configuration of the target data component.

7. The system according to claim 6, characterized in that The target data management module is specifically used for: Reading parameter configuration of the target data component, the parameter configuration including storage path, external target system address, data transmission protocol and authentication information; Determine a data processing method based on the parameter configuration, the processing method including storing the data in a local system or transmitting the data to an external target system through an interface component; In the case of storing in a local system, storing the data in a predetermined storage path according to the parameter configuration; In case of transmission to an external target system, the data is sent to the external target system via the interface component according to the external target system address and the data transmission protocol.

8. The system according to claim 1, characterized in that The scheduling of the job configured by the component management module according to the preset scheduling rule includes: Loading preset scheduling rules according to the job logic configured by the user through the component management module; Adding jobs to a scheduling queue according to their priorities, and monitoring the triggering conditions of the scheduling rules, and executing the scheduling logic of the scheduling rules when the scheduling rules are triggered; The timed scheduling is used to activate the corresponding job when the preset trigger time arrives, the periodic scheduling is used to activate the job cyclically at a preset time interval, and the event-driven scheduling is used to activate the job when a preset event is detected.

9. The system according to claim 1, characterized in that The system further comprises: The data conversion and compatibility module is used to provide automatic adaptation function of protocols and data formats between the data source management module and the target data management module, and supports compatible conversion of multiple standard protocols and custom data formats.

10. The system according to claim 9, characterized in that The data conversion and compatibility module is specifically used for: Utilize an intelligent data conversion engine to perform format analysis on the original data received by the data source management module, and automatically identify the format type and encoding standard of the original data; Parsing the transmission protocol of the original data to extract protocol header information and data content; Determine the data format and protocol type required by the external target system according to the parameter configuration of the target data management module; According to the data format and protocol type, performing a format conversion operation in the intelligent data conversion engine; According to the protocol requirements of the external target system, the protocol content of the format-converted data is repackaged to generate a protocol packet that complies with the transmission rules of the external target system.

Images