A data interaction method and system based on distributed components and cloud platform
Through the data interaction method based on distributed components and cloud platforms, the problems of traditional architecture complexity and high resource consumption are solved, and flexible and maintainable data interaction and real-time processing are achieved to adapt to diversified data sources and industrial-grade needs.
Patent Information
- Application Number
- CN202111335625.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-11-11
AI Technical Summary
Traditional data interaction architectures are complex and difficult to manage, and cannot effectively process large-scale and real-time data. Applications and tools are required to be re-written for each new technology, resulting in high resource consumption and unsuitable for diversified data sources and target endpoints.
Adopt data interaction methods based on distributed components and cloud platforms, component development and binding is carried out through component library and management cloud platform, and the pipeline is configured using a visual interface to support seamless connection between diversified data sources and target endpoints, realizing automatic scaling and Real-time data processing.
It realizes the flexibility and maintainability of data interaction, reduces resource consumption, supports industrial-grade data backtracking and tracking, adapts to different needs, and realizes edge deployment in the Internet of Things.
Smart Images

Figure CN114115838B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data interaction technology, and in particular to a data interaction method and system based on distributed components and a cloud platform, which are used to process data generated by various systems and data interaction between systems. Background Art
[0002] With the development of computer technology, on the one hand, heterogeneous systems, distributed systems and other non-monolithic systems have emerged, and the complexity of the system is increasing; on the other hand, the amount of data generated by the system is increasing, and the complexity of the data interaction logic between various systems is constantly increasing. At present, the data generated by the system needs to be extracted and processed by the system itself, and the coupling between data processing and actual business is serious. The data flow and interaction between systems requires the data provider to provide corresponding data interfaces for the data processor / receiver to read and write data. When a system interacts with external systems frequently, customized end-to-end interfaces will cause the system to be bloated and the development complexity to increase. As a result, a variety of ETL tools have been produced. ETL is the abbreviation of Extract-Transform-Load in English, which is used to describe the process of extracting (extract), transforming (transform), and loading (load) data from the source end to the destination end. ETL tools or related scripts can run independently of the business system, providing limited data interaction capabilities without affecting the architecture and functions of the business system. However, even with the emergence of various ELF tools, the traditional data interaction architecture is still very difficult to manage and very complex, and has the following disadvantages:
[0003] 1. Processing between databases, files and data warehouses is performed in batches.
[0004] 2. Currently, most companies need to analyze and operate real-time data. However, traditional tools are not suitable for analyzing logs, sensor data, measurement data, etc.
[0005] 3. Very large domain data models require a global structure.
[0006] 4. Traditional ETL processing is very slow, time-consuming, and requires a lot of resources.
[0007] 5. Traditional architecture focuses only on existing technologies. Therefore, every time new technologies are introduced, applications and tools must be rewritten.
[0008] Therefore, there is an urgent need for a technology that can solve the above problems and process the data generated by each system, as well as the data interaction between systems. Summary of the invention
[0009] The purpose of the present invention is to provide a data interaction method based on distributed components and a cloud platform, which is used to process data generated by various systems and data interaction between systems.
[0010] The present invention is implemented by the following technical solution: A data interaction method based on distributed components and a cloud platform, comprising the following steps:
[0011] Step S1. Determine whether the component library needs to be developed based on the existing components in the component library. If yes, proceed to step S2; if no, proceed to step S3;
[0012] Step S2. The developer develops the component according to the component template in the component library, packages the component and generates a version number after development. The developer uploads the packaged component package through the management cloud platform and submits the code repository address at the same time;
[0013] Step S3. The platform auditor debugs the uploaded component package and code repository address, and then audits the components in the component library to determine whether the audit is passed. If yes, create the release version information of the code repository through the version information of the API provided by the code repository, and publish the component information and file information records in the management cloud platform. If no, return to step S2;
[0014] Step S4. In the management cloud platform, select an API of a certain version information and multiple components to bind and generate a business suite. When binding, the business suite is processed and released in the background of the management cloud platform according to the business suite assembly logic;
[0015] Step S5. When implementing the business suite, the business suite released in step S4 is combined with the operating platform framework to generate a software package, the software package is deployed on the hardware server to run and the operating platform is started;
[0016] Step S6. After the operation platform is successfully started, pipelines are drawn and created by dragging and dropping through the visual interface, and instance nodes of the pipelines and multiple components in the pipelines are configured;
[0017] Step S7. After the pipeline configuration is completed, the pipeline is checked to determine whether there is an abnormal configuration. If yes, the pipeline cannot be created and a configuration abnormality is prompted. If no, the pipeline is created and the pipeline data is persisted to the file system.
[0018] Step S8. Query the created pipeline in the pipeline list and operate the pipeline;
[0019] Step S9. After the pipeline is started, the data is extracted, processed and written according to the pipeline configuration, the data interaction is completed, the log is recorded and the warning message is sent.
[0020] In this technical solution, modern data processing usually includes the processing of real-time data, and organizations also need real-time insights into the processing process. The system needs to perform ETL on the data stream, cannot use batch processing, and should be able to automatically scale to handle higher data traffic. Some single-server databases have been replaced by distributed data platforms (such as Cassandra, MongoDB, Elasticsearch, SAAS applications, etc.), messaging mechanisms (Kafka, ActiveMQ, etc.) and several other types of endpoints. So the system should be able to add additional data sources or destinations in a manageable way. This technical solution avoids duplicate data processing caused by the "write-now-use-now" architecture. This technical solution changes the way data capture technology is used, from requiring traditional ETL to be integrated with it to supporting traditional operations. This technical solution takes into account the diversification of data sources and the maintainability of new requirements, so that the source and target endpoints should be decoupled from the business logic, and use the data mapping layer to seamlessly connect new sources and endpoints without affecting the data conversion process.
[0021] In order to better implement the present invention, further, the form of the component template in step S2 includes a template function form of a code editor, an integrated development environment form and a package manager form, and different methods are used to develop components according to different forms of the component template;
[0022] When using the template function form of the code editor, the integrated development environment form, and the package manager form, you need to extract the configuration and import the extracted configuration on each development machine to obtain the component template code;
[0023] When using the plug-in package manager form of the package manager, install the corresponding package manager on the development machine and obtain the component template code through the specified command;
[0024] The component template code includes basic dependency information and API version information.
[0025] In order to better implement the present invention, further, step S3 includes:
[0026] Record the components that have passed the review. The components that have passed the review can be downloaded and used and can be combined with the data interaction engine in any way.
[0027] In order to better implement the present invention, further, the business suite assembly logic in step S4 includes:
[0028] Download and decompress the compressed packages of the data interaction engine and components respectively;
[0029] Combine the two folders into the running platform folder;
[0030] Compress the folder and distribute it into different types of compressed packages for use by various systems;
[0031] Decompress it during deployment and execute the script in the running platform folder to start running.
[0032] In order to better implement the present invention, further, the operation mode of running the platform framework in step S5 includes:
[0033] The runtime framework provides a default runtime module;
[0034] Provide customized functions according to the operating platform module and be replaced or developed as needed. The customized functions include user management function, permission management function and UI function for user operation;
[0035] The combined software package is considered as an executable platform, and the software package includes a file directory structure;
[0036] Data interaction is performed within the operational platform according to the data interaction engine rules.
[0037] In order to better implement the present invention, further, the data interaction engine rules include:
[0038] The configuration module determines whether the configuration is normal. If so, the component loading module scans the component path. If not, the loading ends.
[0039] When the configuration is normal, the loader is generated to record the component information, and the component is loaded to determine whether the component is loaded successfully. If yes, the engine is started and the loading is ended. If not, the loading is ended.
[0040] In order to better implement the present invention, further, step S6 includes:
[0041] Configure the pipeline, including the scheduled start and stop configuration of the pipeline, the exception handling configuration of the pipeline, the configuration of whether the pipeline is clustered, the configuration of the pipeline cluster node address, the configuration of the pipeline running interval, the configuration of the pipeline startup parameters, and the configuration of the pipeline email notification;
[0042] Instance nodes of multiple components in the pipeline include data source nodes, data processing nodes, and target nodes;
[0043] Configure instance nodes independently.
[0044] The data source in the pipeline is configured for the data source node. The data source includes the file system, data query API provided by other systems, or database system. The data source node has the feature of reading data.
[0045] Configure processing logic for data processing nodes, and process data processing nodes through remote APIs, scripts, or logic built into components of data processing nodes;
[0046] The final operation of configuring data on the target node includes writing data from the file system or other systems into the API or database system. The target node points to the data source configured by the data source node, and the target node has the feature of writing data.
[0047] In order to better implement the present invention, further, the method of operating the pipeline in step S8 includes:
[0048] Start a pipeline, edit a pipeline, delete a pipeline, and export a pipeline;
[0049] Click Start Pipeline to run according to the pipeline configuration;
[0050] Click Edit Pipeline to jump to the pipeline drawing interface and modify the pipeline configuration and node configuration;
[0051] Click Delete Pipeline to delete it from the file system, database, or other persistent destination.
[0052] Click Export Pipeline to generate a text file for the pipeline configuration.
[0053] In order to better implement the present invention, the present invention further provides a data interaction system based on distributed components and a cloud platform, the system comprising a data interaction engine, a component library, a management cloud platform and an operation platform, wherein:
[0054] The data interaction engine is used to check the pipeline and determine whether there is an abnormal configuration. If yes, the pipeline cannot be created and a configuration abnormality is prompted. If no, the pipeline is created and the pipeline data is persisted to the file system. The data interaction engine includes a set of API specifications for defining the properties and functions of components, pipelines, fragments, and templates. The data interaction engine includes a set of visualization interfaces for displaying components and drawing pipelines, fragments, and templates.
[0055] The component library is used to provide component development templates and quickly develop new components according to the development shift. The component library is highly decoupled from the data interaction engine and operating platform for independent development and management.
[0056] The management cloud platform is used in the development phase. Developers upload packaged component packages through the management cloud platform and submit the code repository address. It is used to manage various versions and component libraries of the data interaction engine, and bind various versions of the engine and multiple components to form different business suites.
[0057] The operating platform is used in the implementation phase to deploy the software package on the hardware server; the operating platform includes the business suite generated on the management cloud platform, which is used for customized development of business-related functions; the operating platform includes a set of default business functions, including account management functions, permission management functions and pipeline management functions; the operating platform is used to draw and create pipelines by dragging and dropping through a visual interface, and to configure instance nodes of pipelines and multiple components in the pipeline.
[0058] In order to better implement the present invention, further, the data interaction engine includes a log module, an early warning module, an event module, a credential module, a cache module, a verification module, a trace module and a debugging module, wherein:
[0059] The log module is used to define a set of log API specifications, to record the log information of components and pipelines at runtime, to provide default implementations of log operations, to customize the recording logic of development logs according to API specifications, and to access external log systems.
[0060] Configuration module, which is used to define engine configuration items and load configuration when the engine starts. The configuration module supports local file configuration and remote configuration.
[0061] The warning module is used to provide a set of API specifications for message warnings; it is used to issue warning notifications when an exception occurs during pipeline operation;
[0062] The event module is used to provide a set of API specifications for event listeners. According to the API specifications, the event listening logic of components or pipelines can be customized. At the same time, the event module engine has some commonly used event listeners built in.
[0063] The credential module is used to manage the credential information generated during the pipeline operation and provide corresponding credentials for the nodes in the pipeline to verify when the pipeline is running;
[0064] The cache module is used to provide a set of API specifications for cache functions; to provide default implementations; to cache pipeline configuration information and load pipeline configuration information into the cache when the engine starts;
[0065] Verification module, used to verify whether the configuration of pipelines, fragments and templates is correct;
[0066] The Trace module is used to record and store the data generated at each node during the pipeline operation and to configure multiple storage methods;
[0067] The debugging module has a built-in web server for independent operation; it is used to debug components, fragments, and template functions during the development phase.
[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0069] (1) The present invention uses a component library to decouple the functions of the data interaction engine itself and the data interaction capabilities that can be provided. The components in the component library are the basis for realizing a pipeline, but the development of the components does not depend on the data interaction engine. They can be independently developed and deployed. The data interaction engine can dynamically load and unload components by scanning the program packages in a specific directory. This makes the components have considerable scalability.
[0070] (2) The present invention provides a simple and intuitive configuration of data processing tasks. Through a visual interface, the data processing flow is configured. Multiple different components can be flexibly combined in a drag-and-drop manner to form a pipeline configuration. The configuration can be modified or updated through fragment upgrades.
[0071] (3) The present invention supports multiple different versions of the same component running simultaneously. When the engine is started, a loader is generated for each component. The loader only loads its own component, and different loaders do not affect each other.
[0072] (4) The present invention uses a visual interface to publish a business suite for implementation. The existing engine version and component library can be viewed on the management center cloud platform. The engine and components can be freely combined to form a complete business suite. At the same time, a basic visual interface is provided for specific business suites for users to use.
[0073] (5) The data interaction logic of the present invention is based on the workflow concept. By configuring a series of components, data is processed from the data source through a series of orderly processing logics and finally flows to the target node.
[0074] (6) The present invention is developed and deployed independently of the business system. When providing powerful data interaction capabilities, the business system does not need to change its architecture or provide additional interfaces.
[0075] (7) The present invention has the characteristics of being distributed and highly available, and can maintain high efficiency and low resource consumption when processing large amounts of data.
[0076] (8) The present invention aims at different demands of the industrial manufacturing industry and generates industry kits in a packaged form through configuration to meet different demands.
[0077] (9) The present invention supports industrial-level point-to-point detailed data backtracking and tracking.
[0078] (10) The present invention supports edge deployment, and obtains required data from industrial equipment in advance in a manner that minimizes resource usage, so as to realize device data interaction under the Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] The present invention is further described in conjunction with the following drawings and embodiments, and all the concepts and innovations of the present invention should be regarded as the disclosed contents and the protection scope of the present invention.
[0080] Figure 1 This is a flow chart of a data interaction method based on distributed components and a cloud platform provided by the present invention.
[0081] Figure 2 This is a schematic diagram of the structure of the functional modules included in the data interaction engine provided by the present invention.
[0082] Figure 3 This is a structural diagram of the pipeline drawing interface built into the data interaction engine provided by the present invention.
[0083] Figure 4 This is a structural diagram of the data interaction engine startup process provided by the present invention.
[0084] Figure 5 This is a structural diagram of the component development and application process provided by the present invention.
[0085] Figure 6 This is a structural schematic diagram of the assembly and use of the service suite provided by the present invention.
[0086] Figure 7 This is a schematic diagram of the structure of the operating platform file directory provided by the present invention.
[0087] Figure 8 This is a structural schematic diagram of a data interaction system based on distributed components and a cloud platform provided by the present invention. DETAILED DESCRIPTION
[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and therefore should not be regarded as limiting the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without making creative work are within the scope of protection of the present invention.
[0089] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0090] Embodiment 1:
[0091] A data interaction method based on distributed components and a cloud platform in this embodiment, such as Figure 1-6 As shown, the data interaction method based on the workflow concept and the loading method of multiple versions of the same component of the present invention highly decouples the components and the data interaction hub itself, and components can be independently developed according to the provided API and templates; at the same time, it supports the visual installation of components, supports the architectural design of the data interaction engine and the data interaction hub, and provides a process for component development management.
[0092] Embodiment 2:
[0093] This embodiment is further optimized on the basis of Embodiment 1. This embodiment uses the component template of the component library to develop components. When the component library already has components and does not need new components, it is not necessary to develop new components. Component templates can be implemented in various forms. They can be implemented by the template function of the code editor, can be configured to implement the integrated development environment, or can be implemented using a package manager such as Maven's prototype plug-in. Based on different implementation methods, the use of templates is also different. When using the template function of the code editor or the integrated development environment, it is necessary to extract the configuration and import the configuration on each development machine to obtain the component template code; when using the package manager plug-in implementation, it is necessary to install the corresponding package manager on the development machine and obtain the template code through the specified command. The component template code contains basic dependencies and engine API version information, and also provides sample code and instructions for various components to help developers become familiar with the component development mode as soon as possible.
[0094] The package manager in this embodiment refers to the Maven package manager, and the specified command is such as mvnarchetype:generate-DarchetypeGroupId=cec.jiutianDarchetypeArtifactId=data-integration-process-component-archetype-DarchetypeVersion= <version>The above commands need to be executed through command line tools such as bash, powershell, etc.
[0095] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0096] Embodiment 3:
[0097] This embodiment is further optimized on the basis of embodiment 1. In this embodiment, after the developer develops the component, he packages it by himself and generates a version number according to the version naming specification. The packaged component package is in the format of a compressed package, which can be directly combined with the engine after decompression. The developer uploads the packaged component package through the management cloud platform and submits the code warehouse address for subsequent code review. The platform reviewer will debug and review the uploaded component package and code to ensure that the component is developed in accordance with the development specifications. At the same time, use the platform's debugging environment to debug the components in the component package to ensure normal functions. If the review fails, it is necessary to return to regenerate the component package, improve or redevelop and improve the component according to the review opinions. If the review passes, the release information of the code warehouse will be created through the API provided by the code warehouse, and the management cloud platform will also record the released component information and related file resources. The component records after the review can be downloaded and used, and can be combined with the data interaction engine at will.
[0098] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0099] Embodiment 4:
[0100] This embodiment is further optimized on the basis of embodiment 1. In this embodiment, a certain version of the engine and multiple components are selected by the management cloud platform for binding to form a business suite. When binding, the background processing logic of the management cloud platform is as follows: Figure 6 The assembly and use of the business suite is shown in the figure, which is released as a software package and can be used during the implementation phase.
[0101] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0102] Embodiment 5:
[0103] This embodiment is further optimized on the basis of embodiment 1. When implementing this embodiment, it is necessary to combine the previously released business suite and the operating platform framework. The operating platform framework generally provides a default minimum operating platform module. This module contains some customized functions, such as user management, authority management, etc.; it contains a part of UI for users to operate the customized functions mentioned above. At the same time, according to the implementation needs, this module can be replaced or other functions can be developed on its basis. The combined software package is regarded as an executable platform, and the file directory structure is as follows: Figure 7 The file directory structure of the operating platform is shown below. The software package can be deployed on a hardware server, usually a Linux or Windows operating system. The operating platform startup process is as follows: Figure 4 The data interaction engine startup process is shown in the figure.
[0104] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0105] Embodiment 6:
[0106] This embodiment is further optimized on the basis of Embodiment 5. In this embodiment, the configuration module is used to define the configuration items of the engine and the loading of the configuration when the engine is started. It supports both local file configuration and remote configuration. Therefore, the configuration module starts the engine configuration to determine whether the configuration is normal.
[0107] The other parts of this embodiment are the same as those of Embodiment 5, and thus will not be described in detail.
[0108] Embodiment 7:
[0109] This embodiment is further optimized on the basis of embodiment 1. After the data engine successfully starts the program, this embodiment can draw and create a pipeline by dragging and dropping through a visual interface. The pipeline is composed of instance nodes of multiple components. Each pipeline contains at least one data source node and one target node, and can contain multiple data source nodes, multiple data processing nodes and multiple target nodes. Each node needs to be configured independently. The data source node needs to configure the data source in this pipeline, which can be a file system, a data query API provided by other systems, or a database system, etc. Such nodes have the feature of "reading" data. The data processing node needs to configure the processing logic, which can be processed through a remote API, a script, or the logic built into the component of the node. The target node needs to configure the final operation of the data, which can be a file system, a data write API of other systems, or a database system, etc. This node can point to the data source configured by the data source node, and such nodes have the feature of "writing" data.
[0110] For example, if the data source node uses the MySQL data reading component, you need to configure the MySQL database connection information, such as the database address, database name, database connection user and password, database table name, field name, etc.; you need to configure the reading behavior, such as the reading interval, the amount of data to be read, the number of batches, etc.
[0111] For example, the data processing node uses a character transcoding component to convert characters into a specific encoding and push them to the next node. The incoming data format needs to be configured, such as json format, xml format, the field name obtained, the original encoding of the data, etc.; the outgoing data format needs to be configured, such as json format, xml format, text format, the encoding to which the data needs to be converted, etc.
[0112] For example, if the target node uses the MySQL data writing component, you need to configure the MySQL database connection information, such as the database address, database name, database connection user and password, database table name, field name, etc.; you need to configure the writing behavior, such as the amount of data to be written, whether to batch process, and the number of batches.
[0113] The configuration of the above three components can form a simple MySQL data encoding conversion pipeline. This example is only one of many pipeline drawing situations. The data source node can be replaced with other data sources, the data processing nodes can be increased or decreased according to business needs, and the target node can be replaced with other write targets.
[0114] In addition to the configuration of each node, the pipeline itself also needs to be configured, such as the scheduled start and stop of the pipeline, the processing type of pipeline exceptions, whether the pipeline is clustered, the cluster node address, the pipeline operation interval, the pipeline startup parameters, the pipeline email notification, etc.
[0115] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0116] Embodiment 8:
[0117] This embodiment is further optimized on the basis of embodiment 1. After the pipeline configuration is completed, the verification module in the data interaction engine will verify the pipeline. If there is an abnormal configuration, the pipeline cannot be created, and the configuration abnormality is prompted. If the verification is passed, the creation is completed, and the pipeline data will be persisted to the file system. The persistence method can be selected or replaced according to the actual situation, such as database system, cache, message queue, etc. In the pipeline list, you can query the created pipeline, and you can perform the following operations on the pipeline: start the pipeline, edit the pipeline, delete the pipeline, and export the pipeline. Click to start the pipeline, and it will run according to the running rules of the pipeline configuration. Click to edit the pipeline, then jump to the pipeline drawing interface to modify the pipeline configuration and node configuration. Click to delete the pipeline to delete it from the file system, database or other persistent targets. Export the pipeline to generate a json format file for the pipeline configuration. After the pipeline is started, data will be extracted, processed and written according to the configuration. If the log module or the early warning module is connected, log records or early warning messages will be sent. The file format in this embodiment can be any text format, such as .json, .xml, .txt, .doc, etc., or a file without a suffix. The data format is usually a structured string, such as a json string, an xml string, etc.
[0118] The other parts of this embodiment are the same as those of Embodiment 1, and thus will not be described in detail.
[0119] Embodiment 9:
[0120] This embodiment provides a data interaction system based on distributed components and cloud platforms. The data interaction central system of this embodiment is divided into four parts: data interaction engine, component library, management cloud platform, and operation platform. Figure 8 As shown, the overall architecture of the data interaction engine in this embodiment is as follows Figure 2 As shown in the figure, the functional modules included in the data interaction engine are Figure 2 As shown in the figure, it includes a set of API specifications to define the properties and functions of components, pipelines, fragments and templates. It also includes a set of UI interfaces that can display components and draw pipelines, fragments and templates. In this embodiment, the pipeline drawing UI built into the data interaction engine is as follows: Figure 3 shown.
[0121] The component library provides component development templates, which can be used to quickly develop new components. At the same time, the component library is highly decoupled from the data interaction engine and the operating platform, and can be developed and managed independently. The underlying drivers of the components can be provided by various manufacturers. Component development can use existing drivers and carry out additional development on this basis. Different component programs should be written for different versions of drivers.
[0122] The management center cloud platform is used in the development stage to manage various versions and component libraries of the data interaction engine. Various versions of the engine and multiple components can be bound to form different business suites and released for implementation.
[0123] The operation platform is used in the implementation phase, including the business suite generated in the management cloud platform, and can be customized to develop business-related functions. It contains a set of default business functions, such as account management, permission management, pipeline management, etc., and a set of default UIs for users to operate the above functions. This part can also be replaced or redeveloped according to business needs. At the same time, the platform does not care about the specific implementation of the data interaction engine, and only calls the API provided by the engine. The other parts of this embodiment are the same as those of Example 9, so they will not be repeated.
[0124] Embodiment 10:
[0125] This embodiment is further optimized on the basis of embodiment 9, and the data interaction engine also includes other functional modules. The log module defines a set of log API specifications to record some log information of components and pipelines at runtime and provide a default implementation of log operations. The log recording logic can also be customized according to the API specifications, or an external log system can be connected. The configuration module is used to define the configuration items of the engine and the loading of the configuration when the engine starts. It supports two forms: local file configuration and remote configuration. The early warning module provides a set of API specifications for message early warnings, and early warning notifications can be made when an exception occurs during the operation of the pipeline. The event module provides a set of API specifications for event listeners, based on which the event listening logic of components or pipelines can be customized. At the same time, the engine has built-in some commonly used event listeners, such as pipeline start listeners and pipeline stop listeners. The credential module is responsible for managing various credential information generated during the operation of the pipeline, and providing corresponding credentials for nodes in the pipeline to verify when the pipeline is running. The cache module provides a set of API specifications for cache functions and provides a default implementation. It is responsible for caching pipeline configuration information and loading pipeline configuration information into the cache when the engine starts. The verification module is responsible for verifying whether the configuration of pipelines, fragments and templates is correct. The Trace module is responsible for recording and storing the data generated at each node during the operation of the pipeline, and can be configured with a variety of storage methods, such as file systems and database systems. The data interaction engine also provides a debugging module, which has a built-in Web server and can run independently to debug components, fragments, and template functions during the development phase.
[0126] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.< / version>
Claims
1. A data interaction method based on distributed components and cloud platform, It is characterized in that The following steps are involved: Step S1. Determine whether the component library needs to be developed based on the existing components in the component library. If yes, proceed to step S2; if no, proceed to step S3; Step S2. The developer develops the component according to the component template in the component library, packages the component and generates a version number after development. The developer uploads the packaged component package through the management cloud platform and submits the code repository address at the same time; Step S3. The platform auditor debugs the uploaded component package and code repository address, and then audits the components in the component library to determine whether the audit is passed. If yes, create the release version information of the code repository through the version information of the API provided by the code repository, and publish the component information and file information records in the management cloud platform. If no, return to step S2; Step S4. In the management cloud platform, select an API of a certain version information and multiple components to bind and generate a business suite. When binding, the business suite is processed and released in the background of the management cloud platform according to the business suite assembly logic; Step S5. When implementing the business suite, the business suite released in step S4 is combined with the operating platform framework to generate a software package, the software package is deployed on the hardware server to run and the operating platform is started; Step S6. After the operation platform is successfully started, pipelines are drawn and created by dragging and dropping through the visual interface, and instance nodes of the pipelines and multiple components in the pipelines are configured; Step S7. After the pipeline configuration is completed, the pipeline is checked to determine whether there is an abnormal configuration. If yes, the pipeline cannot be created and a configuration abnormality is prompted. If no, the pipeline is created and the pipeline data is persisted to the file system. Step S8. Query the created pipeline in the pipeline list and operate the pipeline; Step S9. After the pipeline is started, the data is extracted, processed and written according to the pipeline configuration, the data interaction is completed, the log is recorded and the warning message is sent; The component template in step S2 includes a template function form of a code editor, an integrated development environment form, and a package manager form, and different forms of the component template are used to develop the component in different ways; When using the template function form of the code editor, the integrated development environment form, and the package manager form, you need to extract the configuration and import the extracted configuration on each development machine to obtain the component template code; When using the plug-in package manager form of the package manager, install the corresponding package manager on the development machine and obtain the component template code through the specified command; The component template code includes basic dependency information and API version information; The business suite assembly logic in step S4 includes: Download and decompress the compressed packages of the data interaction engine and components respectively; Combine the two folders into the running platform folder; Compress the folder and distribute it into different types of compressed packages for use by various systems; Decompress it during deployment and execute the script in the running platform folder to start running; The operation method of running the platform framework in step S5 includes: The runtime framework provides a default runtime module; Providing customized functions according to the operating platform module and being replaced or developed as needed, the customized functions including user management functions, authority management functions and UI functions for user operations; The combined software package is regarded as an executable platform, wherein the software package includes a file directory structure; Performing data interaction within the executable platform according to the data interaction engine rules; The data interaction engine rules include: The configuration module determines whether the configuration is normal. If so, the component loading module scans the component path. If not, the loading ends. When the configuration is normal, a loader is generated to record component information, and the component is loaded to determine whether the component is loaded successfully. If yes, the engine is started and loading is ended. If not, loading is ended. The step S6 comprises: Configure the pipeline, including the scheduled start and stop configuration of the pipeline, the exception handling configuration of the pipeline, the configuration of whether the pipeline is clustered, the configuration of the pipeline cluster node address, the configuration of the pipeline running interval, the configuration of the pipeline startup parameters, and the configuration of the pipeline email notification; The instance nodes of the plurality of components in the pipeline include a data source node, a data processing node and a target node; Configure instance nodes independently. The data source in the pipeline is configured for the data source node, the data source includes a data query API or a database system provided by the file system, and the data source node has a feature of reading data; Configuring processing logic for the data processing node, and processing the data processing node through a remote API, a script, or logic built into a component of the data processing node; The final operation of configuring data on the target node includes writing the data of the file system into the API or database system. The target node points to the data source configured by the data source node, and the target node has the feature of writing data.
2. According to claim 1, a data interaction method based on distributed components and cloud platform, It is characterized in that The step S3 includes: recording the components that have passed the review, and the components that have passed the review are downloaded and used and combined with the data interaction engine in any way.
3. According to claim 1, a data interaction method based on distributed components and cloud platform, It is characterized in that The method of operating the pipeline in step S8 includes: Start a pipeline, edit a pipeline, delete a pipeline, and export a pipeline; Click Start Pipeline to run according to the pipeline configuration; Click Edit Pipeline to jump to the pipeline drawing interface and modify the pipeline configuration and node configuration; Click Delete Pipeline to delete it from the file system, database, or other persistent destination. Click Export Pipeline to generate a text file for the pipeline configuration.
4. A data interaction system based on distributed components and a cloud platform, used to implement a data interaction method based on distributed components and a cloud platform as described in any one of claims 1 to 3, It is characterized in that The system includes a data interaction engine, a component library, a management cloud platform and an operation platform, wherein: The data interaction engine is used to check the pipeline and determine whether there is an abnormal configuration. If yes, the pipeline cannot be created and a configuration abnormality is prompted. If no, the pipeline is created and the pipeline data is persisted to the file system. The data interaction engine includes a set of API specifications for defining the properties and functions of components, pipelines, fragments, and templates. The data interaction engine includes a set of visualization interfaces, which are used to display components and draw pipelines, fragments, and templates. The component library is used to provide component development templates, and quickly develop new components based on the development templates; the component library is highly decoupled from the data interaction engine and the operating platform, and is used for independent development and management; The management cloud platform is used in the development phase. Developers upload packaged component packages through the management cloud platform and submit the code repository address. It is used to manage various versions and component libraries of the data interaction engine, and bind various versions of the engine and multiple components to form different business suites. The operating platform is used in the implementation phase to deploy the software package on the hardware server; the operating platform includes a business suite generated on the management cloud platform, which is used for customized development of business-related functions; the operating platform includes a set of default business functions, which include account management functions, permission management functions and pipeline management functions; the operating platform is used to draw and create pipelines by dragging and dropping through a visual interface, and configure instance nodes of pipelines and multiple components in the pipeline.
5. A data interaction system based on distributed components and cloud platform according to claim 4, It is characterized in that The data interaction engine includes a log module, an early warning module, an event module, a credential module, a cache module, a verification module, a trace module and a debugging module, wherein: The log module is used to define a set of log API specifications, to record the log information of components and pipelines at runtime, to provide default implementations of log operations, to customize the recording logic of development logs according to API specifications, and to access external log systems. Configuration module, which is used to define engine configuration items and load configuration when the engine starts. The configuration module supports local file configuration and remote configuration. The warning module is used to provide a set of API specifications for message warnings; it is used to issue warning notifications when an exception occurs during pipeline operation; The event module is used to provide a set of API specifications for event listeners. According to the API specifications, the event listening logic of components or pipelines can be customized. At the same time, the engine of the event module has some commonly used event listeners built in. The credential module is used to manage the credential information generated during the pipeline operation and provide corresponding credentials for the nodes in the pipeline to verify when the pipeline is running; The cache module is used to provide a set of API specifications for cache functions; to provide default implementations; to cache pipeline configuration information and load pipeline configuration information into the cache when the engine starts; Verification module, used to verify whether the configuration of pipelines, fragments and templates is correct; The Trace module is used to record and store the data generated at each node during the pipeline operation and to configure multiple storage methods; The debugging module has a built-in web server for independent operation and is used to debug components, fragments and template functions during the development stage.
Citation Information
Patent Citations
Low-code application software development system and method
CN110825362A
Method for generating visual industrial configuration interaction graph
CN113342327A
Design and construction method of universal front-end UI component library
CN113553035A
Component publishing method and device
CN113626067A
System and method for collaborative designing, development, deployment, execution, monitoring and maintenance of enterprise applications
US20150142949A1
