Workflow configuration method, device, computer equipment and storage medium

Abstract

The present application relates to distributed deployment technology, and provides a workflow configuration method, device, computer equipment and storage medium. The method comprises: receiving a configuration data packet sent by a terminal; when the configuration data packet detects that the configuration is completed and triggers an operation, the configuration data packet is generated according to the workflow data currently displayed in the configuration interface; generating a configuration file according to the configuration data packet; the configuration file includes a component identifier and component parameters corresponding to each component, and a component relationship between the components; querying a corresponding program logic package according to the component identifier; instantiating the program logic package according to the corresponding component parameters to obtain a corresponding entity node; connecting the entity nodes through a distributed message middleware according to the component relationship to obtain a configured workflow; submitting the workflow to a distributed data processing engine to run the workflow through the distributed data processing engine. The use of this method can improve the configuration efficiency of the workflow.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a workflow configuration method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of computer technology, online business has gradually developed with advantages such as convenience and speed, and data processing engines for processing business logic of online business have also gradually developed. At present, the workflow configuration of data processing engines is usually implemented by R&D personnel writing customized programs based on business logic. In this process, R&D personnel and business personnel may need to communicate business logic repeatedly and constantly adjust customized program logic. As a result, a lot of manpower and time are consumed, and there is a problem of low configuration efficiency. Summary of the invention

[0003] Based on this, it is necessary to provide a workflow configuration method, apparatus, computer device and storage medium that can improve the configuration efficiency of the workflow in response to the above technical problems.

[0004] A workflow configuration method, the method comprising:

[0005] Receiving a configuration data packet sent by a terminal; the configuration data packet is generated according to the workflow data currently displayed in the configuration interface when a configuration completion trigger operation is detected;

[0006] Generate a configuration file according to the configuration data packet; the configuration file includes a component identifier and component parameters corresponding to each component, and a component relationship between the components;

[0007] Querying a pre-configured corresponding program logic package according to the component identifier;

[0008] Instantiate the program logic package according to corresponding component parameters to obtain corresponding entity nodes;

[0009] Connecting corresponding entity nodes through distributed message middleware according to the component relationship to obtain the configured workflow;

[0010] The workflow is submitted to a distributed data processing engine so as to be executed by the distributed data processing engine.

[0011] In one embodiment, generating a configuration file according to the configuration data packet includes:

[0012] Extracting a component identifier corresponding to each component from the configuration data packet;

[0013] extracting component parameters from the configuration data packet according to a component parameter template corresponding to the component identifier;

[0014] Extracting component relationships between components from the configuration data packet;

[0015] A corresponding configuration file is generated according to the component identification, the component parameters and the component relationship.

[0016] In one embodiment, instantiating the program logic package according to corresponding component parameters to obtain corresponding entity nodes includes:

[0017] Determine the initial number of entity nodes corresponding to each component according to the configuration file;

[0018] The program logic package corresponding to each component is instantiated according to the corresponding component parameters to obtain entity nodes of the corresponding initial entity node quantity.

[0019] In one embodiment, the connecting corresponding entity nodes through a distributed messaging middleware according to the component relationship to obtain the configured workflow includes:

[0020] Determine the distributed message middleware corresponding to the input interface and the output interface of the corresponding component according to the component parameters of each component in the configuration file;

[0021] The entity nodes corresponding to each component are connected to the distributed message middleware according to the corresponding component relationship to obtain the configured workflow.

[0022] In one embodiment, the method further comprises:

[0023] receiving a workflow adjustment request sent by the terminal;

[0024] Acquire a corresponding configuration data packet according to the workflow adjustment request;

[0025] Sending the acquired configuration data packet to the terminal;

[0026] The corresponding workflow is dynamically adjusted according to the adjustment data packet fed back by the terminal in response to the sent configuration data packet.

[0027] In one embodiment, the steps of the distributed data processing engine running the workflow include:

[0028] The distributed data processing engine dynamically monitors the current load of each entity node in the workflow during the execution of the workflow;

[0029] When the current load is greater than or equal to a preset load threshold, the distributed data processing engine determines the number of newly added entity nodes corresponding to the corresponding component;

[0030] The distributed data processing engine instantiates the corresponding program logic package according to the number of newly added entity nodes and corresponding component parameters to obtain newly added entity nodes.

[0031] In one embodiment, the step of the distributed data processing engine running the workflow further includes:

[0032] The distributed data processing engine dynamically monitors the running status of each entity node in the workflow during the running of the workflow;

[0033] When an entity node with an operation status of operation failure is monitored, the distributed data processing engine processes the business request allocated to the entity node with operation failure through the entity node corresponding to the component corresponding to the entity node with operation failure and with an operation status of normal operation.

[0034] A workflow configuration device, the device comprising:

[0035] A receiving module, configured to receive a configuration data packet sent by a terminal; the configuration data packet is generated according to the workflow data currently displayed in the configuration interface when a configuration completion trigger operation is detected;

[0036] A generating module, used to generate a configuration file according to the configuration data packet; the configuration file includes a component identifier and component parameters corresponding to each component, and a component relationship between the components;

[0037] A query module, used for querying a pre-configured corresponding program logic package according to the component identifier;

[0038] An instantiation module, used to instantiate the program logic package according to corresponding component parameters to obtain corresponding entity nodes;

[0039] A configuration module, used to connect corresponding entity nodes through a distributed messaging middleware according to the component relationship to obtain a configured workflow;

[0040] The submission module is used to submit the workflow to the distributed data processing engine so as to run the workflow through the distributed data processing engine.

[0041] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned embodiments when executing the computer program.

[0042] A computer-readable storage medium stores a computer program, which implements the steps in the above-mentioned embodiments when executed by a processor.

[0043] The above-mentioned workflow configuration method, device, computer equipment and storage medium perform visual configuration of workflow data through a terminal, and the workflow data can be quickly configured to obtain a workflow, thereby reducing the configuration cost of the workflow and improving the configuration efficiency. After receiving the configuration data packet generated and sent by the terminal according to the workflow data of the visual configuration, a configuration file is generated according to the configuration data packet, and the entity nodes constituting the workflow are determined according to the component identification and component parameters in the configuration file and the logical program packages corresponding to each component identification. According to the component relationship in the configuration file, each entity node is connected through a distributed message middleware to obtain a workflow, and the workflow is run through a distributed data processing engine to realize the timely processing of large-scale real-time data. Adjacent entity nodes in the workflow are connected through a distributed message middleware, and the workflow is run through a distributed data processing engine. When the amount of real-time data to be processed increases dramatically, the real-time data with a dramatic increase in data volume can be responded to in a timely manner by dynamically adding entity nodes without reconfiguring the workflow, which can further reduce the configuration cost of the workflow and improve the configuration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An application environment diagram of a workflow configuration method in an embodiment;

[0045] Figure 2 A flowchart of a workflow configuration method in one embodiment;

[0046] Figure 3 A flowchart of a workflow configuration method in another embodiment;

[0047] Figure 4 A structural block diagram of a workflow configuration device in one embodiment;

[0048] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The workflow configuration method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. When the terminal 102 detects that the configuration is completed and triggers the operation, it generates a configuration data packet according to the workflow data currently displayed in the configuration interface, and sends the generated configuration data packet to the server 104. The server 104 generates a configuration file including the component identification and component parameters corresponding to each component and the component relationship between the components according to the received configuration data packet, queries the pre-configured corresponding program logic package according to each component identification, instantiates the queried program logic package according to the corresponding component parameters, obtains the entity node corresponding to the corresponding component, connects the corresponding entity nodes through the distributed message middleware according to the component relationship in the configuration file, obtains the configured workflow, and submits the configured workflow to the distributed data processing engine to run the workflow through the distributed data processing engine. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented with an independent server or a server cluster composed of multiple servers.

[0051] In one embodiment, Figure 2 As shown, a workflow configuration method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0052] Step 202, receiving a configuration data packet sent by a terminal; when a configuration completion trigger operation is detected, the configuration data packet is generated according to the workflow data currently displayed in the configuration interface.

[0053] Among them, the configuration data packet is a data packet generated according to the workflow data of the visual configuration. The configuration data packet may specifically include the component identification and component parameters corresponding to each component specified by the workflow data, as well as the component relationship between the various components, and may also include the location information corresponding to each component. The workflow data is the data dynamically configured when the workflow is visually configured in the configuration interface of the terminal, and may specifically include the components that make up the workflow, the component parameters of each component, and the component relationship between the various components, and may also include the location information of each component.

[0054] The component identifier is used to uniquely identify the component. Component parameters refer to parameters that can change dynamically and are specified by business personnel during the configuration process. For example, for mathematical operation components, component parameters include the data table for parameter component operation, fields in the data table, and assignment fields for operation results. Component relationship refers to the dependency relationship between two components, such as the input data of one component depends on the output data of another component, and the output data of one component is used as the input data of another component. Therefore, according to the sorting and connection relationship of each component in the workflow visually configured in the configuration interface, the dependency relationship between the components can be determined, that is, the component relationship between the components can be determined. The location information of the component can refer to the information of the display position of the component in the configuration interface.

[0055] The configuration completion trigger operation is a trigger operation that indicates that the workflow data configuration is completed, and is used to instruct the terminal to generate and send the corresponding configuration data packet according to the workflow data currently configured in the configuration interface. The configuration completion trigger operation can be a trigger operation of a pre-configured configuration completion trigger control, or other trigger operations, such as clicking, sliding, or long pressing on the configuration interface.

[0056] Specifically, the terminal performs visual configuration of workflow data on the configuration interface according to the configuration trigger operation of the business personnel on the configuration interface, and dynamically detects the configuration completion trigger operation of the business personnel. When the configuration completion trigger operation is detected, the terminal generates a corresponding configuration data packet according to the workflow data currently displayed on the configuration interface, and sends the generated configuration data packet to the server. Correspondingly, the server receives the configuration data packet generated and sent by the terminal.

[0057] In one embodiment, the configuration trigger operation includes a component selection operation, a component deletion operation, a component position adjustment operation, a component parameter editing operation, and a component relationship establishment operation. The component selection operation includes selecting a component from a candidate component and dragging it to the corresponding position of the configuration interface for display. The component deletion operation includes deleting the selected component from the configuration interface. The component position adjustment operation includes adjusting the display position of the component in the configuration interface by dragging and dropping. Component parameter editing refers to editing component parameters by entering or selecting. The component relationship establishment operation includes connecting the configured components in the configuration interface with arrow lines to establish a dependency relationship between components. In this way, business personnel dynamically adjust components in the terminal configuration interface by dragging and dropping to achieve visual configuration of the workflow.

[0058] Step 204, generating a configuration file according to the configuration data packet; the configuration file includes a component identifier and component parameters corresponding to each component, and a component relationship between the components.

[0059] The configuration file is a reference file for configuring the workflow, and includes component identifiers and component parameters corresponding to each component specified by the workflow data, as well as component relationships between the components.

[0060] Specifically, the server parses the received configuration data packet to obtain the component identification and component parameters corresponding to each component specified by the corresponding workflow data, as well as the component relationship between each component, and generates a corresponding configuration file based on the obtained component identification and corresponding component participation, as well as the component relationship between each component.

[0061] In one embodiment, the server extracts the component identifier corresponding to each component from the configuration data packet, extracts the corresponding component parameters from the configuration data packet according to each extracted component identifier, extracts the component relationship between the corresponding components from the configuration data packet according to each component identifier, and obtains the corresponding configuration file according to the extracted component identifier and the corresponding component parameters and the component relationship. Specifically, the server can extract the component identifier corresponding to each component from the configuration data packet by traversing the configuration data packet.

[0062] In one embodiment, the server generates a configuration file in a preset format according to the received configuration data packet. The server maps the components in the configuration data packet to the description entities in the configuration file, and accordingly maps the component identifiers to the description entity identifiers, maps the component relationships to the entity relationships, and maps the component parameters to the entity parameters. The preset format is, for example, XML (Extensible Markup Language). The configuration file in the preset format can be understood as an XML description file.

[0063] Step 206: query the pre-configured corresponding program logic package according to the component identifier.

[0064] Among them, the program logic package is a program package that includes the data processing logic of the component. The program logic package also refers to a program package that encapsulates a program or function that can implement the data processing function of the component. After instantiation, the program logic package can be used to implement the data processing function of the component. Each component can be used to implement a single data processing function, which is a common function or basic function in business processing. Thus, a specific combination of multiple components can be used to implement the corresponding business logic. Components that implement a single data processing function, such as data type conversion components, mathematical operation components, data aggregation components, data splitting components, data merging components, data sorting components, and components that implement similar common functions are not listed here one by one.

[0065] Specifically, the server pre-stores a corresponding program logic package locally for each component. After generating a configuration file according to the configuration data package, the server queries the pre-configured program logic package for each component identifier locally according to each component identifier in the configuration file.

[0066] In one embodiment, the server is locally pre-configured with a logic node library, which pre-stores a program logic package pre-configured for each component identifier. The server parses the configuration file to obtain one or more component identifiers, and queries the program logic package corresponding to each component identifier from the pre-configured logic node library.

[0067] In one embodiment, the program logic package corresponding to each component is pre-configured by the developer and stored locally on the server, so that after the server determines the components that constitute the workflow based on the configuration data packet generated and sent by the terminal, it can quickly obtain the corresponding program logic package based on the component identifier, and then configure the workflow based on the program logic package. In this way, business personnel can perform visual configuration of workflow data through the terminal, that is, visual configuration of the workflow can be achieved through the terminal, without the need for R&D personnel to write customized programs to configure the corresponding workflow, which can improve the configuration efficiency of the workflow.

[0068] Step 208: instantiate the program logic package according to the corresponding component parameters to obtain the corresponding entity node.

[0069] The entity node is a node obtained by instantiating a program logic package and can be used to implement the data processing function of the component.

[0070] Specifically, for each component identifier in the configuration file, the server instantiates the program logic package corresponding to the component identifier according to the component parameters corresponding to the component identifier, and obtains the entity node corresponding to the component identifier. Thus, the server instantiates the program logic packages corresponding to each component identifier in the configuration file respectively, and obtains the entity nodes corresponding to each component identifier.

[0071] In one embodiment, the server instantiates the program logic package corresponding to each component identifier as an entity node under the corresponding working node. The workflow is composed of one or more working nodes, each working node corresponds to a component, and each working node includes one or more entity nodes. Each entity node under each working node is instantiated by the program logic package corresponding to the corresponding component, so that each entity node under each working node is used to implement the same data processing function. The entity nodes under different working nodes are usually used to implement different data processing logics. It can be understood that a single workflow may include multiple working nodes corresponding to the same component. When the entity nodes under the multiple working nodes are instantiated based on the program logic package corresponding to the component, the parameter values ​​of the component parameters based on which the instantiation is based may be different, that is, the data processing logic implemented when the entity nodes under the multiple working nodes are run is similar, but the data processing functions specifically implemented are different.

[0072] In one embodiment, the server instantiates the program logic package corresponding to each component identifier to obtain one or more entity nodes corresponding to each component identifier. The number of entity nodes instantiated for the program logic package corresponding to each component identifier can be specifically specified by a configuration file.

[0073] In one embodiment, the program logic package can be understood as a program package including the data processing logic of the component. After the program logic package is instantiated to obtain the corresponding entity node, the data processing logic of the component can be implemented by running the entity node, thereby completing the data processing function of the component.

[0074] In one embodiment, after obtaining the program logic package corresponding to each component identifier in the configuration file, the server can instantiate each obtained program logic package according to the corresponding component parameters to obtain the corresponding entity node. The server can also instantiate the currently obtained program logic package according to the corresponding component parameters after obtaining the program logic package corresponding to a single component identifier to obtain the entity node corresponding to the component identifier. It can be understood that the step of instantiating the program logic packages corresponding to each component identifier in the configuration file to obtain the corresponding entity node can be executed sequentially by a single thread or in parallel by multiple threads.

[0075] Step 210: Connect corresponding entity nodes through distributed message middleware according to component relationships to obtain the configured workflow.

[0076] Distributed message middleware is a middleware used to receive, store and distribute messages. Distributed message middleware includes but is not limited to ActiveMQ (an open source message middleware), RabbitMQ (a message-oriented middleware) and Kafka (a high-throughput distributed publish-subscribe messaging system / distributed message queue).

[0077] Specifically, the server determines the component relationships between the components and the distributed message middleware corresponding to each component according to the configuration file. The server connects the entity nodes corresponding to the corresponding components through the corresponding distributed message middleware according to the determined component relationships to obtain the workflow configured according to the configuration file.

[0078] In one embodiment, the configuration file specifies the distributed message middleware between two adjacent components, and the adjacent two components refer to two components that have a component relationship in the configuration file. The server can determine the distributed message middleware between the two components that have a component relationship according to the configuration file, and connect the entity nodes corresponding to the two components according to the determined component relationship and the corresponding distributed message middleware.

[0079] In one embodiment, when there are multiple entity nodes corresponding to a component, it is necessary to connect the multiple entity nodes corresponding to the component with the entity nodes corresponding to the corresponding component through the distributed message middleware corresponding to the component according to the component relationship corresponding to the component. In this way, the multiple entity nodes corresponding to each component read the message or data to be processed from the corresponding distributed message middleware, or write the processed data to the corresponding distributed message middleware, so that each entity node corresponding to the next component can read the message or data to be processed by itself from the distributed message middleware.

[0080] In one embodiment, a complete workflow corresponds to a complete business logic, and the corresponding business logic can be implemented by running the workflow.

[0081] Step 212: submit the workflow to the distributed data processing engine so as to run the workflow through the distributed data processing engine.

[0082] Specifically, after the server obtains the corresponding workflow according to the configuration file, it submits the workflow to the distributed data processing engine, which implements the corresponding business logic by running the workflow.

[0083] In one embodiment, after configuring the workflow, the server dynamically compiles the configured workflow locally. When the workflow is successfully compiled, the server submits the successfully compiled workflow to the distributed data processing engine. In this way, it is possible to avoid submitting a workflow that may fail to compile to the distributed data processing engine, thereby improving the configuration efficiency of the submitted workflow.

[0084] In one embodiment, the configured workflow includes one or more work nodes, each work node corresponds to a component, and each work node includes one or more entity nodes instantiated by the program logic package corresponding to the corresponding component. The distributed data processing engine can be a distributed computing cluster, and each work node in the workflow can be a container in the distributed computing cluster, and the input and output of the container are connected through a distributed message middleware.

[0085] In one embodiment, a distributed data processing engine such as Apache Flink.

[0086] The above workflow configuration method performs visual configuration of workflow data through a terminal, and the workflow data can be quickly configured to obtain a workflow, thereby reducing the configuration cost of the workflow and improving the configuration efficiency. After receiving the configuration data packet generated and sent by the terminal according to the workflow data of the visual configuration, a configuration file is generated according to the configuration data packet, and the entity nodes constituting the workflow are determined according to the component identification and component parameters in the configuration file and the logical program packages corresponding to each component identification. According to the component relationship in the configuration file, each entity node is connected through a distributed message middleware to obtain a workflow, and the workflow is run through a distributed data processing engine to realize the timely processing of large-scale real-time data. Adjacent entity nodes in the workflow are connected through a distributed message middleware, and the workflow is run through a distributed data processing engine. When the amount of real-time data to be processed increases dramatically, the real-time data with a dramatic increase in data volume can be responded to in a timely manner by dynamically adding entity nodes without reconfiguring the workflow, which can further reduce the configuration cost of the workflow and improve the configuration efficiency.

[0087] In one embodiment, step 204 includes: extracting a component identifier corresponding to each component from a configuration data packet; extracting component parameters from the configuration data packet according to a component parameter template corresponding to the component identifier; extracting component relationships between components from the configuration data packet; and generating a corresponding configuration file according to the component identifier, component parameters and component relationships.

[0088] The component parameter template is a template for specifying component parameters corresponding to a component, and may specifically include a component parameter identifier corresponding to each component parameter corresponding to the component. The component parameter template may also include parameter relationships between various component parameters.

[0089] Specifically, the server extracts the component identifier corresponding to each component from the configuration data packet, obtains the corresponding component parameter template according to each component identifier, and extracts the corresponding component parameter from the configuration data packet according to the component parameter template and the corresponding component identifier to obtain the component parameter corresponding to each component identifier. The server extracts the component relationship between each component from the configuration data packet, and generates a corresponding configuration file according to the component identifier and component parameter corresponding to each component, and the component relationship between each component.

[0090] In one embodiment, the server pre-configures a corresponding component parameter template for each component, and stores the component parameter template and the corresponding component identifier in a local storage. The server can query the corresponding component parameter template locally based on the component identifier extracted from the configuration data packet. The server can determine the component parameter identifier corresponding to the corresponding component based on the component parameter template, and can extract the corresponding component parameters from the configuration data packet based on the component parameter identifier, thereby obtaining one or more component parameters corresponding to each component identifier. It can be understood that if the component parameter templates corresponding to different components may include the same component parameter identifier, the server can extract the corresponding component parameters from the configuration data packet based on the component parameter identifier and the corresponding component identifier.

[0091] In one embodiment, after the server extracts the component identifier from the configuration data packet, it matches each extracted component identifier with a preconfigured component identifier set to verify the correctness of the component identifier. When it is determined that there is a component identifier among the extracted component identifiers that fails to match the component identifier set, the server sends the component identifier that fails to match to the terminal to instruct the terminal to further review the component identifier and feedback the corresponding review result. The review result may include a component identifier for replacing the component identifier that fails to match. In this way, the accuracy of the component identifier can be guaranteed, so that subsequent workflow configuration operations can be performed based on component identifiers with higher accuracy, which can improve the configuration accuracy of the workflow.

[0092] In the above embodiment, based on the component parameter template corresponding to each component identifier, the component parameters corresponding to the corresponding component identifier are extracted from the configuration data packet respectively, which can eliminate irrelevant data in the configuration data packet and only extract relevant data required for the configuration workflow, thereby improving the accuracy of the configuration file. It can be understood that the configuration data packet is the front-end data generated by the terminal according to the workflow data of the configuration interface, and may usually include irrelevant data not involved in the server configuration workflow, such as the location information of each component in the configuration interface. Therefore, the process of the server generating the configuration file according to the configuration data packet is to generate the back-end data based on the configuration workflow according to the front-end data, and the irrelevant data of the configuration workflow has been eliminated in the back-end data, and includes the relevant data required for the configuration workflow, such as the component relationship between components. In this way, the process of generating the configuration file based on the configuration data packet is the process of eliminating irrelevant data.

[0093] In one embodiment, step 208 includes: determining the initial number of entity nodes corresponding to each component according to the configuration file; instantiating the program logic package corresponding to each component according to the corresponding component parameters to obtain entity nodes corresponding to the initial number of entity nodes.

[0094] The initial number of entity nodes corresponding to a component refers to the number of entity nodes instantiated from a program logic package corresponding to the component during configuration of the workflow.

[0095] Specifically, the configuration file also includes the initial number of entity nodes corresponding to each component. The server extracts the initial number of entity nodes corresponding to each component from the configuration file. For each component in the configuration file, the server instantiates the program logic package corresponding to the component according to the component parameters corresponding to the component and the initial number of entity nodes, and obtains entity nodes corresponding to the initial number of entity nodes of the component.

[0096] In one embodiment, the server instantiates the program logic package corresponding to each component according to the component parameters corresponding to the component, so as to give physical meanings to the component parameters in the program logic package and obtain corresponding entity nodes. In this way, the process of running the entity node is the process of calling the program logic package to which the component parameters have been given corresponding physical meanings, so that the corresponding data processing function can be realized by running the entity node.

[0097] In the above embodiment, the initial number of entity nodes corresponding to each component in the initially configured workflow is determined based on the configuration file. In this way, entity nodes corresponding to the initial number of entity nodes are configured in the workflow for each component to implement the corresponding data processing function of the component, so that the initially configured workflow can be suitable for processing real-time data with a large amount of data, so as to avoid the need to reconfigure the workflow due to excessive data processing pressure of the workflow when the real-time data volume is large, thereby improving the configuration efficiency of the workflow and reducing the configuration cost.

[0098] In one embodiment, step 210 includes: determining the distributed message middleware corresponding to the input interface and the output interface of the corresponding component according to the component parameters of each component in the configuration file; connecting the entity node corresponding to each component with the distributed message middleware according to the corresponding component relationship to obtain the configured workflow.

[0099] Specifically, after the server extracts the component parameters corresponding to the component from the configuration file, it extracts the distributed message middleware corresponding to the input interface and output interface of the component from the component parameters. According to the component relationship between the components in the configuration file, the server connects the input interface and output interface of the entity node corresponding to each component to the corresponding distributed message middleware, so as to realize the connection of the entity nodes corresponding to the adjacent components through the distributed message middleware, and obtain a configured workflow. It can be understood that for two adjacent components with a component relationship, the distributed message middleware corresponding to the output interface of the former component is consistent with the distributed message middleware corresponding to the input interface of the latter component.

[0100] For example, suppose that the configuration file includes three components, namely component 1, component 2 and component 3. The component relationship between component 1 and 2 is that component 1 is the previous adjacent component of component 2, and the component relationship between component 2 and component 3 is that component 2 is the previous adjacent component of component 3. The distributed message middleware corresponding to the output interface of component 1 is X1, and the distributed message middleware corresponding to the input interface and output interface of component 2 are X1 and X2 respectively, and the distributed message middleware corresponding to the input interface of component 3 is X2. It can be seen that according to the component relationship between each component and the corresponding distributed message middleware, the entity nodes corresponding to component 1 are connected to the entity nodes corresponding to component 2 through the distributed message middleware X1, and the entity nodes corresponding to component 2 are connected to the entity nodes corresponding to component 3 through the distributed message middleware X2, and a workflow composed of the entity nodes corresponding to components 1-3 is obtained.

[0101] In one embodiment, the entity nodes corresponding to adjacent components are connected through a distributed message middleware, so that the entity node corresponding to the previous component stores the output data into the distributed message middleware for caching, so that the entity node corresponding to the next component can read the cached data from the distributed message middleware and process it. In this way, business decoupling between entity nodes can be achieved, thereby improving the applicability of the workflow while reducing the efficiency of workflow configuration.

[0102] In the above embodiment, the entity nodes corresponding to two adjacent components are connected in series through a distributed message middleware so that the entity nodes can be dynamically expanded according to the amount of real-time data to be processed, thereby improving the scalability of the workflow implementation and avoiding the need to reconfigure the workflow when the amount of real-time data increases dramatically, thereby improving the configuration efficiency of the workflow and reducing the configuration cost.

[0103] In one embodiment, the workflow configuration method further includes: receiving a workflow adjustment request sent by a terminal; obtaining a corresponding configuration data packet according to the workflow adjustment request; sending the obtained configuration data packet to the terminal; and dynamically adjusting the corresponding workflow according to an adjustment data packet fed back by the terminal in response to the sent configuration data packet.

[0104] The workflow adjustment request is a request for triggering a workflow adjustment operation, and may specifically include a workflow identifier of the workflow to be adjusted.

[0105] Specifically, the server receives a workflow adjustment request sent by the terminal, determines a corresponding workflow identifier according to the received workflow adjustment request, obtains a corresponding configuration data packet according to the determined workflow identifier, and sends the obtained configuration data packet to the terminal. The terminal visually displays the corresponding workflow data on the configuration interface according to the received configuration data packet, so that business personnel can visually adjust the workflow data through the configuration interface. When the terminal detects that the adjustment completed trigger operation triggered by the business personnel, it triggers the generation and sending of the corresponding adjustment data packet. After receiving the adjustment data packet sent by the terminal, the server dynamically updates the corresponding configuration file according to the adjustment data packet, dynamically adjusts the corresponding workflow according to the updated configuration file, and resubmits the dynamically adjusted workflow to the distributed data processing engine to achieve visual dynamic adjustment of the workflow.

[0106] In one embodiment, the server queries the corresponding configuration file locally according to the determined workflow identifier, and converts the queried configuration file into a configuration data packet. The server may also store the corresponding configuration data packet locally for the configured workflow, so that the server directly queries the corresponding stored configuration data packet locally according to the workflow identifier in the received workflow adjustment request.

[0107] In one embodiment, when the server detects the adjustment completion trigger operation, it can generate a corresponding adjustment data packet according to the workflow data adjusted at the time in the configuration interface, or it can generate a corresponding adjustment data packet according to the workflow data currently displayed in the configuration interface, and send the generated adjustment data packet to the server. It can be understood that the workflow data adjusted at the time includes one or more of the components adjusted at the time, component parameters, and component relationships.

[0108] In one embodiment, the server may dynamically adjust the configured workflow based on the dynamically updated configuration file, or reconfigure the workflow according to the above workflow configuration process based on the dynamically updated configuration file, and submit the reconfigured workflow to the distributed data processing engine.

[0109] In the above embodiment, visually modifying the configured workflow through the terminal can improve the workflow modification efficiency, and dynamically adjusting the configured workflow according to the adjustment data packet can further improve the workflow modification efficiency and reduce the complexity of the modification operation.

[0110] In one embodiment, the steps of running a workflow by a distributed data processing engine include: the distributed data processing engine dynamically monitors the current load of each entity node in the workflow during the running of the workflow; when the current load is greater than or equal to a preset load threshold, the distributed data processing engine determines the number of new entity nodes corresponding to the corresponding component; the distributed data processing engine instantiates the corresponding program logic package according to the number of new entity nodes and the corresponding component parameters to obtain new entity nodes.

[0111] The preset load threshold is a preset threshold used to compare with the current load of the physical node to determine whether it is necessary to dynamically add a corresponding physical node, which can be a load amount or a load proportion. The number of newly added physical nodes refers to the number of physical nodes that need to be dynamically added.

[0112] Specifically, during the operation of the configured workflow, the distributed data processing engine dynamically monitors the current load of each entity node in the workflow, and compares the dynamically monitored current load with the corresponding preset load threshold. When it is determined that the current load of the entity node is greater than or equal to the corresponding preset load threshold, the distributed data processing engine dynamically determines the number of newly added entity nodes corresponding to the component corresponding to the entity node, and instantiates the program logic package corresponding to the component according to the component data corresponding to the component and the number of newly added entity nodes, and obtains the newly added entity nodes corresponding to the number of newly added entity nodes corresponding to the component.

[0113] In one embodiment, when it is determined that the current load of the entity node is greater than or equal to the preset load threshold, the distributed data processing engine dynamically determines the corresponding number of newly added entity nodes based on the current load and the preset load threshold. For example, when the CPU resources currently occupied by the entity node are greater than or equal to 90% of the resource quota allocated to the entity node, the number of entity nodes corresponding to the component corresponding to the entity node is doubled.

[0114] In one embodiment, for a dynamically added entity node, the distributed data processing engine connects the newly added entity node to the corresponding distributed message middleware according to the connection relationship between the existing entity nodes corresponding to the component corresponding to the newly added entity node and the distributed message middleware, so as to realize the connection between the newly added node and the entity nodes corresponding to other components.

[0115] In one embodiment, the distributed data processing engine dynamically monitors the current load of each working node in the workflow. When the current load of the working node is greater than or equal to the corresponding specified load threshold, the program logic package corresponding to the component corresponding to the working node is instantiated to dynamically increase the number of entity nodes under the working node.

[0116] In the above embodiment, since the entity nodes corresponding to adjacent components in the workflow are connected through the distributed message middleware, when it is monitored that the current load of the entity node is greater than or equal to the preset load threshold, the corresponding entity node can be dynamically added, and the dynamically added entity node is still connected through the corresponding distributed message middleware, so that the data processing logic of the corresponding component can be processed jointly by the newly added entity node and the existing entity node, which can reduce the data processing pressure of each entity node and thus improve the data processing efficiency.

[0117] In one embodiment, the step of the distributed data processing engine running the workflow also includes: the distributed data processing engine dynamically monitors the running status of each entity node in the workflow during the running of the workflow; when monitoring the entity node with a running status of running failure, the distributed data processing engine processes the business request allocated to the entity node with running failure through the entity node corresponding to the component corresponding to the entity node with running failure and with a running status of running normally.

[0118] The running state may refer to the running state of the physical node, or the running state of the host running the physical node. The running state includes normal running and running failure.

[0119] Specifically, during the operation of the workflow, the distributed data processing engine dynamically monitors the operation status of each entity node in the workflow. When monitoring an entity node with an operation status of an operation failure, the distributed data processing engine determines other entity nodes corresponding to the components corresponding to the entity node with the operation failure in the workflow, selects an entity node with a current operation status of normal operation from the other entity nodes, and reallocates the business request allocated to the entity node with the operation failure to the selected entity node, so as to continue to process the reallocated business request through the selected entity node. It can be understood that the distributed data processing engine can select one or more entity nodes that are operating normally from the workflow in the above manner.

[0120] In one embodiment, the distributed data processing engine dynamically monitors the running status of each entity node in the workflow. When a failure is detected in the running of an entity node, the service request assigned to the entity node with the failure is reallocated to other entity nodes under the working node where the entity node is located and whose current running status is normal. It can be understood that since the entity nodes under each working node are deployed in a distributed manner, when an entity node fails to run, the service request can be continued to be processed by other entity nodes under the corresponding working node to ensure the reliability of the workflow.

[0121] In one embodiment, since the multiple entity nodes corresponding to each component are deployed in a distributed manner, when the host running a certain entity node fails, the distributed data processing engine can continue to process business requests through other entity nodes corresponding to the same component running on other hosts, and can also re-instantiate an entity node for responding to corresponding business requests on a host that has not yet run any entity node corresponding to the component corresponding to the entity node.

[0122] In the above embodiment, the operation status of the entity nodes in the workflow is dynamically monitored, and the service requests allocated to the entity nodes with operation failures are timely reallocated to other entity nodes running normally, so as to ensure the timeliness of the response to the service requests and the reliability of the workflow.

[0123] like Figure 3 As shown, in one embodiment, a workflow configuration method is provided, and the method specifically includes the following steps:

[0124] Step 302, receiving a configuration data packet sent by a terminal; when a configuration completion trigger operation is detected, the configuration data packet is generated according to the workflow data currently displayed in the configuration interface.

[0125] Step 304: extract the component identifier corresponding to each component from the configuration data packet.

[0126] Step 306: extract component parameters from the configuration data packet according to the component parameter template corresponding to the component identifier.

[0127] Step 308: extract the component relationship between the components from the configuration data packet.

[0128] Step 310: Generate a corresponding configuration file according to the component identification, component parameters and component relationship.

[0129] Step 312, querying the pre-configured corresponding program logic package according to the component identifier;

[0130] Step 314, determining the initial number of entity nodes corresponding to each component according to the configuration file;

[0131] Step 316: instantiate the program logic package corresponding to each component according to the corresponding component parameters to obtain entity nodes of the corresponding initial number of entity nodes.

[0132] Step 318: Determine the distributed message middleware corresponding to the input interface and the output interface of the corresponding component according to the component parameters of each component in the configuration file.

[0133] Step 320 , connect the entity node corresponding to each component with the distributed message middleware according to the corresponding component relationship to obtain the configured workflow.

[0134] Step 322: submit the workflow to the distributed data processing engine so as to run the workflow through the distributed data processing engine.

[0135] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0136] In one embodiment, Figure 4 As shown, a workflow configuration device 400 is provided, comprising: a receiving module 401, a generating module 402, a query module 403, an instantiation module 404, a configuration module 405 and a submission module 406, wherein:

[0137] The receiving module 401 is used to receive a configuration data packet sent by a terminal; when a configuration completion trigger operation is detected, the configuration data packet is generated according to the workflow data currently displayed in the configuration interface;

[0138] The generating module 402 is used to generate a configuration file according to the configuration data packet; the configuration file includes a component identifier and component parameters corresponding to each component, and a component relationship between the components;

[0139] A query module 403 is used to query a pre-configured corresponding program logic package according to a component identifier;

[0140] The instantiation module 404 is used to instantiate the program logic package according to the corresponding component parameters to obtain the corresponding entity node;

[0141] Configuration module 405, used to connect corresponding entity nodes through distributed message middleware according to component relationships to obtain a configured workflow;

[0142] The submission module 406 is used to submit the workflow to the distributed data processing engine so as to run the workflow through the distributed data processing engine.

[0143] In one embodiment, the generation module 402 is also used to extract the component identifier corresponding to each component from the configuration data packet; extract the component parameters from the configuration data packet according to the component parameter template corresponding to the component identifier; extract the component relationship between the components from the configuration data packet; and generate a corresponding configuration file according to the component identifier, component parameters and component relationship.

[0144] In one embodiment, the instantiation module 404 is further used to determine the initial number of entity nodes corresponding to each component according to the configuration file; instantiate the program logic package corresponding to each component according to the corresponding component parameters to obtain entity nodes of the corresponding initial number of entity nodes.

[0145] In one embodiment, the configuration module 405 is also used to determine the distributed message middleware corresponding to the input interface and output interface of the corresponding component according to the component parameters of each component in the configuration file; the entity node corresponding to each component is connected to the distributed message middleware according to the corresponding component relationship to obtain the configured workflow.

[0146] In one embodiment, the workflow configuration device 400 further includes: an adjustment module;

[0147] The adjustment module is used to receive a workflow adjustment request sent by a terminal; obtain a corresponding configuration data packet according to the workflow adjustment request; send the obtained configuration data packet to the terminal; and dynamically adjust the corresponding workflow according to the adjustment data packet fed back by the terminal for the sent configuration data packet.

[0148] In one embodiment, a distributed data processing engine is used to dynamically monitor the current load of each entity node in the workflow during the execution of the workflow; when the current load is greater than or equal to a preset load threshold, determine the number of new entity nodes corresponding to the corresponding component; instantiate the corresponding program logic package according to the number of new entity nodes and the corresponding component parameters to obtain new entity nodes.

[0149] In one embodiment, the distributed data processing engine is also used to dynamically monitor the operating status of each entity node in the workflow during the execution of the workflow; when an entity node with an operating status of an operating failure is monitored, the business request allocated to the entity node with an operating failure is processed through the entity node corresponding to the component corresponding to the entity node with an operating failure and with an operating status of normal operation.

[0150] For the specific definition of the workflow configuration device, please refer to the definition of the workflow configuration method above, which will not be repeated here. Each module in the above-mentioned workflow configuration device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store program logic packages pre-configured for each component. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a workflow configuration method is implemented.

[0152] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0153] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0154] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0155] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0156] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A workflow configuration method, characterized in that: The method comprises: Receiving a configuration data packet sent by a terminal; the configuration data packet is generated according to the workflow data currently displayed in the configuration interface when a configuration completion trigger operation is detected; Extracting a component identifier corresponding to each component from the configuration data packet, and for each component, determining a component parameter template corresponding to the component from a preconfigured component parameter template according to the component identifier corresponding to the component, wherein the component parameter template includes the component parameter identifier; Extracting component parameters from the configuration data packet according to component parameter identifiers in the component parameter template corresponding to the component, extracting component relationships between components from the configuration data packet, and generating a corresponding configuration file according to the component identifiers, the component parameters and the component relationships; the configuration file includes the component identifiers and component parameters corresponding to each component, and the component relationships between the components; Querying a pre-configured corresponding program logic package according to the component identifier; Instantiate the program logic package according to corresponding component parameters to obtain corresponding entity nodes; Connecting corresponding entity nodes through distributed message middleware according to the component relationship to obtain the configured workflow; Submit the workflow to the distributed data processing engine, determine the distributed message middleware corresponding to the input interface and output interface of the corresponding component according to the component parameters of each component in the configuration file, connect the entity nodes corresponding to each component with the distributed message middleware according to the corresponding component relationship, and obtain the configured workflow.

2. The method according to claim 1, characterized in that The instantiating the program logic package according to the corresponding component parameters to obtain the corresponding entity node includes: Determine the initial number of entity nodes corresponding to each component according to the configuration file; The program logic package corresponding to each component is instantiated according to the corresponding component parameters to obtain entity nodes of the corresponding initial entity node quantity.

3. The method according to claim 1, characterized in that The method further comprises: receiving a workflow adjustment request sent by the terminal; Acquire a corresponding configuration data packet according to the workflow adjustment request; Sending the acquired configuration data packet to the terminal; The corresponding workflow is dynamically adjusted according to the adjustment data packet fed back by the terminal in response to the sent configuration data packet.

4. The method according to any one of claims 1 to 3, characterized in that: The steps of the distributed data processing engine running the workflow include: The distributed data processing engine dynamically monitors the current load of each entity node in the workflow during the execution of the workflow; When the current load is greater than or equal to a preset load threshold, the distributed data processing engine determines the number of newly added entity nodes corresponding to the corresponding component; The distributed data processing engine instantiates the corresponding program logic package according to the number of newly added entity nodes and corresponding component parameters to obtain newly added entity nodes.

5. The method according to claim 4, characterized in that The step of the distributed data processing engine running the workflow further includes: The distributed data processing engine dynamically monitors the running status of each entity node in the workflow during the running of the workflow; When an entity node with an operation status of operation failure is monitored, the distributed data processing engine processes the business request allocated to the entity node with operation failure through the entity node corresponding to the component corresponding to the entity node with operation failure and with an operation status of normal operation.

6. A workflow configuration device, characterized in that: The device comprises: A receiving module, configured to receive a configuration data packet sent by a terminal; the configuration data packet is generated according to the workflow data currently displayed in the configuration interface when a configuration completion trigger operation is detected; A generating module, configured to extract a component identifier corresponding to each component from the configuration data packet, and for each component, determine a component parameter template corresponding to the component from a preconfigured component parameter template according to the component identifier corresponding to the component, wherein the component parameter template includes a component parameter identifier; extract component parameters from the configuration data packet according to the component parameter identifier in the component parameter template corresponding to the component, extract component relationships between components from the configuration data packet, and generate a corresponding configuration file according to the component identifier, the component parameters and the component relationships; the configuration file includes the component identifier and component parameters corresponding to each component, and the component relationships between components; A query module, used for querying a pre-configured corresponding program logic package according to the component identifier; An instantiation module, used to instantiate the program logic package according to corresponding component parameters to obtain corresponding entity nodes; A configuration module, used to connect corresponding entity nodes through a distributed messaging middleware according to the component relationship to obtain a configured workflow; A submission module is used to submit the workflow to a distributed data processing engine, and determine the distributed message middleware corresponding to the input interface and output interface of the corresponding component according to the component parameters of each component in the configuration file; the entity nodes corresponding to each component are connected to the distributed message middleware according to the corresponding component relationship to obtain the configured workflow.

7. The device according to claim 6, characterized in that The device also includes an adjustment module, which is used to receive a workflow adjustment request sent by a terminal; obtain a corresponding configuration data packet according to the workflow adjustment request; send the obtained configuration data packet to the terminal; and dynamically adjust the corresponding workflow according to the adjustment data packet fed back by the terminal for the sent configuration data packet.

8. The device according to claim 7, characterized in that The instantiation module is also used to determine the initial number of entity nodes corresponding to each component according to the configuration file; instantiate the program logic package corresponding to each component according to the corresponding component parameters to obtain entity nodes of the corresponding initial number of entity nodes.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images