Distributed data flow processing method and device, electronic equipment and storage medium

Abstract

The invention provides a distributed data flow processing method and device, electronic equipment and a storage medium. The method comprises the following steps: defining a data structure, a data processing requirement and a data flow process for each data flow, and defining a distributed data processing interface in the data flow process; establishing an event response mechanism based on the data flow, the event response mechanism being used for responding to the system boundary event, generating a corresponding processing instruction, and adding the processing instruction into an instruction queue; extracting the processing instruction from the instruction queue according to a preset strategy, and executing the processing instruction according to the data flow; and establishing a process state management mechanism according to a predefined process cooperation relationship, the process state management mechanism being used for monitoring a data process state, and performing collaborative management of the data process according to the data process state. According to the method, data processing and process management can be integrated, cross-system boundary and heterogeneous system collaborative data process processing is supported, and the data circulation efficiency and timeliness are improved.

Description

Technical Field

[0001] The present application relates to the field of data flow technology for intelligent manufacturing, and in particular to a distributed data flow processing method, device, electronic device and storage medium. Background Art

[0002] In the era of digital transformation of enterprises, data has gradually become a key resource and has shown diversified characteristics. This is not only reflected in the traditional interconnection and interoperability of data, but also involves the standardization and unified management requirements of the entire life cycle of data, covering multiple links such as data generation, circulation, processing, and archiving. Especially in the process of data circulation, cross-system boundary processing, process integration, and collaborative execution are particularly important. However, the data flow within the enterprise is often complicated by heterogeneous systems. These processes may cross different levels of the ISA-95 model or involve multiple system boundaries, resulting in insufficient interoperability between systems.

[0003] In the existing technology, commonly used data process management tools have obvious limitations. For example, workflow tools focus on process control and lack data processing capabilities; ETL tools focus on data processing but lack process management and cross-system collaboration capabilities. These technologies cannot meet the needs of cross-system data flow in fields such as intelligent manufacturing. Especially in heterogeneous system architectures, the difficulties in data flow and tracking management will lead to poor process execution, thereby affecting the timeliness of data and even hindering the key business processing of enterprises. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a distributed data flow processing method, device, electronic device and storage medium to solve the problems existing in the prior art such as difficulty in cross-system collaboration of processes, poor data flow execution and poor data timeliness.

[0005] In a first aspect of an embodiment of the present application, a distributed data flow processing method is provided, including: defining a data structure, data processing requirements and a data flow process for each data flow, and defining a distributed data processing interface in the data flow process; establishing an event response mechanism based on the data flow, the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue; taking out processing instructions from the instruction queue according to a predetermined strategy, and executing the processing instructions according to the data flow, so as to process the data or initiate the next system boundary event; establishing a process status management mechanism according to a predefined process collaboration relationship, the process status management mechanism is used to monitor the data flow status, and collaboratively manage the data flow according to the data flow status.

[0006] According to a second aspect of an embodiment of the present application, a distributed data flow processing device is provided, comprising: a definition module, configured to define a data structure, data processing requirements and a data flow process for each data flow, and to define a distributed data processing interface during the data flow process; an establishment module, configured to establish an event response mechanism based on the data flow, the event response mechanism being used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue; an execution module, configured to retrieve processing instructions from the instruction queue according to a predetermined strategy, and execute the processing instructions according to the data flow, so as to process the data or initiate the next system boundary event; a management module, configured to establish a process status management mechanism according to a predefined process collaboration relationship, the process status management mechanism being used to monitor the data flow status, and to collaboratively manage the data flow according to the data flow status.

[0007] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0008] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0010] By defining the data structure, data processing requirements and data flow process for each data flow, and defining the distributed data processing interface in the data flow process; establishing an event response mechanism based on the data flow, the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to the instruction queue; taking out the processing instructions from the instruction queue according to the predetermined strategy, and executing the processing instructions according to the data flow, so as to process the data or initiate the next system boundary event; establishing a process state management mechanism according to the predefined process collaboration relationship, the process state management mechanism is used to monitor the data flow state, and collaboratively manage the data flow according to the data flow state. This application can integrate data processing and process management, support data flow processing across system boundaries and heterogeneous system collaboration, and improve data flow efficiency and timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 It is a schematic diagram of the overall architecture of the distributed data flow processing system provided in the embodiment of the present application;

[0013] Figure 2 It is a flowchart of a distributed data flow processing method provided in an embodiment of the present application;

[0014] Figure 3 It is a structural diagram of a distributed data flow processing device provided in an embodiment of the present application;

[0015] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0017] In the current era of enterprise digital transformation, data has become a key resource, especially its standardization and lifecycle management are becoming more and more important. The life cycle of data includes the entire process from data generation to archiving. This data not only needs to flow within the same system, but also needs to flow and integrate across systems.

[0018] However, enterprise data flows often involve the coordinated operation of multiple different levels and systems, such as across different levels of the ISA-95 standard (industrial automation hierarchical model), involving production management systems (MES), warehouse management systems (WMS), enterprise resource planning systems (ERP), etc. These heterogeneous systems have different architectures, data definitions, and processing methods. Therefore, the existing technologies still have the following technical problems in data flow across system boundaries:

[0019] Complex heterogeneous system architecture: Different systems are designed with different intentions, which leads to differences in system architecture and data format. Direct interoperability and integration are difficult, and additional data conversion and integration processing steps are required when data is transmitted and transferred between different systems.

[0020] Difficulty in cross-system collaboration: In order to complete complex business processes, enterprises often need to collaborate across multiple systems. However, workflow tools in existing systems (such as Flowable and Activiti) mainly focus on process management and do not have complex data processing capabilities; while ETL tools (such as Kettle and DataX) focus on data processing and have weak process management capabilities, making it difficult to effectively respond to the cross-system collaboration needs of distributed systems.

[0021] Data flow is not smooth and timeliness is poor: When data flows across systems are not smooth, the timeliness and effectiveness of data will be affected. For example, when data cannot be transmitted in real time or is not processed in time, it may hinder the progress of key business processes.

[0022] The above-mentioned existing technical problems have caused enterprises to face bottlenecks in data processing. It is difficult for data within the enterprise to flow smoothly and collaboratively between different systems, making it difficult to support efficient business process management.

[0023] In view of the problems existing in the prior art, this application proposes a distributed data flow processing method. The technical solution of this application is aimed at the field of data flow integration of manufacturing enterprises, especially the distributed data processing process across heterogeneous system boundaries. Through distributed data flow integration, the concept of system boundaries is eliminated, and a super process or a global process is formed to manage and coordinate the existing process, and the process can be reorganized and optimized on this basis. Compared with the currently popular open source data processing and process tools, there are two technical directions, but neither can meet the above data flow integration needs well. One is workflow tools, such as flowable, activi ti, etc., which focus on process management and do not have the required data processing capabilities; the other is ETL tools, such as kettl e, datax, etc., which focus on data processing, have weak process management capabilities, and lack the ability to collaborate between distributed processes. This technical solution introduces a distributed data flow integration method, which effectively manages data transmission, data processing, process integration, distributed collaboration and other links, combines data processing capabilities and process integration capabilities into one, and can effectively solve the problem of data flow processing between systems for intelligent manufacturing enterprises.

[0024] For example, in some examples, taking the production order execution in the discrete industry as an example, its typical data processing process is summarized as follows: 1. The MES system sends the production order to the control system and sends the corresponding delivery order to the WMS system; 2. The WMS system prepares materials and ships them out, and reports the material preparation status and delivery details to the MES system; 3. The MES system monitors the material preparation status of the WMS system and the ready status of the control system, and starts the production process if the conditions are met; 4. During the production process, the control system reports the order execution status and execution details to the MES system; 5. After generation is completed, the control system reports the completion status to the MES system, and the MES system reports the order execution statistics to the ERP system.

[0025] In the above example, different systems have different architectures and data definitions, and data transmission across systems requires conversion and processing; starting the production process requires the coordination of the WMS material preparation process and the control system preparation process; completing the entire order production execution requires crossing multiple information systems, coordinating multiple processes, and processing and transmitting data between systems and processes.

[0026] Before describing the technical solution of the present application in detail, the architecture of the distributed data flow processing system involved in the technical solution of the present application in actual scenarios is first described in detail with reference to the accompanying drawings and specific embodiments. Figure 1 It is a schematic diagram of the overall architecture of the distributed data flow processing system provided in an embodiment of the present application.

[0027] like Figure 1 As shown, the distributed data flow processing system may specifically include the following functional modules:

[0028] 1. Interface module

[0029] The interface module provides a data communication infrastructure to eliminate protocol differences and solve data flow barriers between various interface protocols (such as TCP, UDP, http, JDBC, WebService, FTP, etc.) between heterogeneous systems or businesses. The interface module consists of a set of high-level data access interfaces and a series of drivers. The data access interface defines common data read / write methods and rules, and the driver handles the communication of different protocols. The driver supports plug-in mode and can be plugged in and out as a plug-in, which facilitates driver expansion to support more communication methods.

[0030] 2. Event Response Module

[0031] The event response module receives and responds to system boundary events, converting non-reproducible external system events into internal instructions. First, the event response module decouples event response from event processing (business processing of data), improving the system's responsiveness while making the system more stable and reliable. Second, the use of internal instructions converts non-reproducible external events into reproducible internal processing, providing a redo mechanism for subsequent processing steps of the event after an error.

[0032] The event response module maintains an event response instruction queue to store instructions converted from event responses, which are executed by the process processing module.

[0033] 3. Process processing module

[0034] The process processing module has three main tasks: one is to process the instructions generated by the system boundary event response; the second is to process the subsequent steps of the instructions; the third is to generate system boundary events for the external system (corresponding to three major categories of processing steps: event response class, data processing class, and event initiation class).

[0035] The above three tasks constitute the steps of process processing. These steps are defined in advance by the process and are executed sequentially according to the process definition during event response and process processing.

[0036] The event-initiating steps are not executed immediately, but generate an instruction and put it into the event generation instruction queue. These instructions are executed by another process or time slice of the process processing module.

[0037] Taking the receiving and forwarding of material picking orders as an example, after receiving the data push event from MES, the event response module generates a material picking order receiving instruction, which is executed by the process processing module. The process processing module parses the instruction and performs data processing according to the process definition (converting the MES format to the WMS format); after the data processing is completed, the data is pushed to the WMS according to the process definition. Here, the event generation and WMS system processing are decoupled. Data is not pushed directly to the WMS, but a data push instruction (carrying the converted data and the WMS data receiving interface) is generated and executed by another process or time slice of the process processing.

[0038] 4. Process collaboration module

[0039] The process collaboration module is used for cross-process scheduling and collaboration. It manages a process running status table and manages related processes according to the process running status. The process status includes: ready, executing, successful, failed, and retry. Because all processes are triggered by events, the process changes to the continuing state after the event response module responds to the system boundary event and generates instructions; during the processing of instructions and the subsequent steps of the process where the instructions are located, the process is in the execution state; all processing steps defined in the process are successful, and the process is in the success state; if there are processing steps in the process that fail to execute, the process is in the failure state. If there are processing steps configured for failure retry, the process is in the retry state before the number of retries is completed, and the final state after the retry is completed is success or failure.

[0040] The process coordination module coordinates the order, parallelism, and conditional execution relationships among multiple processes according to the above process status. The result of the coordination between processes is nothing more than starting one process or not starting another process. There are two ways for the process coordination module to start a process: for internal processes, directly execute its processing steps; for external processes (i.e. distributed processes across system boundaries), use event initiation to generate an instruction for the process management module to execute.

[0041] 5. System Boundary Event Storage Module

[0042] Used to record system boundary events during system operation. System boundary events can be divided into two categories according to their direction: incoming events (events triggered by external systems) and outgoing events (events triggered to external systems). The original data of these events and the corresponding instructions of the events must be recorded in order to achieve the decoupling of events and process processing, the basic breakpoint retention and recovery of system abnormal exits, and the retry of failed steps in cross-system boundary processes. The system boundary event storage module is also the basis for simulating system boundary events / instructions, debugging and diagnosing cross-system boundary interfaces during system debugging.

[0043] Based on the architecture of the above-mentioned distributed data flow processing system, the distributed data flow and scenario examples involved in the technical solution of this application are described in detail below, which may specifically include the following contents:

[0044] 1. Distributed data flow:

[0045] The data flow process often needs to cross system or business boundaries, which will cause obstacles to data transmission. Such obstacles may be reflected in differences in data definition, data format, communication protocol, usage, etc. Traditional data transmission between different databases, between SOA interfaces, between file services, and between microservices that have become popular in recent years all belong to the distributed data flow process. The distributed data flow in this embodiment refers to the data processing process across the above-mentioned system or business boundaries. Usually, a distributed data flow crosses at least one system or business boundary, and can also cross multiple systems or business boundaries.

[0046] In actual use, there are situations where multiple data flows cooperate and collaborate to complete a task. The collaborative relationship between flows includes sequence (one flow must be executed after another flow succeeds), parallel (multiple flows can be executed at the same time), conditional selection (determine which flow to execute based on a certain event status), etc. A special type of flow relationship is sub-flow. A main flow can contain multiple sub-flows, and a sub-flow can be executed repeatedly multiple times.

[0047] The execution of data flow is driven by events. There are many types of events, such as system timing events, manual trigger events, automatic condition events, etc.

[0048] 2. Scenario example:

[0049] Taking the order execution process as an example, MES sends the production order to the control system and sends the delivery order to the WMS system. These are two processes that can be executed in parallel. WMS prepares materials for delivery and MES executes production. These are also two processes that must be executed sequentially. If there are subtasks in the production order processing, such as a batch of drugs packed according to standards, the reporting process of each box of data is a repeatable subprocess in the production order execution, and this subprocess will be repeated many times during the order execution process.

[0050] In the event-driven level of the above examples, for example, when an MES worker issues an order, it is a manually triggered event; when the WMS system automatically reports the completion status of the outbound shipment, it is a conditional event automatically triggered by the system; if the MES periodically checks whether the outbound shipment is completed, it is a system timed event.

[0051] In summary, the data processing process consists of the data to be processed, the data flow process, and the events that drive the process execution. Event processing across system or business boundaries is the key to ensuring the smooth execution of the process. System boundary events are generated by manual, external systems, or automatic conditions. Many events are not repeatable, so it is necessary to ensure that each system boundary event is responded to in a timely manner and handled correctly.

[0052] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Figure 2 Schematic diagram of the distributed data flow processing method provided in the embodiment of the present application. Figure 2 As shown, the distributed data flow processing method may specifically include:

[0054] S201, define the data structure, data processing requirements and data flow process for each data flow, and define a distributed data processing interface in the data flow process;

[0055] S202, establishing an event response mechanism based on the data flow, the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue;

[0056] S203, taking out a processing instruction from the instruction queue according to a predetermined strategy, and executing the processing instruction according to the data flow, so as to process the data or initiate the next system boundary event;

[0057] S204, establishing a process status management mechanism according to the predefined process collaboration relationship, the process status management mechanism is used to monitor the data flow status and perform collaborative management of the data flow according to the data flow status.

[0058] In some embodiments, a data structure, data processing requirements, and data flow process are defined for each data flow, and a distributed data processing interface is defined in the data flow process, including:

[0059] Determine the data structure for each data flow. The data structure is used to standardize the organization of data content.

[0060] Define the processing steps required when data is transferred between different systems according to the needs of each data process;

[0061] The data flow process between different systems is defined. The data flow process includes a distributed data processing interface. The distributed data processing interface is used to manage the data flow across system boundaries and to process system boundary events.

[0062] Specifically, in order to achieve cross-system management and processing of data flows, first define the data structure, data processing requirements, and data flow process for each data flow, and define a distributed data processing interface during the data flow process to support the processing of system boundary events and cross-system data flow. In some examples, the definition of a data flow may include the following aspects:

[0063] Definition of data structure: Determine the data structure for each data process. The data structure is used to standardize the organization of data content and ensure the uniformity of data transmission between different systems. For example, in the "receiving and forwarding material order" process, the data structure contains the material order number, material ID, quantity, source system, target system and other fields to ensure the standardization of data content between the MES system and the WMS system.

[0064] Definition of data processing requirements: Based on the requirements of each data flow, define the processing steps required when data is transferred across systems. For the processing requirements of the picking list, the data processing steps include: converting the picking list information from the format of the MES system to the standard format of the WMS system to ensure that the target system can recognize and use the received data.

[0065] Definition of the data flow process: The data flow process defines the data transfer path and related interfaces between different systems, and includes distributed data processing interfaces to support data flow management across system boundaries and the processing of system boundary events. In this process, the core of data flow is the calling and management of distributed data processing interfaces.

[0066] Definition of distributed data processing interface: In the data flow across system boundaries, a distributed data processing interface is defined to initiate and respond to events in the data flow between systems. For example, the MES system actively initiates a data push event to push the material picking list data to the distributed interface. After receiving the data, the interface converts the format and sends the data to the WMS system, triggering a data reception event in the WMS system.

[0067] For example, in an example scenario, taking the "receiving and forwarding material picking lists" process as an example, the specific flow of data is described. First, the MES system generates a material picking list based on production needs, and pushes the material picking list information to the WMS system through a distributed data processing interface. The MES system calls the system interface to push the material picking list data. After receiving the data, the interface performs data conversion processing, generates corresponding processing instructions, and triggers data push events through the event response mechanism. Subsequently, after the data conversion is completed, the interface in the data flow process calls the receiving interface of the WMS system to forward the material picking list data to the WMS system, and the WMS system receives and processes the material picking list data.

[0068] Through the definition of the above process and the management of the interface, this embodiment realizes the cross-system data flow process between the MES system and the WMS system, effectively supports the triggering and response of system boundary events, and ensures the smooth transmission of data between different systems. The design of this distributed data processing interface is the core feature of the cross-system boundary data flow integration.

[0069] In some embodiments, the event response mechanism is used to respond to the system boundary event, generate a corresponding processing instruction, and add the processing instruction to the instruction queue, including:

[0070] When a system boundary event occurs, the system boundary event is matched with a predetermined data flow to determine the corresponding data flow;

[0071] Use data flow to respond to system boundary events and record logs of system boundary events;

[0072] According to the type of system boundary event and the definition of data flow, the system boundary event is converted into an internal processing instruction, and the processing instruction is added to the instruction queue.

[0073] Specifically, the system boundary event response mechanism of the present application is used to respond to system boundary events in a timely manner, generate corresponding processing instructions, and add the instructions to the instruction queue for execution. This event response mechanism is suitable for processing various event types such as system timing events, manually triggered events, and automatic condition events. In some examples, the implementation process of the event response mechanism is as follows:

[0074] First, when a system boundary event occurs, the event is first matched with the predetermined data flow to determine the data flow corresponding to the event. For example, after the MES system initiates a pick order push event, the event will be matched to the predefined data flow for picking order reception and forwarding.

[0075] Furthermore, after matching the corresponding data flow, the event response mechanism responds to the system boundary event and immediately records the log of the system boundary event. The log contains information such as the time when the event occurred, the event type, and the data content, so as to facilitate subsequent event tracking and process monitoring. For example, when the MES system initiates a pick order push event, the system records the time of the event, the pick order number, and related content in the log.

[0076] Furthermore, according to the type of system boundary event and the data flow definition, the system boundary event is converted into an internal processing instruction. The processing instruction usually contains specific data content and the direction of the subsequent steps for execution in the process. For example, for the material picking list receiving and forwarding process, when the push event of the MES system is received, the system generates a "material picking list receiving instruction" and adds it to the instruction queue, waiting for further processing steps.

[0077] Furthermore, in this mechanism, the event response process is decoupled from the actual data processing process to ensure fast response and high throughput of system boundary events. That is, while responding to an event, the system puts the generated instructions into a queue, and the subsequent process processes the instructions to avoid delays in the event response process. For example, after receiving the pick order push event from the MES system, the system immediately generates a pick order receiving instruction and adds it to the queue, and the specific instruction processing will be executed later in the order of the instruction queue.

[0078] Furthermore, the generated processing instructions are instances of system boundary events, with the actual data content of the event attached, and include associations with the data flow. The instruction points to the next data processing step or event initiation step, ensuring that events across system boundaries can flow smoothly. Since system boundary events may not be repeatable, by converting them into internal processing instructions, the execution of events can be localized, thereby achieving critical fault-tolerant processing. For example, the receiving instruction of the material picking list includes a format conversion task from MES to WMS. After the data conversion is completed, a new data push event will be triggered to WMS to ensure the correct connection of the entire process.

[0079] For example, in the "material picking list receiving and forwarding" scenario in the above example, the MES system actively pushes the material picking list data to the interface. After the system receives the event, it matches the corresponding data flow, immediately records the event log and generates a material picking list receiving instruction, and puts it into the instruction queue. Subsequently, the system takes out the instruction according to the predetermined strategy and converts the data format. After the processing is completed, the system initiates a new push event to the WMS system to complete the cross-system transmission of the material picking list.

[0080] In some embodiments, taking out a processing instruction from an instruction queue according to a predetermined strategy and executing the processing instruction according to a data flow includes:

[0081] During the process of process processing, the instruction queue is checked periodically. If there are processing instructions to be executed in the instruction queue, the processing instructions are taken out according to a predetermined priority or sequence strategy, and the processing instructions are executed according to the data flow corresponding to the processing instructions;

[0082] During the execution of the processing instruction, if the processing instruction contains data processing tasks, data processing is performed in the current data flow; if the processing instruction contains event initiation tasks, a new processing instruction is generated and added to the instruction queue.

[0083] Specifically, processing instructions are taken from the instruction queue according to the predetermined strategy, and the processing instructions are executed according to the data flow to realize cross-system data processing and event initiation. This process processing mechanism is suitable for processing event response, data processing and event initiation instructions, ensuring that the system can execute various tasks according to the predetermined strategy and guaranteeing the timeliness and correctness of data flow.

[0084] First, during the process of process processing, the system will periodically check the instruction queue to determine whether there are processing instructions to be executed. If there are instructions to be executed in the instruction queue, the instructions will be taken out according to the predetermined priority or sequence strategy. The predetermined strategy can be set according to the urgency of the task, system resource conditions or process dependencies to achieve reasonable ordering of instructions.

[0085] Furthermore, after taking out the instruction, the system executes the processing task according to the data flow corresponding to the instruction. The execution of the instruction includes the operation of the instruction itself and the subsequent processing steps of the flow in which the instruction is located to ensure the integrity of the task. For example, if the instruction taken out is a "material picking list receiving instruction", the system will perform format conversion processing according to the instruction and prepare for the execution of subsequent steps.

[0086] Furthermore, during the execution of the instruction, if the instruction contains a data processing task, the system performs the data processing operation in the current data flow. For example, in the "receiving and forwarding the material picking list" scenario, the data processing task includes converting the material picking list data from the format of the MES system to the format of the WMS system. The system completes the format conversion operation during the execution of the instruction and saves the processed data in memory or temporary storage for subsequent steps.

[0087] Furthermore, during the execution of the instruction, if the instruction contains an event-initiating task, the system generates a new processing instruction and adds the instruction to the instruction queue for execution. For example, after completing the format conversion of the material picking list, the system generates a "data push instruction" that contains the converted data and the data receiving interface of the WMS system. The system adds the "data push instruction" to the instruction queue and waits for the next available process or time slice to execute.

[0088] Furthermore, in this embodiment, instructions can be divided into three categories, namely, event response, data processing, and event initiation. Event response instructions are derived from the system boundary event response mechanism, generated when the system receives a boundary event, and added to the instruction queue; data processing instructions perform specific data operations in the current data flow; event initiation instructions trigger new system events, allowing the process to be executed across systems. For example, when the MES system initiates a pick order push event, the system generates an event response instruction, receives the pick order data, and adds it to the instruction queue. After that, the system takes the instruction out of the queue to perform data processing tasks, that is, data format conversion, and further generates an event initiation class "data push instruction" to push the processed pick order to the WMS system.

[0089] Furthermore, in the management of the instruction queue, the system decouples event response from specific processing tasks, realizing the ability to process tasks in parallel. Event response is only responsible for generating and adding instructions, and the specific processing process is executed sequentially in the queue according to priority. This mechanism ensures high throughput of instructions and avoids the timeliness of event response affected by the complexity of instructions.

[0090] For example, in the "material order receiving and forwarding" scenario in the above example, after the system receives the material order push event from the MES system, it generates a "material order receiving instruction" and puts it into the queue. The system then takes out the "material order receiving instruction" according to the priority strategy, and completes the processing task (data processing task) of converting the format from MES to WMS during execution. After the conversion is completed, the system generates a "data push instruction" and puts it into the queue to initiate a push event for the format-converted data through the WMS interface to complete the data flow across systems.

[0091] In some embodiments, a process state management mechanism is established according to a predefined process collaboration relationship, including:

[0092] In the definition of data flow, the process collaboration relationship between multiple data flows is set, where the process collaboration relationship includes sequential execution, parallel execution and conditional selection;

[0093] Based on the process collaboration relationship, a process status management mechanism is established, where the process status management mechanism is used to monitor the execution status of each data process in real time and record the status changes of the data process, so as to collaboratively manage the data process according to the status changes.

[0094] Specifically, this embodiment can also establish a process state management mechanism based on the predefined process collaboration relationship to monitor and manage the execution status of the data flow in real time, thereby realizing collaborative management between data flows. The process state management mechanism is applicable to a variety of collaborative relationships, such as sequential execution, parallel execution, and conditional selection, ensuring that different data flows can trigger corresponding subsequent operations in a timely manner when the state changes.

[0095] First, in the process of defining data flow, set the collaborative relationship between multiple data flows. The collaborative relationship can include sequential execution, parallel execution and conditional selection. For example, set the "Get department data" process and the "Get personnel data" process as a sequential relationship, that is, execute department data acquisition first, then execute personnel data acquisition; set the "Order distribution process" and the "Material collection process" as a parallel relationship, that is, order distribution can send order information to multiple systems at the same time.

[0096] Furthermore, based on the above-mentioned collaborative relationship, a process status management mechanism is established to monitor the real-time execution status of each data process. The status management mechanism records the state changes of the process, including the start, execution, and completion of the process, to support cross-process collaborative management. For example, after the status management mechanism monitors the completion status of the "obtain department data" process in real time, it immediately passes the status information to the "obtain personnel data" process, thereby triggering the acquisition process of personnel data.

[0097] Furthermore, the process status management mechanism can record the status changes of each process in real time and store the process status records in the log for subsequent monitoring, tracking and backtracking. This not only ensures the orderliness of each process execution, but also enables exception handling. For example, in the "feeding process", the system monitors the "material verification" status in real time; if the verification passes, the process status changes to "feeding started"; if the verification fails, the process status management mechanism will change the process status to "starting the return process" and trigger the corresponding return processing operation.

[0098] Furthermore, when the process status changes, the process status management mechanism automatically executes cross-process event response and management according to the predefined collaborative relationship. If a process needs to trigger the next process after completion, the system will automatically generate event response instructions and put them into the instruction queue for execution according to priority order or conditional dependency. For example, when the "production order reception" process is completed, the status management mechanism distributes the order information to the MES system and WMS system at the same time according to the parallel collaborative relationship, realizing synchronous data processing between multiple systems.

[0099] Furthermore, the process status management mechanism supports conditional selection control. According to the real-time state changes of the process and specific condition judgments, the system can selectively start different subsequent processes. For example, in the "material verification" process, the status management mechanism chooses to start the material feeding or material return process according to the verification results. When the material verification passes, the material feeding process is triggered; if the verification fails, the material return process is triggered.

[0100] For example, in a sample scenario, taking the "production order processing" scenario as an example, the order receiving, verification and distribution processes work together. The specific contents of the collaborative management between data processes based on the process state management mechanism are as follows:

[0101] Sequential collaboration: First, the "order receiving" process is executed to load the order data into the system; then, after the status management mechanism detects that the order receiving process is completed, the "order verification" process is triggered to verify the order content.

[0102] Parallel collaboration: When an order passes verification, the status management mechanism sends the order information to the MES system and WMS system at the same time according to the preset parallel collaboration relationship, so that production planning and material preparation can be carried out in parallel, thereby speeding up order processing efficiency.

[0103] Conditional control collaboration: In the material preparation stage, if the status management mechanism detects that the material has passed the verification, the material feeding process will be triggered; if it detects that the verification has failed, the material return process will be triggered to ensure the accuracy and timeliness of the materials in the production process.

[0104] Through the method of the above embodiment, the process state management mechanism realizes real-time state monitoring and collaborative management across processes, and triggers different event responses and task executions according to state changes, thereby ensuring efficient operation of the system.

[0105] In some embodiments, the method further comprises:

[0106] Under the action of the process state management mechanism, event responses between related data processes are triggered according to the current data process state, where the event response includes generating new processing instructions and adding them to the instruction queue;

[0107] According to the status changes of the process collaboration relationship, the corresponding data processing tasks or event initiation operations are executed to trigger new data processes or perform cross-process data transmission when the preset conditions are met.

[0108] Specifically, under the action of the process state management mechanism, this embodiment automatically triggers the event response of the associated data process according to the state change of the current data process. The event response includes generating new processing instructions and adding them to the instruction queue, thereby executing data processing tasks or event initiation operations according to the process collaboration relationship, so as to trigger a new data process or realize cross-process transmission of data between different processes when the preset conditions are met.

[0109] First, when the process status management mechanism detects a change in the status of the current data process, it triggers the event response of the associated data process according to the predefined process collaboration relationship. For example, when the "order receiving" process is completed, the process status management mechanism detects that the process status has changed to "completed", and then triggers the subsequent "order verification" process to pass the order data to the next processing stage.

[0110] Furthermore, after the state change triggers an event response, the system generates a corresponding processing instruction based on the event and adds the instruction to the instruction queue so that it can be processed according to the queue priority order or preset strategy. The generated processing instructions may include data processing tasks, event initiation tasks, etc. to ensure the smooth execution of subsequent processes. For example, when the "order verification" process is completed, the system generates a "material preparation" instruction and adds it to the instruction queue for execution.

[0111] Furthermore, when the state changes, the process state management mechanism transfers data across processes according to the preset collaborative relationship. In a parallel collaborative scenario, state changes may trigger multiple associated processes. For example, after the "production order" process is completed, the process state management mechanism detects this state and generates two parallel processing instructions, one pointing to the "production preparation" process and the other pointing to the "material preparation" process, passing the order information to both processes at the same time to speed up the processing progress.

[0112] Furthermore, when the process status management mechanism detects that specific conditions are met, different subsequent data processes are triggered. Specifically, the conditional control process performs different tasks according to the current process status and the set conditions. For example, in the "material verification" process, if the material status verification passes, a "feed" instruction is generated and added to the instruction queue; if the verification fails, a "return material" instruction is generated and added to the instruction queue to ensure the accuracy of the material flow.

[0113] For example, in the "production order processing" scenario example, the system implements event response and cross-process data transmission based on the process state management mechanism, including:

[0114] Sequential execution trigger: The system first executes the "Order Receiving" process, and after it is completed, generates the "Order Verification" instruction, and passes the order data to the "Order Verification" process. At this time, the state change of the "Order Receiving" process triggers the start of the "Order Verification" process, forming a sequential execution collaboration relationship.

[0115] Parallel execution trigger: After the "order verification" process is completed, the state management mechanism generates two instructions based on the parallel collaboration relationship. One instruction starts the "production preparation" process, and the other instruction starts the "material preparation" process. At this time, the order information is passed to multiple processes, realizing parallel processing of orders.

[0116] Conditional selection trigger: In the "material verification" process, if the system detects that the material meets the verification conditions, the process status management mechanism generates a "feed" instruction and adds it to the queue; if the material does not meet the verification conditions, a "return" instruction is generated to implement branch operations based on status conditions.

[0117] Through the process state management mechanism, this method can realize event response and cross-process data transmission according to the state changes of the data process, thereby supporting the collaborative management of complex data processes and improving the processing efficiency of the system.

[0118] In some embodiments, after the data flow is executed, the method further includes:

[0119] Track and record the execution status, event records, and data processing steps of a single data process, multiple collaborative processes, and data-related processes to monitor and trace the data process;

[0120] Among them, tracking of a single data flow includes tracking the execution status of each data flow to record the complete execution process of the data flow;

[0121] Tracking of multiple collaborative processes includes visualizing related data flows to facilitate correlation viewing and status monitoring between related data flows;

[0122] Tracking of data-related processes includes tracking the data processing records of multiple related processes with the data flow as the center, so as to trace back the entire process of data flow.

[0123] Specifically, after the data flow is executed, the method also includes tracking and recording the execution status, event records and data processing steps of a single data flow, multiple collaborative processes and data-related processes, so as to monitor and trace the execution of the data flow, thereby ensuring the traceability of the data flow process and efficient management of the process.

[0124] First, each individual data flow is tracked, including recording its execution status, system boundary events, and internal processing steps. By tracking the execution status of each data flow, the system can record the complete execution process of the process and capture key events in the data flow process. For example, in the "Order Verification" process, the system will track the start, in progress, and completion status of the process, and record detailed data for each processing step, including the triggered system boundary events (such as data reception, data sending) and related instructions.

[0125] Furthermore, the collaborative relationship between multiple related data processes is tracked so that users can view the associated status of each process during monitoring and management. The tracking of collaborative processes is visualized through flowcharts, which display each related process in the form of a flowchart connection, and mark the execution status, event records and data processing steps of the process, so that operators can intuitively view the relationship between processes and data flow. For example, in the "production order" processing scenario, the flowchart can display the sequential and parallel execution of multiple processes such as "order reception", "order verification", "material preparation" and "production preparation", supporting cross-process status monitoring and real-time adjustment.

[0126] Furthermore, with data flow as the center, the related data processing steps across processes are tracked so that the entire data flow process can be traced back when needed later. This data dimension tracking method allows the system to use specific data (such as orders, batches, etc.) as clues to associate data processing records in each process to form a complete execution path. For example, the system can use a production order as a clue to trace the entire process from order issuance, order verification, material preparation to order status reporting, and record the processing details and related data in each process for easy analysis and backtracking.

[0127] In some examples, we take the scenario of tracking data flow based on orders as a clue. For example, the system tracks the flow of an order, and the specific implementation process is as follows:

[0128] Order issuing process: The system records the status of orders issued from the ERP system to the MES system, and records the issuing time and receiving status.

[0129] Order status reporting process: After the MES system completes order processing, it returns the order status to the ERP system. The status record includes the order completion status, generated events and related data.

[0130] Order execution sub-process: Track the specific execution status of the order (such as production progress, material usage), and record the processing status and related data of each subtask.

[0131] Order reporting process: When the order is executed, the system will report the execution status to the ERP system, record the data in the reporting process, and generate a tracking report for the entire order process.

[0132] Through the tracking and recording method of the above embodiment, the system can provide comprehensive traceability from a single process to multiple collaborative processes and then to the data dimension, forming a retrospective record covering all data processes and events, thereby providing comprehensive support for the monitoring, analysis and optimization of the data flow process.

[0133] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0134] Figure 3 Schematic diagram of the structure of the distributed data flow processing device provided in the embodiment of the present application. Figure 3 As shown, the distributed data flow processing device includes:

[0135] A definition module 301 is configured to define a data structure, data processing requirements and a data flow process for each data flow, and define a distributed data processing interface in the data flow process;

[0136] An establishing module 302 is configured to establish an event response mechanism based on the data flow, the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue;

[0137] The execution module 303 is configured to take out the processing instruction from the instruction queue according to a predetermined strategy, and execute the processing instruction according to the data flow, so as to process the data or initiate the next system boundary event;

[0138] The management module 304 is configured to establish a process state management mechanism according to the predefined process collaboration relationship. The process state management mechanism is used to monitor the data flow state and perform collaborative management of the data flow according to the data flow state.

[0139] In some embodiments, Figure 3 The definition module 301 determines the data structure for each data flow, and the data structure is used to standardize the organization form of the data content; according to the needs of each data flow, it defines the processing steps required for data transmission between different systems; defines the data flow process between different systems, and the data flow process includes a distributed data processing interface. The distributed data processing interface is used to manage the data flow when crossing the system boundary and is used to process system boundary events.

[0140] In some embodiments, Figure 3 When a system boundary event occurs, the establishment module 302 matches the system boundary event with a predetermined data flow to determine the corresponding data flow; responds to the system boundary event using the data flow and records the log of the system boundary event; converts the system boundary event into an internal processing instruction according to the type of the system boundary event and the definition of the data flow, and adds the processing instruction to the instruction queue.

[0141] In some embodiments, Figure 3 The execution module 303 periodically checks the instruction queue during the process processing. If there are processing instructions to be executed in the instruction queue, the processing instructions are taken out according to a predetermined priority or sequence strategy, and the processing instructions are executed according to the data flow corresponding to the processing instructions. During the execution of the processing instructions, if the processing instructions contain data processing tasks, data processing is performed in the current data flow; if the processing instructions contain event initiation tasks, new processing instructions are generated and added to the instruction queue.

[0142] In some embodiments, Figure 3The management module 304 sets the process collaboration relationship between multiple data processes in the definition of the data flow, wherein the process collaboration relationship includes sequential execution, parallel execution and conditional selection; according to the process collaboration relationship, a process status management mechanism is established, wherein the process status management mechanism is used to monitor the execution status of each data flow in real time, and record the status changes of the data flow, so as to coordinate the management of the data flow according to the status changes.

[0143] In some embodiments, Figure 3 The management module 304, under the action of the process status management mechanism, triggers event responses between related data processes according to the current data flow status, where the event response includes generating new processing instructions and adding them to the instruction queue; according to the state changes of the process collaboration relationship, executes corresponding data processing tasks or event initiation operations, so as to trigger new data flows or transfer data across processes when the preset conditions are met.

[0144] In some embodiments, Figure 3 After the data flow is executed, the tracking module 305 tracks and records the execution status, event records and data processing steps of a single data flow, multiple collaborative processes and data-related processes to monitor and backtrack the data flow; wherein, tracking of a single data flow includes tracking the execution status of each data flow to record the complete execution process of the data flow; tracking of multiple collaborative processes includes displaying related data flows in a visual manner to facilitate association viewing and status monitoring between related data flows; tracking of data-related processes includes tracking the data processing records of multiple related processes centered on the data flow to backtrack the entire process of data flow.

[0145] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0146] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0147] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.

[0148] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0149] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0150] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0152] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A distributed data flow processing method, characterized in that: include: Define the data structure, data processing requirements and data flow process for each data flow, and define a distributed data processing interface in the data flow process; Establishing an event response mechanism based on the data flow, wherein the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue; Taking out the processing instruction from the instruction queue according to a predetermined strategy, and executing the processing instruction according to the data flow, so as to process the data or initiate the next system boundary event; A process state management mechanism is established according to a predefined process collaboration relationship, and the process state management mechanism is used to monitor the data flow state and perform collaborative management of the data flow according to the data flow state.

2. The method according to claim 1, characterized in that Defining the data structure, data processing requirements and data flow process for each data flow, and defining a distributed data processing interface in the data flow process, includes: Determine a data structure for each of the data flows, wherein the data structure is used to standardize the organization form of the data content; Define the processing steps required when data is transferred between different systems according to the requirements of each of the data flows; The data flow process between different systems is defined, and the data flow process includes a distributed data processing interface, which is used to manage the data flow when crossing system boundaries and to process system boundary events.

3. The method according to claim 1, characterized in that The event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to the instruction queue, including: When the system boundary event occurs, matching the system boundary event with a predetermined data flow to determine a corresponding data flow; Responding to the system boundary event using the data flow and recording a log of the system boundary event; According to the type of the system boundary event and the definition of the data flow, the system boundary event is converted into an internal processing instruction, and the processing instruction is added to the instruction queue.

4. The method according to claim 1, characterized in that: The taking out the processing instruction from the instruction queue according to a predetermined strategy and executing the processing instruction according to the data flow includes: During the process of process processing, the instruction queue is periodically checked, and if there are processing instructions to be executed in the instruction queue, the processing instructions are taken out according to a predetermined priority or sequence strategy, and the processing instructions are executed according to the data flow corresponding to the processing instructions; During the execution of the processing instruction, if the processing instruction contains data processing tasks, data processing is performed in the current data flow; if the processing instruction contains event initiation tasks, a new processing instruction is generated and added to the instruction queue.

5. The method according to claim 1, characterized in that The process state management mechanism is established according to the predefined process collaboration relationship, including: In the definition of the data flow, a process collaboration relationship between multiple data flows is set, wherein the process collaboration relationship includes sequential execution, parallel execution and conditional selection; According to the process collaboration relationship, the process status management mechanism is established, wherein the process status management mechanism is used to monitor the execution status of each data process in real time and record the status changes of the data process so as to perform collaborative management of the data process according to the status changes.

6. The method according to claim 5, characterized in that The method further comprises: Under the action of the process state management mechanism, an event response between related data processes is triggered according to the current data process state, wherein the event response includes generating a new processing instruction and adding it to the instruction queue; According to the state changes of the process collaboration relationship, corresponding data processing tasks or event initiation operations are executed to trigger new data processes or perform cross-process data transfer when preset conditions are met.

7. The method according to claim 1, characterized in that After the data flow is executed, the method further includes: Track and record the execution status, event records, and data processing steps of a single data process, multiple collaborative processes, and data-related processes to monitor and trace the data process; The tracking of the single data flow includes tracking the execution status of each data flow to record the complete execution process of the data flow; Tracking the multiple collaborative processes includes displaying the relevant data processes in a visual manner to facilitate association viewing and status monitoring between the relevant data processes; The tracking of the data-related processes includes tracking the data processing records of multiple related processes with the data flow as the center, so as to trace back the entire process of data flow.

8. A distributed data flow processing device, characterized in that: include: A definition module, configured to define a data structure, data processing requirements and a data flow process for each data flow, and to define a distributed data processing interface in the data flow process; An establishing module is configured to establish an event response mechanism based on the data flow, wherein the event response mechanism is used to respond to system boundary events, generate corresponding processing instructions, and add the processing instructions to an instruction queue; an execution module, configured to take out the processing instruction from the instruction queue according to a predetermined strategy, and execute the processing instruction according to the data flow, so as to process the data or initiate the next system boundary event; The management module is configured to establish a process state management mechanism according to a predefined process collaboration relationship, wherein the process state management mechanism is used to monitor the data flow state and perform collaborative management of the data flow according to the data flow state.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images