Data processing method and device based on twin platform supporting multiple execution engines

Abstract

The application discloses a data processing method and device supporting multiple execution engines based on a twin platform, comprising the following steps: according to the token of each data element, pulling the data element result to the message queue of the twin platform through the data element interface of the data element transaction platform; calling a complex event processing module, triggering the associated business model based on the data element result in the message queue, performing corresponding operations based on the business model, and writing the execution result into the data asset library. The embodiment of the application can overcome the security and compliance problems of original data, and realize the scheduling and task execution supporting multiple execution engines based on the twin platform.

Description

Technical Field

[0001] This application relates to the field of data element technology, and in particular to a data processing method and device based on a twin platform that supports multiple execution engines. Background Technology

[0002] From a data security and compliance perspective, traditional systems directly connect to raw data, which poses a serious challenge to data security and the compliance of data sources. It cannot guarantee whether the raw data has been leaked through the system or whether the data source is compliant.

[0003] From the perspective of business model construction and execution, traditional systems are highly coupled from the business model presentation layer to the model execution layer. Changes to the model execution layer will lead to overall system-wide changes from top to bottom. Typically, a system's underlying model execution is a separate system. Since the execution of DAGs with large amounts of data currently incurs startup costs, regardless of the simplicity or complexity of the model or the amount of computation required, the startup costs of these frameworks are very high. Given the need to support both the rapid execution of simple business models based on small amounts of data and the complex computations of complex business models on large datasets, the demand for loosely coupling the model presentation layer to the execution layer while simultaneously supporting multiple execution engines is becoming increasingly strong. Summary of the Invention

[0004] This application provides a data processing method and device based on a twin platform that supports multiple execution engines, in order to overcome the security and compliance issues of raw data, and to realize the scheduling and task execution of multiple execution engines based on the twin platform.

[0005] This application provides a data processing method based on a twin platform that supports multiple execution engines, including:

[0006] Through the data element interface of the data element trading platform, the data element results are pulled into the message queue of the twin platform according to the token of each data element;

[0007] The complex event processing module is invoked, and based on the data element results in the message queue, the associated business model is triggered to perform corresponding operations based on the business model, and the execution results are written to the data asset library.

[0008] Optionally, the construction and execution process of the business model is implemented through a three-layer architecture consisting of a model presentation layer, a model abstraction layer, and a model execution layer, wherein the model presentation layer is configured as follows:

[0009] Obtain the associated entities involved in the target business corresponding to the data element results, and determine the transactions to be processed by the associated entities;

[0010] Provides a canvas for visually configuring the attribute information of the business model required by the target business, and establishes the connection relationship between the business model and related entities to form a directed graph; and

[0011] The target data fields in the technical metadata are pulled into the canvas, and the connection relationship between the target data fields and the attribute information of the corresponding business model is constructed to expand the directed graph.

[0012] Optionally, the model presentation layer uses the following data storage method:

[0013] Retrieve the model and layout information passed from the front end to the back end in JSON format;

[0014] According to the JSON structure, the data is stored in the model graph data table of the model presentation layer to record the node information of each first node in the directed graph and the relationship between the nodes. The node information includes node code, node name, node type, business attributes contained in the node, operators involved, conditions and expressions. The relationship between the nodes is recorded with the preceding node code of each node.

[0015] Optionally, the model abstraction layer is configured to convert the directed graph of the JSON structure stored in the model presentation layer into the structure of the abstraction layer, mapping the first node of the presentation layer to one or more second nodes in the abstraction layer; and

[0016] The business model is converted into a list of nodes and node contexts. Each node context corresponds to an operator in the model presentation layer node. Any node context consists of node input parameters and output parameters. The input parameters include the data tables and data fields involved, and the output parameters include the intermediate tables and data fields output.

[0017] Optionally, the model execution layer is configured as follows:

[0018] Create corresponding task nodes based on the content of the node context in the model abstraction layer;

[0019] Use a uniform type of task package to perform task scheduling, and execute tasks based on the created task nodes.

[0020] Optionally, executing tasks based on created task nodes includes executing the task nodes corresponding to the business model according to the triggering conditions of the business model to obtain the execution result.

[0021] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned data processing method based on a twin platform supporting multiple execution engines.

[0022] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned data processing method based on a twin platform supporting multiple execution engines.

[0023] The embodiments of this application can overcome the security and compliance issues of raw data, and realize the scheduling and task execution of multiple execution engines based on the twin platform.

[0024] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 This is an example of a three-tier architecture for constructing and executing a business model according to an embodiment of this application;

[0027] Figure 2 This is a specific example of the business model construction and execution of the three-tier architecture in this application embodiment. Detailed Implementation

[0028] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0029] This application provides a data processing method based on a twin platform that supports multiple execution engines, including:

[0030] The data processing method of this application integrates the results of data elements and constructs a main body 360-degree micro database based on the data elements. In this example, the original data is physically isolated in a data vault by a one-way shutter. The data element results are processed by the element processing center in the data vault and dynamically transferred to the data element library outside the shutter according to the authorization agreement. This mechanism overcomes the security and compliance issues of the original data.

[0031] Data elements are data features formed by modeling data based on a dataset of related fields or data-related fields, after data has been anonymized.

[0032] A data vault refers to a data storage and management facility that is regulated by the competent authority, standardized, independently controllable, secure, and reliable, and that stores core data, important data, sensitive data, and data elements. It is primarily deployed in governments, organizations, industries, and large enterprises. The vault mentioned in this application is located within an area physically isolated by optical shutters.

[0033] A data element processing center refers to a large-scale, fully automated data element processing production line that realizes the development and management of data elements throughout their entire lifecycle, from data collection to data element processing and trading.

[0034] "Ferrying" refers to sending the results of data elements to the area outside the optical gate through instructions via a specific protocol. The protocol here is UDP, which is a one-way protocol.

[0035] Data element results refer to the dataset after the data elements have been defined and approved, and then processed by the feature processing center.

[0036] After the data vault transfers the data element results to the data element library, in step S101, the data element results are retrieved to the message queue of the twin platform through the data element interface of the data element trading platform, based on the token of each data element. Specifically, for example, the data element results can be retrieved to the twin platform's message queue Kafka via HTTP using the token of each element.

[0037] In step S102, the complex event processing module is invoked, and based on the data element results in the message queue, the associated business model is triggered to perform corresponding operations based on the business model, and the execution results are written into the data asset library.

[0038] The Flink twin platform receives data element results by listening to a specified Kafka topic. It performs sink operations through two channels: one sinks to the twin platform's data repository, currently using MySQL; the other invokes the CEP (Complex Event Processing) module to trigger related business models based on the received data element results. In this example, the twin platform refers to a capability platform upon which users can build data ontology models, supporting rapid model creation and dynamic combination, as well as model expansion and customized inference. The data repository refers to the database containing the data objects that the model execution layer can manipulate; currently, data elements are stored in this repository through Flink's sink operations.

[0039] An exemplary implementation of a CEP module-triggered business model is as follows:

[0040] Define the event triggering conditions, such as: when the rule expression engine in the CEP module detects that the value of the epidemic-related xx detection field (xxjc) in the person element result data is positive, it will trigger the execution of the corresponding transmission chain model.

[0041] When $person.xxjc = 'positive'

[0042] Then

[0043] The code that triggers the execution of the corresponding propagation chain model

[0044] End

[0045] Once a specified business model is triggered, it will execute an operation, and the execution result will be written to the data asset repository, which is the 360-degree micro-database that constructs the main body. The construction and execution of the business model, and how business personnel use this system to twinnize the physical world's business processes, are the focus of this application. How the execution result is used to construct the main body's 360-degree micro-database will be accomplished using existing methods and will not be described in detail here, but only as a prerequisite. The main body's 360-degree micro-database and its corresponding database table fields form technical metadata, i.e., a technical metadata graph.

[0046] The embodiments of this application can overcome the security and compliance issues of raw data, and realize the scheduling and task execution of multiple execution engines based on the twin platform.

[0047] In some embodiments, the construction and execution process of the business model is supported by a three-layer architecture consisting of a model presentation layer, a model abstraction layer, and a model execution layer to achieve loose coupling between the layers. Figure 1 , Figure 2 As shown, the model presentation layer is configured as follows:

[0048] Obtain the associated entities involved in the target business corresponding to the data element result, and determine the transactions to be processed by the associated entities.

[0049] Provide a canvas to visually configure the attribute information of the business model required by the target business, and establish the connection relationship between the business model and the associated subject to form a directed graph;

[0050] In this example, the model presentation layer supports the construction of business processes through a graphical interface using drag and drop. For each node in the business process model, business personnel specify the ontologies involved in that node and the operations to be performed on those ontologies.

[0051] This example uses the scenario of handling business Q through the OneNet platform to illustrate the process. Handling business Q involves three commissioned bureaus: Bureau A, Bureau B, and Bureau C. Each bureau handles different matters, and there may be overlap. Bureau A's related matters are matters a and b, while Bureau B's related matters are matters b and c. Item b requires joint approval from both Bureau A and Bureau B. Each matter is associated with relevant certificates and forms. For example, matter b requires two certificates: Certificate E and License F. The associated forms are the relevant security personnel registration form and the relevant security equipment registration form.

[0052] This solution provides a canvas approach that allows business personnel to configure business models visually, which means configuring business metadata. Each business model will then be associated with technical metadata, which are the specific database table fields corresponding to certificates and business forms.

[0053] Taking the configuration process of the business model corresponding to business Q as an example, the specific business metadata construction adopts the following process: Drag three commissions / bureaus from the ontology category and name them Bureau A, Bureau B, and Bureau C respectively. Then, based on the provided canvas, drag out items and name them Item a, Item b, and Item c. Drag out licenses and corresponding forms in the same way. Associate corresponding attributes with the licenses and forms, such as License F in the license category, and personnel-related information associated with the relevant security personnel registration form. The relationship between each commission / bureau and item is represented by selecting the corresponding commission / bureau and item and drawing a directed line from the commission / bureau to the item. In this way, the business logic is input into the system through the business model, which is the process of constructing the business metadata graph.

[0054] The canvas technique in this example can be implemented using standard Vue.js 2 features, and can also utilize operators encapsulated in this technique, including data filtering, data collision, and mathematical calculations on certain fields. The operator referred to in this example is a filtering operation on the subject; that is, a filtering operator. The filtering condition in this example is the transaction processing subject of the applicant's region, with the aim of finding the relevant commissions and bureaus within that region through this operator.

[0055] It can also pull target data fields from the technical metadata into the canvas and build a connection relationship between the target data fields and the attribute information of the corresponding business model to expand the directed graph.

[0056] Specifically, the association between business metadata and technical metadata involves business personnel finding the corresponding data fields in the technical metadata and dragging them onto the canvas. Then, they connect the attributes in the business model with the data fields in the technical metadata by using lines, thereby achieving the connection between business metadata and technical metadata and realizing the integration of the business metadata graph and the technical metadata graph.

[0057] The front-end interface records detailed information about the business model on the canvas and the association between the business model and technical metadata, including: entities involved in the business model, the layout of entities on the canvas, entity attribute relationships, and the relationship between attributes and technical metadata. Examples include bureaus A, B, and C related to government departments; licenses F and certificates S related to permits; various attributes related to forms; relationships between entities, such as the relationship between bureau A and the relevant security personnel registration form; and the association between certificate S in the permits and the corresponding table in the technical metadata. In some embodiments, the model presentation layer uses the following data storage method:

[0058] Retrieve the model and layout information passed from the front end to the back end in JSON format;

[0059] According to the JSON structure, the data is stored in the model graph data table of the model presentation layer to record the node information of each first node in the directed graph and the relationship between the nodes. The node information includes node code, node name, node type, business attributes contained in the node, operators involved, conditions and expressions. The relationship between the nodes is recorded with the preceding node code of each node.

[0060] In some embodiments, the model abstraction layer is configured to convert the directed graph of the JSON structure stored in the model presentation layer into the structure of the abstraction layer, and to map the first node of the presentation layer to one or more second nodes of the abstraction layer. The number of nodes in the abstraction layer is determined by the number of operators in the presentation layer.

[0061] The business model is converted into a list of nodes and node contexts. Each node context corresponds to an operator in the model presentation layer node. Any node context consists of node input parameters and output parameters. The input parameters include the data tables and data fields involved, and the output parameters include the intermediate tables and data fields output.

[0062] Specifically, the node context of the model abstraction layer decouples the presentation layer and the execution layer. This means the presentation layer does not depend on the technical architecture and DAG execution framework used by the execution layer. Whether the execution layer depends on Spark's execution on DolphinScheduler has no impact on the presentation layer. The operators encapsulated in the presentation layer also do not depend on the implementation of the execution layer. The presentation layer is designed to facilitate understanding and operation by business users. The model abstraction layer also standardizes node actions, that is, it transforms them into which operator is used on two datasets. The abstraction of operators includes filter, map, union, join, distinct, and aggregate.

[0063] Traditional systems typically assign a task node to a JAR file or Python code snippet, which is binding specific code to a custom node. However, this is not feasible for systems where business models are created by business users and support dynamic execution. It's impractical to have developers create the task node code and then upload it to the execution layer after each business user creates the model; this doesn't meet the need for rapid business response in a big data environment. In some embodiments, the model execution layer is configured as follows:

[0064] Create corresponding task nodes based on the content of the node context in the model abstraction layer;

[0065] Use a uniform type of task package to perform task scheduling, and execute tasks based on the created task nodes.

[0066] Due to the existence of the model abstraction layer, this application's architecture supports multiple model execution layers. This example uses Dolphinscheduler to illustrate the model execution process. The model execution layer in this example runs Spark tasks on Dolphinscheduler. Normally, Spark tasks require uploading a specific JAR file each time a DAG execution graph is created. This solution uses a unified Spark task JAR file, requiring only one upload when creating the DAG. Different task nodes are created based on the different node contexts of the model abstraction layer, and subsequent execution also parses the node context of the abstraction layer. The generic Spark task JAR file encapsulates how various operators from the abstraction layer—filter, map, union, join, distinct, and aggregate—are executed in the Dolphinscheduler environment. For example, the abstraction layer's filter is parsed to determine which fields to select from the Spark Dataset and what filtering conditions are used; map is parsed to perform one or more specified mathematical calculations; and join is parsed to perform a join operation between two Datasets based on join conditions. Dolphinscheduler's role in the execution layer is task scheduling; the actual execution is performed by each task node.

[0067] Based on the foregoing description, this embodiment only requires uploading a general JAR file once. Then, during the model creation phase, the interface of the model execution layer is called. Based on the JSON structure of the DAG passed from the abstraction layer, the number of task nodes is parsed and assembled into task nodes of DolphinScheduler, which also includes the dependencies of each task.

[0068] The DolphinScheduler interface for generating DAGs is a standard operation and will not be described in detail here. The specific structure of the node context described in this solution is as follows:

[0069] Basic information, including node name, node type, and description. Node types include start, stage, and end.

[0070] Input parameters include the input table and field names. Generally, there are two input tables. In the presentation layer business node, the input parameters can be multiple tables and multiple operators. In the model abstraction layer, a transformation has been performed to convert multiple operators into a single operator.

[0071] Output parameters, including the output table and fields.

[0072] Operators currently support filter, map, union, join, distinct, and aggregate.

[0073] Most current task executors only retain the final calculation result after the entire process is completed. This application's solution focuses on business twins. In some embodiments, executing tasks based on created task nodes includes executing the task nodes corresponding to the business model according to the triggering conditions of the business model to obtain the execution result. That is, this application's solution executes the specific model according to the model's triggering conditions to obtain the execution result. This design is equivalent to running the entire process of real production, which is the business twin mentioned in this example. It not only needs to see the final execution result but also understand the reasons for its impact; that is, the calculation results of each step also need to be saved as a basis.

[0074] Data twinning refers to the simulation achieved on the system provided by this solution before the triggering conditions are input into the production system. In this embodiment, an optimization model of disease detection points related to epidemiology is used as an example. This model reflects the impact of the addition and reduction of detection points in a certain area on the detection pressure of other detection points.

[0075] This checkpoint optimization model is based on the existing multiple checkpoints on a certain street, and involves adding or removing checkpoints. Using this sample data, the probability p of people within the street area visiting each checkpoint can be calculated.

[0076] By combining probability data with a distance table of people to each testing point, and using linear regression to assume the walking distance of people to the testing point is s, the specific regression formula p = f(s) is obtained by performing linear regression through Weka. This regression formula can be used to calculate the probability of all people going to all nucleic acid testing points.

[0077] The specific implementation of the above-mentioned detection point optimization model in the background system of this application is as follows:

[0078] Prepare the relevant data tables for the checkpoints. The specific steps for the checkpoint optimization model are: calculating the probability table of personnel going to each checkpoint, statistical nodes before changes in the number of people at each checkpoint, estimating the number of people at each checkpoint, and comparing changes in the number of people at each checkpoint. Specifically, how to implement this model on the system using a graph approach is similar to the aforementioned drag-and-drop method of describing the model on a canvas. Data storage depends on the operators and expressions specified at each node.

[0079] If a detection point is added or removed, the probability table is recalculated, and then the checkpoint optimization model is manually triggered on the interface to obtain the predicted result and compare it with the result before adjustment.

[0080] In summary, this application integrates the results of data elements and constructs a main 360-degree micro-database based on the data elements. In this application, the original data is in a data vault physically isolated by a one-way shutter. The data element results are processed by the element processing center in the data vault and dynamically transferred to the data element library outside the shutter according to the authorization agreement. This mechanism overcomes the security and compliance issues of the original data.

[0081] Traditional systems are highly coupled from the business model presentation layer to the page model execution layer; changes to the model execution layer will lead to changes in the entire system. The method in this application adopts a three-layer design of model presentation layer, model abstraction layer, and model execution layer, which loosely couples the model presentation layer to the execution layer and supports multiple execution engines simultaneously.

[0082] The method in this application features a unified node context structure, which supports the creation of different task nodes in the execution layer by a unified JAR package based on parameters, and execution based on the node context, thus supporting rapid response to business needs in the context of big data.

[0083] This application also proposes a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the steps of the aforementioned data processing method based on a twin platform supporting multiple execution engines.

[0084] This application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned data processing method based on a twin platform supporting multiple execution engines.

[0085] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0086] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A data processing method based on a twin platform supporting multiple execution engines, characterized in that, include: Through the data element interface of the data element trading platform, the data element results are pulled into the message queue of the twin platform according to the token of each data element; The complex event processing module is invoked to trigger the associated business model based on the data element results in the message queue, so as to perform corresponding operations based on the business model and write the execution results into the data asset library; The construction and execution process of the business model are implemented through a three-layer architecture: a model presentation layer, a model abstraction layer, and a model execution layer. The model presentation layer is configured as follows: Obtain the associated entities involved in the target business corresponding to the data element result, and determine the transactions to be processed by the associated entities; Provides a canvas for visually configuring the attribute information of the business model required by the target business, and establishes the connection relationship between the business model and related entities to form a directed graph; and Pull the target data field from the technical metadata into the canvas, and construct the connection relationship between the target data field and the attribute information of the corresponding business model to expand the directed graph; The model presentation layer uses the following data storage method: Retrieve the model and layout information passed from the front end to the back end in JSON format; According to the JSON structure, the data is stored in the model graph data table of the model presentation layer to record the node information of each first node in the directed graph and the relationship between the nodes. The node information includes node code, node name, node type, business attributes contained in the node, operators involved, conditions and expressions. The relationship between the nodes is recorded with the preceding node code of each node. The model abstraction layer is configured to convert the directed graph of the JSON structure stored in the model presentation layer into the structure of the abstraction layer, and map the first node of the presentation layer to one or more second nodes of the abstraction layer. as well as The business model is converted into a list of nodes and node contexts. Each node context corresponds to an operator in the model presentation layer node. Any node context consists of node input parameters and output parameters. The input parameters include the data tables and data fields involved, and the output parameters include the intermediate tables and data fields output. The model execution layer is configured as follows: Create corresponding task nodes based on the content of the node context in the model abstraction layer; Use a uniform type of task package to perform task scheduling, and execute tasks based on the created task nodes.

2. The data processing method based on a twin platform supporting multiple execution engines as described in claim 1, characterized in that, Executing tasks based on created task nodes involves executing the task nodes corresponding to the business model according to the triggering conditions of the business model, in order to obtain the execution result.

3. A computer device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the data processing method based on a twin platform supporting multiple execution engines as described in claim 1 or 2.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method based on a twin platform supporting multiple execution engines as described in claim 1 or 2.

Images