Real-time data product scheduling method, device and equipment based on node routing

Abstract

The application provides a real-time data product scheduling method, device and equipment based on node routing. Wherein, the data product can be visually configured in advance through the management platform, and the configured data product is stored in the database for subsequent user use; then, the target data product to be executed can be determined according to the user operation and called from the database, and then the target data product is parsed, the execution process of the target data product is generated and executed. By such setting, firstly, the data product does not need to be realized by hard coding, only the visual configuration of each node of the data product in the management system is needed, and the configuration takes effect immediately after completion, which greatly improves the development and online efficiency and is convenient for maintenance; secondly, all data product configurations can flexibly configure the calling link of the data product through the self-defined routing expression, which improves the flexibility and expansibility of the data product.

Description

Technical Field

[0001] This application relates to a method, apparatus, and device for scheduling real-time data products based on node routing, belonging to the field of computer technology. Background Technology

[0002] With the rapid development of the internet finance industry, risk management and control have become crucial to ensuring stable business operations. However, the performance of traditional risk control products is no longer sufficient to meet the demands of the complex and ever-changing financial environment.

[0003] In implementing real-time computing functions for data products, multi-threaded parallel calls to multiple data sources have become a common strategy. First, data is retrieved from different environments or databases using multiple threads. Then, after all the data is returned, risk control-related logic is executed, such as executing a credit scoring model to generate a score or calculating a large number of feature variables, and finally, the results are returned to the customer.

[0004] However, most of the above strategies are implemented using hard-coding. This approach has several significant drawbacks, such as high code redundancy, high development and maintenance costs, poor scalability, and poor flexibility. Summary of the Invention

[0005] This application provides a method, apparatus, and device for scheduling real-time data products based on node routing, in order to solve the problems of high code redundancy, high development and maintenance costs, poor scalability, and poor flexibility in existing technical solutions.

[0006] In a first aspect, embodiments of this application provide a real-time data product scheduling method based on node routing, including:

[0007] Based on the user's configuration operations on the management platform, data products are configured and stored in the database. The management platform displays multiple optional configuration items. The data products include multiple nodes of the same or different types, node routing configurations, and dependencies between nodes. The nodes are used to execute corresponding processing procedures. The routing configuration of a node is a routing expression configured on that node to determine the next node that the node points to. The dependencies between nodes are the execution order between connected nodes.

[0008] The target data product to be executed is determined based on user operations, and the target data product is called from the database.

[0009] The target data product is parsed, and the execution flow of the target data product is generated and executed.

[0010] Based on the above method, optionally, storing the configured data product in the database includes:

[0011] Generate a unique product code for each configured data product;

[0012] Establish a mapping relationship between all nodes included in the data product and the product code;

[0013] The mapping relationship is stored in the database.

[0014] Based on the above method, optionally, the step of parsing the target data product, generating the execution flow of the target data product, and executing it includes:

[0015] All starting nodes are determined from the set of nodes included in the target data product; the starting nodes are nodes that do not depend on any other nodes.

[0016] Based on the type of the node, all starting nodes are executed in parallel;

[0017] The node processing flow is executed repeatedly until all nodes have been executed. The output of the last node executed is then used as the output of the target data product.

[0018] The node processing flow includes:

[0019] After any node has been executed, it is determined whether the node has a routing expression configured. If so, the routing expression is parsed and the next node to be executed is determined based on the parsing result. If not, the next node connected to the node is determined as the next node to be executed.

[0020] Determine whether all dependent nodes of the next node to be executed have been executed. If yes, execute the next node. If not, wait for all dependent nodes of the next node to be executed to be executed before executing the next node.

[0021] Based on the above method, optionally, the types of nodes include data acquisition nodes and data processing nodes; the data acquisition node is used to obtain metadata through network calls, and the data processing node is used to calculate and process the metadata obtained by the data acquisition node based on a pre-configured algorithm or model.

[0022] Based on the above method, optionally, parsing the routing expression and determining the next node to be executed based on the parsing result includes:

[0023] Extract the condition expression from the routing expression;

[0024] The conditional expression is parsed to determine whether there is data in the data layer of the conditional expression, and the next node to be executed is determined based on the determination result.

[0025] Based on the above methods, optionally, the following further methods may also be included:

[0026] The target data product will be visualized.

[0027] Secondly, embodiments of this application also provide a real-time data product scheduling device based on node routing, which includes:

[0028] The configuration module is used to configure data products based on user configuration operations on the management platform and store the configured data products in the database. The management platform displays multiple optional configuration items. The data products include multiple nodes of the same or different types, node routing configurations, and dependencies between nodes. The nodes are used to execute corresponding processing procedures. The node routing configuration is a routing expression configured on the node to determine the next node that the node points to. The dependencies between nodes are the execution order between connected nodes.

[0029] The invocation module is used to determine the target data product to be executed based on user operations and invoke the target data product from the database.

[0030] The execution module is used to parse the target data product, generate the execution flow of the target data product, and execute it.

[0031] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor calls and executes the computer program, it implements the real-time data product scheduling method based on node routing as described in any one of the first aspects.

[0032] The technical solution provided in this application has at least the following beneficial effects:

[0033] This application provides a method, apparatus, and device for real-time data product scheduling based on node routing. In this method, data products can be pre-configured visually through a management platform, and the configured data products are stored in a database for later user access. Subsequently, the target data product to be executed can be determined based on user operations and called from the database. The target data product is then parsed to generate its execution flow and executed. This setup offers several advantages: First, data products do not require hard-coding; they only need to be visually configured at each node within the management system, and the configuration takes effect immediately, greatly improving development and deployment efficiency while also facilitating maintenance. Second, all data product configurations can be flexibly configured using custom routing expressions to define the data product's call chain, enhancing the flexibility and scalability of the data products. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Furthermore, these drawings and textual descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments.

[0035] Figure 1 A flowchart illustrating a real-time data product scheduling method based on node routing provided in one embodiment of this application;

[0036] Figure 2 A dependency graph of nodes for a data product provided in one embodiment of this application;

[0037] Figure 3 for Figure 2 The corresponding execution flowchart;

[0038] Figure 4 A schematic diagram of the structure of a real-time data product scheduling device based on node routing is provided for one embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0041] Existing methods for calculating report formulas, whether through manual editing of calculation logic by users or the use of third-party report components, have numerous problems.

[0042] Existing technical solutions for implementing real-time computing functions in data products suffer from numerous problems, such as high code redundancy, high development and maintenance costs, poor scalability, and poor flexibility. Specifically:

[0043] 1. High development and maintenance costs: Each product requires additions or modifications to the source code, which generates a large amount of repetitive and non-reusable code, making later maintenance difficult. Moreover, this approach leads to longer development and testing times, increasing time and manpower costs.

[0044] 2. Poor scalability and flexibility: When business rules change or new data sources are added, the entire system may need to be modified extensively, resulting in a large amount of redundant code and increased system complexity.

[0045] To address the aforementioned issues, this application provides a real-time data product scheduling scheme based on node routing. This scheme employs a single node configuration strategy and a single codebase can meet the needs of the vast majority of data products. The following examples and embodiments illustrate the specific implementation scheme in a non-limiting manner.

[0046] Some embodiments of this application provide a real-time data product scheduling method based on node routing, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a real-time data product scheduling method based on node routing, provided in one embodiment of this application.

[0047] like Figure 1 As shown, the real-time data product scheduling method based on node routing in this embodiment includes the following steps:

[0048] Step S101: Based on the user's configuration operations on the management platform, configure the data products and store the configured data products in the database.

[0049] The management platform displays several optional configuration items. Data products include multiple nodes of the same or different types, node routing configurations, and dependencies between nodes. Nodes are used to execute corresponding processing procedures. The routing configuration of a node is a routing expression configured on that node, used to determine the next node that the node points to. The dependencies between nodes are the execution order between connected nodes.

[0050] Specifically, the management platform DPM is a back-end operation platform that provides a series of functions required for configuring data products (hereinafter referred to as products), including but not limited to essential functions such as product management, node management, and routing configuration.

[0051] The management platform provides a visual configuration interface that displays various optional configuration items, including different types of nodes and routing expressions. Through the management platform, users can perform operations such as clicking, dragging, and manually entering data to configure, modify, and store data products.

[0052] From a static perspective, a configured product is a collection of one or more nodes; from a dynamic perspective, it is a call chain. Through node dependencies and routing configurations, a dynamic call chain is generated during execution. The combination of static configuration and dynamic calls forms a complete data product.

[0053] Once configured, the data products are stored in the database for later use. This way, when using the same data product, users don't need to reconfigure it each time; they can simply select the pre-configured data product from the database, greatly reducing user operations and improving convenience.

[0054] In some embodiments, the step of storing the configured data product in the database may specifically include:

[0055] Generate a unique product code for each configured data product;

[0056] Establish a mapping relationship between all nodes included in the data product and the product code;

[0057] Store the mapping relationships in the database.

[0058] Specifically, in this embodiment, after the data product is configured, the management platform generates a product code (prod_code) as a unique identifier for the product. All nodes under this product are mapped to the prod_code and then stored in the database to ensure that all nodes under the product can be found through the prod_code later. The specific form of the product code is not limited; for example, it can be a string composed of multiple numbers, letters, and symbols, either individually or in combination.

[0059] like Figure 2 As shown, after the product configuration is complete, a dependency graph of all nodes of the product can be viewed on the DPM platform. Nodes can be categorized into several types, including but not limited to: data acquisition nodes (std nodes) and data processing nodes (mode l nodes). Data acquisition nodes are used to retrieve metadata via network calls, while data processing nodes are used to perform calculations and processes on the metadata retrieved by the data acquisition nodes based on pre-configured algorithms or models.

[0060] Furthermore, nodes can be configured with one or more dependent nodes (i.e., dependencies between nodes), where the dependent node executes before the current node, such as... Figure 2 In the context of data products, the std2 node depends on the std1 node, and the mode l node depends on std1, std2, and std3 (a total of three nodes). Nodes that do not depend on any other nodes are identified as start nodes, such as std1 and std3. When executing data products, the start nodes are executed first.

[0061] Furthermore, routing expressions can be configured on nodes, allowing the system to determine which node to execute next based on the presence or absence of data on that node. Figure 2 If the std1 node is hit (data exists), the mode 1 node is executed; otherwise, the std2 node is executed.

[0062] Step S102: Determine the target data product to be executed based on user operations, and call the target data product from the database.

[0063] Specifically, users can select the data product they want to execute, i.e., the target data product, in the user interface. For example, the user interface can use... Figure 2 Multiple data products are displayed in a format that allows users to select the data product they wish to execute; alternatively, users can directly enter the product code (prod_code) of a data product to automatically identify the target data product.

[0064] The system will load all nodes and dependency information of the product from the DPM platform into its own memory for logical processing based on the product code prod_code.

[0065] In some embodiments, the above method may further include: visualizing the target data product.

[0066] In other words, after identifying the target data product, it can be visualized to allow users to easily check whether it meets their requirements. Furthermore, users can be allowed to further modify and adjust the target data product.

[0067] Step S103: Analyze the target data product, generate the execution flow of the target data product, and execute it.

[0068] Specifically, after the system retrieves the target data product to be executed from the database, it will parse it, determine the execution process, and then execute it.

[0069] The parsing process involves determining all the nodes included in the data product, their types, dependencies, and routing configurations, and then determining the corresponding execution flow.

[0070] In some embodiments, the specific implementation process of step S103 may include:

[0071] Identify all starting nodes from the set of nodes included in the target data product;

[0072] Based on the type of the node, all starting nodes are executed in parallel;

[0073] The node processing flow is executed repeatedly until all nodes have been executed. The output of the last executed node is then used as the output of the target data product.

[0074] The node processing flow includes:

[0075] After any node has been executed, it is determined whether the node has a routing expression configured. If so, the routing expression is parsed and the next node to be executed is determined based on the parsing result. If not, the next node connected to the current node is determined as the next node to be executed.

[0076] Determine whether all dependent nodes of the next node to be executed have been executed. If yes, execute the next node. If not, wait for all dependent nodes of the next node to be executed to be executed before executing the next node.

[0077] In this process, all starting nodes are executed in parallel, meaning that multiple starting nodes execute without interfering with each other, which can improve processing speed.

[0078] In some embodiments, the specific process of parsing the routing expression and determining the next node to be executed based on the parsing result in the above process may include: extracting the condition expression from the routing expression; parsing the condition expression, determining whether there is data in the data layer of the condition expression, and determining the next node to be executed based on the determination result.

[0079] To better understand, examples are provided to illustrate the configuration and parsing methods of route expressions. In this example, the configuration form of the route expression is as follows:

[0080] If ${data.0|data.0|model_id} != isNull)#shuchu1 else #shuchu2 == shuchu1, the parsing rules for this route expression are as follows:

[0081] a) Extract the conditional expression (${data.0|data.0|model_id} != isNull) from the if expression;

[0082] b) Parse the conditional expression, take the first data from the data layer (data.0) of the data returned by the current node, and then take the first data from the data layer (data.0) of this data, and determine whether the mode l_id value is empty;

[0083] c) If it is not empty, then the next node is the shuchu1 after the first #, where the shuchu1 after == is used to identify the next node;

[0084] d) If empty, then the next node is the shuchu2 node after the second # in el se.

[0085] In addition, according to Figure 2 The node dependency graph and the execution flowchart of this product are as follows: Figure 3 As shown.

[0086] from Figure 3 It can be seen from this:

[0087] 1. The std1 and std3 nodes do not depend on any other nodes, indicating that they are the start nodes and will be executed first.

[0088] 2. After the std1 node finishes execution, the execution of subsequent nodes will be determined based on whether the result of the std1 node hits the target (i.e., the routing expression).

[0089] a) If std1 is a hit, then std2 will be ignored and mode l will be executed. Also, since mode l depends on std1 and std3, before executing mode l, it will first check whether the two predecessor nodes std1 and std3 of mode l have been executed. Only after std1 and std3 have been executed will model node be executed.

[0090] b) If the result of std1 is a miss, then ignore the mode l node and execute the std2 node. Finally, return to the mode l node. When executing the mode l node, it is also necessary to wait for the std3 node to finish executing.

[0091] 3. Finally, the mode l node is used as the end node, and the execution result of the mode l node is the output result of the product.

[0092] To better understand the entire plan, Figure 2 Based on this, a typical product application scenario is as follows:

[0093] 1. A customer calls a product with product code 10001 and passes business data such as ID number and name;

[0094] 2. Locate the payment data node std1 and consumption data node std3 used by the product code 10001;

[0095] 3. Pass the ID number, name, and other information to the two std nodes (std1 and std3) to query transaction data such as payment and consumption records;

[0096] 4. If there is missing data, then the backup node std2 is selected as a further supplement using the routing expression;

[0097] 5. Transmit the streaming data to the big data model node mode l to calculate the model score;

[0098] 6. Return the model to the customer.

[0099] Therefore, based on the solutions described in the above embodiments, data products can be pre-configured visually through the management platform. The configured data products are then stored in the database for subsequent user use. Subsequently, the target data product to be executed can be determined based on user operations and called from the database. The target data product is then parsed, its execution flow is generated, and it is executed. This setup offers several advantages: First, data products do not require hard-coding; instead, each node of the data product is visually configured within the management system, and the configuration takes effect immediately, greatly improving development and deployment efficiency while also facilitating maintenance. Second, all data product configurations can be flexibly configured using custom routing expressions to optimize the data product's call chain, enhancing the flexibility and scalability of the data products.

[0100] Furthermore, embodiments of this application provide a real-time data product scheduling device based on node routing, referring to... Figure 4 The real-time data product scheduling device based on node routing includes:

[0101] Configuration module 41 is used to configure data products based on user configuration operations on the management platform and store the configured data products in the database. The management platform displays multiple optional configuration items. Data products include multiple nodes of the same or different types, node routing configurations, and dependencies between nodes. Nodes are used to execute corresponding processing procedures. The routing configuration of a node is a routing expression configured on that node, used to determine the next node that the node points to. The dependencies between nodes are the execution order between connected nodes.

[0102] Module 42 is used to determine the target data product to be executed based on user operations and to call the target data product from the database.

[0103] Execution module 43 is used to parse the target data product, generate the execution flow of the target data product, and execute it.

[0104] The specific implementation methods of each module of the above-mentioned node routing-based real-time data product scheduling device can be found in the corresponding content of the aforementioned method embodiments, and will not be repeated here.

[0105] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, the electronic device includes a memory 51 and a processor 52; wherein, the memory 51 stores a computer program, and when the processor 52 calls and executes the computer program, it implements the real-time data product scheduling method based on node routing in any of the above embodiments.

[0106] The electronic device can be a desktop computer, a laptop computer, or a server, etc.

[0107] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0108] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0109] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0110] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0111] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0112] Furthermore, the functional units in the various embodiments of this invention can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0113] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0114] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for real-time data product scheduling based on node routing, characterized in that, The method comprises: configuring a data product based on a user's configuration operation on a management platform, and storing the configured data product into a database; the management platform displays a plurality of selectable configuration items, the data product comprises a plurality of same or different types of nodes, routing configuration of the nodes, and dependency relationship between the nodes, the nodes are used to execute corresponding processing processes, the routing configuration of the nodes is a routing expression configured in the node, used to determine a next node pointed by the node, and the dependency relationship between the nodes is an execution order between connected nodes; determining a target data product to be executed based on a user operation, and calling the target data product from the database; parsing the target data product, generating an execution flow of the target data product, and executing the target data product, wherein the parsing the target data product, generating the execution flow of the target data product, and executing the target data product comprise executing a node processing flow; the node processing flow comprises: after any node is executed, determining whether the node is configured with a routing expression, if yes, parsing the routing expression, and determining a next node to be executed according to a parsing result, if no, determining a next node connected with the node as the next node to be executed; determining whether all dependent nodes of the next node to be executed are executed, if yes, executing the next node, if no, waiting until all dependent nodes of the next node to be executed are executed, and then executing the next node.

2. The method of claim 1, wherein, The storing the configured data product into the database comprises: generating a unique product code for the configured data product; establishing a mapping relationship between all nodes included in the data product and the product code; storing the mapping relationship into the database.

3. The method of claim 1, wherein, The parsing the target data product, generating the execution flow of the target data product, and executing the target data product further comprise: determining all start nodes from a node set included in the target data product; the start node is a node not dependent on any other node; parallelly executing all start nodes according to a type of the nodes; recursively executing the node processing flow until all nodes are executed, and taking an output of a last executed node as an output of the target data product.

4. The method of claim 3, wherein, The type of the nodes comprises a data acquisition node and a data processing node; the data acquisition node is used to acquire metadata through network calling, and the data processing node is used to calculate and process the metadata acquired by the data acquisition node based on a pre-configured algorithm or model.

5. The method of claim 3, wherein, The parsing the routing expression, and determining the next node to be executed according to the parsing result comprise: taking a condition expression from the routing expression; parsing the condition expression, determining whether data exists in a data layer of the condition expression, and determining the next node to be executed according to a determination result.

6. The method of claim 1, wherein, Further comprising: visually displaying the target data product.

7. A node routing based real-time data product scheduling apparatus, characterized by, comprise: The configuration module is configured to configure a data product based on a configuration operation of a user on a management platform, and store the configured data product into a database; the management platform displays a plurality of selectable configuration items, the data product includes a plurality of same or different types of nodes, routing configuration of the nodes, and dependency relationship between the nodes, the node is used to execute a corresponding processing process, the routing configuration of the node is a routing expression configured in the node, used to determine a next node pointed by the node, and the dependency relationship between the nodes is an execution order between connected nodes; The calling module is configured to determine a target data product to be executed based on a user operation, and call the target data product from the database; The execution module is configured to parse the target data product, generate an execution flow of the target data product, and execute the target data product, wherein the parsing the target data product, generating the execution flow of the target data product, and executing the target data product include: executing a node processing flow; the node processing flow includes: after any node is executed, determining whether the node is configured with a routing expression, if yes, parsing the routing expression, and determining a next node to be executed according to a parsing result, if no, determining a next node connected with the node as the next node to be executed; determining whether all dependent nodes of the next node to be executed are executed, if yes, executing the next node, if no, waiting until all dependent nodes of the next node to be executed are executed, and then executing the next node.

8. An electronic device, comprising: A memory and a processor are included, the memory stores a computer program, and the processor calls and executes the computer program to implement the real-time data product scheduling method based on node routing according to any one of claims 1 to 6.

Images