Real-time and offline compatible decision engine implementation method and engine system

Abstract

The invention discloses a real-time and offline compatible decision engine implementation method and an engine system, relates to the technical field of data processing, and solves the technical problem that when a certain strategy of a user needs to act on two scenes at the same time, the strategy needs to be configured twice, and the labor cost is increased. The method comprises the following steps: step 1, acquiring a strategy configured by a user and converting the strategy into a naive expression; 2, judging the type of the naive expression; 3, judging the type of the naive expression in the step 2, and deducing the result type of the naive expression; 4, repeating the steps 1-3 to process each strategy configured by the user, and checking the grammar correctness and return values of all expressions until the strategy configuration of the user is completed; 5, determining an application scene of the strategy and performing corresponding processing; the method can meet the requirement that the same strategy can be applied to real-time calculation and offline calculation at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for implementing a decision engine compatible with real-time and offline and an engine system. Background Art

[0002] In the field of credit risk control, the traditional credit granting and loan disbursement processes usually require manual approval. Risk managers must read and analyze a large number of credit reports to comprehensively evaluate the qualifications of customers. However, credit strategies in the field of financial risk control are often frequently adjusted due to policy and market changes. Therefore, the strategy configuration needs to be highly flexible and cannot be directly written into the program through hard coding, which poses higher requirements for the flexible allocation ability of the decision engine. The birth of the decision engine not only solves this problem, but also greatly improves the work efficiency in the financial field, laying a solid foundation for the realization of inclusive finance.

[0003] Existing technical solutions can be divided into two types: real-time decision engines and offline decision engines.

[0004] (1) The real-time decision engine was the first to be developed, and currently, all pre-loan strategies in the field of credit risk control are implemented using such engines. A well-known one is the ILOG decision engine produced by IBM. Its implementation solution is to compile the user's strategy into an executable file and dynamically load it in the program. When the call interface exposed by the decision engine is invoked, the system will pass the parameters transmitted by the upstream system into the entry of the dynamically loaded strategy function, and after completing the strategy calculation, synchronously return it to the upstream system. In this process, for the upstream system, the system strategy is a black box, and when the strategy input parameters remain unchanged, the upstream is unaware of the changes in the strategy, which well decouples the systems. At the same time, for users, the strategy upgrade is smooth and there will be no phenomenon such as carding. Finally, since the strategy has been compiled into an executable file and the entire process is implemented in memory, the latency is often very low. Usually, when the real-time decision engine executes credit strategies, its latency is within 20 ms.

[0005] The biggest problem with the real-time decision engine is that it cannot perform batch calculations, which results in almost all mid-loan and marketing strategies being unable to be implemented in such engines. For example, when a certain bank needs to re-evaluate the risk situations of tens of millions of existing customers, or fish out customer groups with higher marketing success rates from a large number of dormant customers. If directly using the real-time decision engine, the upstream system needs to pre-load the information of all customers in advance and initiate calls to the decision engine one by one to screen customers that meet the requirements. In this process, there are at least the following problems:

[0006] a. The upstream system cannot have infinite memory. Therefore, it is impossible to load all customer information at once. The file needs to be split and processed multiple times. Here, the code logic is very error-prone and the efficiency is very low.

[0007] b. Although the real-time decision engine can make decisions quickly for single customer data, it is still powerless for tens of millions of calls. Therefore, to meet the timeliness requirements, the real-time engine can only be expanded, using more computing nodes to complete batch calculations. More nodes require purchasing more decision engine licenses, resulting in high costs.

[0008] c. When a large number of calls suddenly pour into the decision engine, it will slow down the engine's response, which may squeeze the pre-loan policies that should have been calculated by the engine, resulting in order blocking.

[0009] (2) To solve the problem that the real-time decision engine cannot perform batch calculations, the prior art proposed a method for implementing a decision engine based on a decision tree CN111638883B. This method no longer compiles the user's policies into executable files for in-memory calculation. Instead, according to the policies configured by the user, the policies are converted into SQL statements and submitted to the HIVE cluster of big data for calculation. When making decisions on massive data, the upstream system only needs to tell the decision engine the database name and table name where the data to be decided is stored. The decision engine can automatically submit SQL to the HIVE cluster for calculation according to the policy situation, and write the final execution result into the result table and asynchronously notify the upstream system. In this way, the upstream system no longer needs to read all customer information in chunks by itself. At the same time, because the computing power throughput of the HIVE cluster is extremely large, decisions on tens of millions of data can often be completed within 5 minutes. At the same time, because big data resources are used, it will not squeeze the pre-loan policies online at all, perfectly solving various problems of the real-time engine.

[0010] The offline engine is the opposite of the real-time engine. It is aimed at batch calculations and is very suitable for policy calculations in scenarios such as mid-loan and marketing. Correspondingly, it is very difficult for the offline engine to make decisions on single data, and its latency is often very high.

[0011] Defects of the prior art: Due to the different implementation principles of the offline engine and the real-time engine, it is impossible to ensure that the same policy will execute the same result in the two engines. When a certain policy of a user acts on two scenarios at the same time, the policy needs to be configured twice, which increases a very high labor cost. Summary of the Invention

[0012] The present invention provides a method for implementing a decision engine that is compatible with both real-time and offline scenarios, taking into account the compatibility of policies in real-time or offline scenarios. For the same policy, the user only needs to configure it once, and when the policy is executed with the same input parameters, regardless of whether it is in an offline scenario or a real-time scenario, the same output result can be guaranteed, solving the technical problem that when a certain policy of the user needs to act on both scenarios simultaneously, the policy needs to be configured twice, increasing the labor cost.

[0013] A method for implementing a decision engine that is compatible with both real-time and offline scenarios includes the following steps:

[0014] Step 1: Obtain the policy configured by the user and convert it into a naive expression;

[0015] Step 2: Judge the type of the naive expression. The naive expression includes a naive expression composed of functions and a naive expression composed of addition, subtraction, multiplication, division, and parentheses;

[0016] Step 3: Determine the type of the naive expression in Step 2 and infer the result type of the naive expression;

[0017] Step 4: Repeat Steps 1 to 3 to process each policy configured by the user, verify the syntax correctness and return value of all expressions until the user's policy configuration is completed;

[0018] Step 5: Determine the application scenario of the policy and perform corresponding processing. The application scenario includes an offline scenario and a real-time scenario.

[0019] Further, Step 1 includes:

[0020] Step 1.1: Obtain the policy configured by the user, remove all spaces in the original string to obtain an expression string S;

[0021] Step 1.2: According to the parentheses matching algorithm and the preset function list, divide the expression string S into naive expressions.

[0022] Further, Step 2 includes:

[0023] Step 2.1: For a naive expression composed of functions, judge whether the type of each parameter is correct; for a naive expression composed of addition, subtraction, multiplication, division, and parentheses, it needs to be first converted into a reverse Polish expression;

[0024] Step 2.2: For the one converted into a reverse Polish expression in Step 2.1, verify whether the basic index type and the operator match, and infer the return value type.

[0025] Furthermore, if the application scenario of the strategy in step 5 is an offline scenario, the function in the expression is translated into a corresponding SQL function and submitted to the hive cluster for calculation; if the application scenario of the strategy is a real-time scenario, the expression string is compiled as a groovy script to generate a class file and directly loaded for use.

[0026] Furthermore, the step 1.2 comprises:

[0027] Step 1.2.1: Initialize an empty stack

[0028] Step 1.2.2: Scan the string from left to right. When a left bracket is encountered, push the current subscript onto the stack. When a right bracket is encountered, pop the stack. When the string is scanned, if the stack is empty, the brackets are matched correctly. If the stack is not empty or the right bracket is popped before the scan is completed, the brackets are matched incorrectly, indicating a syntax error.

[0029] Step 1.2.3: According to the regular expression, find the preset function directly by the preset function name, and use step 1.2.2 to find all the parameters of the preset function. The specific method is to set an empty stack s, push the left bracket into the stack, and pop the right bracket into the stack. When the empty stack appears for the first time, the position is the end position of the current function. The remainder of the entire function can be replaced by a variable;

[0030] Step 1.2.4: After step 1.2.3, there are no functions in the expression, and all brackets are matched. The remaining expression should naturally form a simple expression.

[0031] Furthermore, the step 3 comprises:

[0032] Step 3.1: Because the plus and minus signs have a lower priority, the expression is split according to the +- sign first. If there is no +- sign, use * / to split;

[0033] Step 3.2: According to the expression list cut in 3.1, recursive processing is performed separately. When the expression is left with only a single element indicator, it means that the recursion has been completed and the current indicator name is returned.

[0034] Furthermore, the step 4 comprises:

[0035] Step 4.1: Scan the reverse Polish expression from left to right and initialize an empty stack s;

[0036] Step 4.2: When an indicator variable is encountered during the scanning process, the variable is pushed into stack s;

[0037] Step 4.3: If an operator symbol such as +, -, *, / , etc. is encountered, pop two variables from the stack. At this time, check whether the operator and the types of the two variables meet the requirements. For example, the + of a string can directly determine a syntax error. If the determination is successful, push the result variable into s;

[0038] After the scanning is completed, the return type of the entire expression can be obtained, and there will be no abnormal situations such as popping the stack when the stack is empty or there are still remaining elements in the stack after the scanning is completed.

[0039] A real-time and offline compatible decision engine system includes a policy input module, an offline module, and an online module. The policy input module uses a real-time and offline compatible decision engine implementation method to assist users in writing policies. The offline module is used to translate the functions in the expression into corresponding SQL functions and submit them to the hive cluster for calculation; the online module is used to compile the expression string as a groovy script to generate a class file and directly load and use it.

[0040] The beneficial effects of the present invention include:

[0041] 1. Because algorithms such as reverse Polish notation and bracket matching are used to shield the conflicting syntax between SQL and common groovy in-memory calculations, the same policy can be applied to both real-time calculation and offline calculation.

[0042] 2. Only the syntax is parsed and shielded, and the operations are still completed using groovy or SQL. Therefore, the logical calculation optimization of the underlying engine for expressions is not discarded, and it has high performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a real-time and offline compatible decision engine implementation method involved in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0045] Embodiment 1

[0046] The implementation principles of the offline engine and the real-time engine are different, and it is impossible to guarantee that the same policy will produce the same results in the two engines. The specific manifestations include: in most real-time engines that use in-memory computing, when calculating the expression "10" + "10", it is defaulted that the + operation at this time is a string paste operation, and the result "1010" will be obtained. In the same calculation logic in SQL, it will be implicitly converted to 10 + 10, and finally the result 20 will be obtained. To solve the above problems, this embodiment discloses a method for implementing a decision engine that is compatible with real-time and offline, such as Figure 1 shown, including the following steps:

[0047] Step 1: Obtain the policy configured by the user and convert it into a naive expression;

[0048] Step 2: Judge the type of the naive expression, where the naive expression includes a naive expression composed of functions and a naive expression composed of addition, subtraction, multiplication, division, and parentheses;

[0049] Step 3: Judge the type of the naive expression in Step 2 and infer the result type of the naive expression;

[0050] Step 4: Repeat Steps 1 to 3 to process each policy configured by the user, verify the syntax correctness and return value of all expressions until the user's policy configuration is completed;

[0051] Step 5: Determine the application scenario of the policy and perform corresponding processing, where the application scenario includes an offline scenario and a real-time scenario.

[0052] The said Step 1 includes:

[0053] Step 1.1: Obtain the policy configured by the user, remove all spaces in the original string, and obtain the expression string S;

[0054] Example 1: For example, if the user configures ln(a + b * c - (d * f * (f + b + c * d) - (a * e)) * t)>0, where a, b, c, d, e, f, t are existing indicators stored in the database respectively, after removing all spaces, we get:

[0055] ln(a + b×c - (d×f×(f + b + c×d) - (a×e))×t)>0

[0056] Step 1.2: According to the parentheses matching algorithm and the preset function list, split the expression string S into naive expressions.

[0057] The said Step 1.2 includes:

[0058] Step 1.2.1: Initialize an empty stack stack;

[0059] Step 1.2.2: Scan the string from left to right. When a left parenthesis is encountered, push the current subscript onto the stack; when a right parenthesis is encountered, pop the stack. After the string scanning is completed, if the stack is empty, the parentheses match correctly; if the stack is not empty or a right parenthesis is encountered when the stack becomes empty before the scanning is completed and a pop operation is attempted, it indicates a failure in parentheses matching, which means there is a syntax error.

[0060] Step 1.2.3: According to the regular expression, directly find the preset function by the preset function name, and use Step 1.2.2 to find all the parameters of this preset function after this preset function. The specific method is to set an empty stack s, push when encountering a left parenthesis, and pop when encountering a right parenthesis. When the stack becomes empty for the first time, this position is the end position of the current function. The modulo operation on the entire function can be replaced by a variable.

[0061] Step 1.2.4: After Step 1.2.3, there are no functions left in the expression, and all parentheses match. The remaining expression should naturally form a naive expression.

[0062] Example 2: For example, the expression in Example 1 can be split into multiple naive expressions

[0063] A1 = a + b × c - (d × f × (f + b + c × d) - (a × e)) × t

[0064] A2 = ln(A1)

[0065] A3 = A2 > 0

[0066] A naive expression is either entirely composed of addition, subtraction, multiplication, division, and parentheses, or is a function.

[0067] The said Step 2 includes:

[0068] Step 2.1: For a naive expression composed of functions, determine whether the type of each parameter is correct; for a naive expression composed of addition, subtraction, multiplication, division, and parentheses, it is necessary to first convert it into a reverse Polish expression.

[0069] Step 2.2: For those converted into reverse Polish expressions in Step 2.1, verify whether the basic index types and operators match, and infer the return value type.

[0070] Example 3: For the 3 naive expressions in Example 2:

[0071] A1 is converted into the reverse Polish expression bc × fbcd × fb ++ df × × ae × -t × abc × + -

[0072] A2 is a function and does not need to be converted. It only needs to verify whether the input parameter A1 is of numerical type and the return value is numerical

[0073] Convert A3 to Reverse Polish Notation A20>;

[0074] If the application scenario of the strategy in step 5 is an offline scenario, translate the functions in the expression into corresponding SQL functions and submit them to the hive cluster for calculation; if the application scenario of the strategy is a real-time scenario, compile the expression string as a groovy script to generate a class file and directly load and use it.

[0075] Example 6:

[0076] Offline: Translate the ln function of the expression to LN in SQL, and splice the strategy to the where condition, we get:

[0077] select*from table_name where LN(a+b*c-(d*f*(f+b+c*d)-(a*e))*t)>0

[0078] Real-time: The expression is directly compiled into a groovy script and encapsulated into the main function:

[0079] def ln(x) / / This function is pre-set, and all strategy expressions will have the definition of this function

[0080] {

[0081] return Math.log(x)

[0082] }

[0083] def main()

[0084] {

[0085] return ln(a+b*c-(d*f*(f+b+c*d)-(a*e))*t)>0

[0086] }

[0087] When using, directly call the main function of this script

[0088] Step 4 includes:

[0089] Step 4.1: Scan the Reverse Polish Notation from left to right and initialize an empty stack s;

[0090] Step 4.2: During the scanning process, when encountering an index variable, push the variable onto the stack s;

[0091] Step 4.3: If an operator symbol such as +, -, *, / , etc. is encountered, pop two variables from the stack. At this time, check whether the operator and the types of the two variables meet the requirements. For example, the + operator for strings can directly determine a syntax error. If the determination is successful, push the result variable into s;

[0092] Step 4.4: After the scanning is completed, the return type of the entire expression can be obtained, and there will be no abnormal situations such as popping from an empty stack in the middle or there being remaining elements in the stack after the scanning is completed.

[0093] Example 4: Since it is necessary to be compatible with both real-time and offline scenarios, for grammars that are ambiguous for real-time and offline, they should be blocked. For example, in the present invention, the + operator is not allowed to have the situation of adding strings. In the above example, it can be seen that the result type of A1 is a numerical type, and it requires that all its input parameters a, b, c, d, e, f, t must be numerical, otherwise the verification fails; for A3, it is necessary to ensure that A2 is numerical and the result type is boolean.

[0094] A real-time and offline compatible decision engine system includes a policy input module, an offline module, and an online module. The policy input module uses a real-time and offline compatible decision engine implementation method to assist users in writing policies. The offline module is used to translate the functions in the expression into corresponding SQL functions and submit them to the hive cluster for calculation; the online module is used to compile the expression string as a groovy script to generate a class file and directly load and use it.

[0095] The above embodiments only represent the specific implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the protection scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the technical solution of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application.

Claims

1. A real-time offline compatible decision engine implementation method, characterized in that: The following steps are involved: Step 1: Get the user-configured policy and convert it into a simple expression; Step 2: Determine the type of the naive expression, which includes a naive expression consisting of a function and a naive expression consisting of addition, subtraction, multiplication and division brackets; Step 3: Determine the type of the naive expression in step 2 and infer the result type of the naive expression; Step 4: Repeat steps 1 to 3 to process each policy configured by the user, verify the syntax correctness and return value of all expressions, until the user's policy configuration is complete; Step 5: Determine the application scenarios of the strategy and perform corresponding processing. The application scenarios include offline scenarios and real-time scenarios.

2. The method for implementing a real-time offline compatible decision engine according to claim 1, characterized in that: The step 1 comprises: Step 1.1: Get the policy configured by the user, remove all spaces from the original string, and obtain the expression string S; Step 1.2: Split the expression string S into simple expressions according to the bracket matching algorithm and the preset function list.

3. The method for implementing a real-time offline compatible decision engine according to claim 1, characterized in that: The step 2 comprises: Step 2.1: For the simple expression composed of functions, determine whether the type of each parameter is correct; for the simple expression composed of addition, subtraction, multiplication and division brackets, it is necessary to convert it into Reverse Polish Notation first; Step 2.2: For the reverse Polish notation in step 2.1, check whether its basic indicator type and operator match, and infer the return value type.

4. The method for implementing a real-time offline compatible decision engine according to claim 1, characterized in that: If the application scenario of the strategy in step 5 is an offline scenario, the function in the expression is translated into a corresponding SQL function and submitted to the hive cluster for calculation; if the application scenario of the strategy is a real-time scenario, the expression string is compiled as a groovy script to generate a class file and directly loaded for use.

5. The method for implementing a real-time offline compatible decision engine according to claim 3, characterized in that: The step 1.2 comprises: Step 1.2.1: Initialize an empty stack; Step 1.2.2: Scan the string from left to right. When a left bracket is encountered, push the current subscript onto the stack. When a right bracket is encountered, pop the stack. When the string is scanned, if the stack is empty, the brackets are matched correctly. If the stack is not empty or the right bracket is popped before the scan is completed, the brackets are matched incorrectly, indicating a syntax error. Step 1.2.3: According to the regular expression, find the preset function directly by the preset function name, and use step 1.2.2 to find all the parameters of the preset function. The specific method is to set an empty stack s, push the left bracket into the stack, and pop the right bracket into the stack. When the empty stack appears for the first time, the position is the end position of the current function. The remainder of the entire function can be replaced by a variable; Step 1.2.4: After step 1.2.3, there are no functions in the expression, and all brackets are matched. The remaining expression should naturally form a simple expression.

6. The method for implementing a real-time offline compatible decision engine according to claim 1, characterized in that: The step 3 comprises: Step 3.1: First, split the expression according to the +- sign. If there is no +- sign, use * / to split. Step 3.2: According to the expression list cut in 3.1, recursively process them one by one. When only a single element indicator is left in the expression, the recursion is completed and the current indicator name is returned.

7. The method for implementing a real-time offline compatible decision engine according to claim 1, characterized in that: The step 4 comprises: Step 4.1: Scan the reverse Polish expression from left to right and initialize an empty stack s; Step 4.2: When an indicator variable is encountered during the scanning process, the variable is pushed into stack s; Step 4.3: If an operation symbol is encountered, two variables are popped from the stack. At this time, the operator and the two variable types are checked to see if they meet the requirements. If not, a syntax error is directly determined. If successful, the result variable is pushed into the stack s. Step 4.4: After the scan is completed, the return type of the entire expression is obtained, and there will be no abnormal situation in which the stack is still popped when it is empty or there is a remaining stack after the scan is completed.

8. A real-time offline compatible decision engine system, characterized in that: It includes a strategy input module, an offline module and an online module. The strategy input module adopts a real-time offline compatible decision engine implementation method as described in any one of claims 1 to 7 to assist users in writing strategies. The offline module is used to translate the functions in the expression into corresponding SQL functions and submit them to the hive cluster for calculation. The online module is used to compile the expression string as a groovy script and then generate a class file for direct loading and use.

Images