Data processing method and system based on memory rule engine and medium

Abstract

The invention discloses a data processing method and system based on a memory rule engine and a medium, and belongs to the technical field of data processing. The method comprises the following steps: establishing a memory erasable operator library which can be dynamically expanded; selecting operators required by the task; setting a data source for each operator, converting all the operators into independent operator objects, and generating a task formula text according to the connection relation of all the operator objects; storing the task formula text in a database; when the target data are processed, the target data are input into the data preprocessing module to be preprocessed to obtain first data; inputting the first data into a task execution engine, and loading a task formula text from a database by the task execution engine to analyze and calculate the first data to obtain a calculation result value; and finally, inputting a calculation result value into a result signal processing module to execute a target task. The method is suitable for time sequence and non-time sequence data service scenes, and is also suitable for scenes without memorizing historical values and scenes with multiple historical memory values.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, system and medium based on a memory rule engine. Background Art

[0002] A rule engine is a component embedded in an application that separates business decisions from application code and writes business decisions using predefined semantic modules. It does this by separating business decisions from application code and writing business decisions using predefined semantic modules, accepting data input, interpreting business rules, and making business decisions based on business rules. Existing rule engines are generally used in application scenarios where numerical logic operations are relatively simple but parameters are flexible. For example, in financial risk control scenarios, rule engines are used to automatically determine whether to approve a loan based on the borrower's credit score, income level, and other financial information [if the borrower's credit score is >= 700 and the borrower's monthly income is >5000, then approve the borrower's loan]; in medical diagnosis scenarios, rule engines can assist doctors in providing possible diagnosis and treatment plans based on the patient's symptoms and examination results [if the patient suffers from high blood pressure and the patient's age is >60, then it is recommended that the patient undergo regular checkups]; in the insurance claims industry, rule engines are used to automate the claims process, determine whether the claim application complies with company policies, and quickly process simple claims requests [if the claim application amount is <5000 and the accident date of the claim application is within the policy validity period, then quickly process the claim].

[0003] The existing rule engines are all used for simple data operations, such as four simple addition, subtraction, multiplication and division operators. When using these operators to generate formulas, there is no need to consider the impact of historical values ​​on the latest calculation results. Instead, the same operator is treated as an independent new call each time it is called, and no consideration is given to the need to use the previous input value or result value to calculate the latest result.

[0004] Traditional data processing technologies are often limited to specific data types and business scenarios, such as processing solutions specifically for time series data or non-time series data. These solutions are incapable of handling complex and changing business needs. In particular, when dealing with mixed business scenarios that include both time series features and non-time series data, existing data processing tools and methods often lack sufficient flexibility and adaptability, and it is difficult to meet the urgent needs of enterprises for efficient and accurate data analysis. For time series data business scenarios, such as financial transaction analysis and IoT device monitoring, the characteristics of data changing over time are crucial, and it is necessary to be able to capture the time series relationship and trend of data. However, traditional time series data processing technologies usually only focus on the time dimension of data, ignoring the multi-historical memory value processing that may be required in business logic, that is, the ability to make current decisions or predictions based on data from multiple time points in the past. On the other hand, non-time series data business scenarios, such as customer relationship management and inventory management, although not involving explicit time series analysis, also need to handle complex data relationships and rules, especially in decision support systems, which often require intelligent reasoning and judgment based on historical data and business rules. Existing non-time series data processing solutions often lack sufficient memory mechanisms and are difficult to effectively support such business logic that requires multi-step and multi-condition judgments. Most existing data processing systems use hard-coding or fixed configurations to implement specific data processing logic, which not only limits the flexibility of the system but also increases the cost of maintenance and upgrades. When business personnel conduct data analysis, they usually need to rely on professional technicians to perform complex programming or configuration work, which not only reduces work efficiency but also limits the ability of business personnel to explore and innovate independently. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a data processing method, system and medium based on a memory rule engine.

[0006] The object of the present invention is achieved through the following technical solutions: The first aspect of the present invention provides: a data processing method based on a memory rule engine, comprising the following steps: Establish a dynamically scalable memory-erasable operator library; On the analysis task creation page, select the operator required for the task; Set data sources for each operator, transform all operators into independent operator objects, and generate task formula text based on the connection relationship of all operator objects; Save the task formula text into the database; When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0007] Preferably, the internal processing logic of the operators in the dynamically expandable memory-erasable operator library comprises the following steps: The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned a value according to the operator operation parameters and the operator operation logic; If the environment parameter clear passed into the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset, otherwise the value in the memory unit corresponding to the operator object is retained for continued use in subsequent calculation processes; then the memory value is taken out from the memory unit corresponding to the operator object according to the objectName of the operator object, recorded as memory[objName]; Perform calculations according to the operator logic based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result; After completing the update of the memory cell value according to the operator calculation logic, the final calculation result object result is returned; The memory unit values ​​are stored in a three-layer tree storage structure, the first layer of the three-layer tree storage structure stores the names of each operator object, the second layer of the three-layer tree storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree storage structure stores the actual memory unit values.

[0008] Preferably, for a memoryless operator, after the calculation result object result is initialized and assigned a value, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and after the obtained operation result value is updated to the calculation result object result, the final calculation result object result is directly returned.

[0009] Preferably, when setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required for the operator and nested calculation of the same operator.

[0010] Preferably, when the expression of the down edge operator Down is called as Down(name1,a), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the down edge operator Down includes the following steps: Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result; According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit; If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated; Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

[0011] Preferably, when the expression of the CrossOver operator being called is CrossOver(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the CrossOver operator includes the following steps: Initialize the value field of the calculation result object result of the CrossOver operator to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit; Use the first parameter value, the second parameter value and the operator operation parameters a and b to perform calculations. If the first parameter value retrieved from the memory unit is less than or equal to the second parameter value and the value of the operator operation parameter a passed in this time is greater than the value of b, then update the value of the calculation result object result to 1, otherwise do not update the value of the calculation result object result; Update the memory value of the operator object in the memory unit of the CrossOver operator to the values ​​of the two operator operation parameters a and b of this call, and return the calculation result object result.

[0012] Preferably, when the expression of the memoryless multiplication operator Mul is called as Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the memoryless multiplication operator Mul includes the following steps: Initialize the value field of the calculation result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

[0013] Preferably, when the dynamically expandable erasable operator library with memory does not have the operator required for the task, the operator required for the task is added to the expandable erasable operator library with memory according to the operator creation specification.

[0014] The second aspect of the present invention provides: a data processing system based on a memory rule engine, used to implement any of the above-mentioned data processing methods based on a memory rule engine, comprising: Operator library establishment module, used to establish a dynamically expandable and erasable operator library with memory; The operator selection module is used to select the operators required for the task on the analysis task creation page; The task formula text generation module is used to set the data source for each operator, transform all operators into independent operator objects, and generate the task formula text according to the connection relationship of all operator objects; Database module, used to save task formula text into database; The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0015] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned data processing methods based on a memory rule engine is implemented.

[0016] The beneficial effects of the present invention are: 1) It is applicable to both time series data business scenarios and non-time series data business scenarios. It can handle both simple scenarios that do not require historical value memory and scenarios with multiple historical memory values. At the same time, the technical solution of the present invention has good scalability, and can continuously enrich and improve the available operators of the operator library, supporting business personnel to complete business data analysis in the simplest drag-and-drop manner; it simplifies the data analysis operations and efficiency of business personnel, better supports the separation of business and technology, and improves enterprise work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a data processing method based on a memory rule engine; Figure 2 Create a flowchart for the task formula text; Figure 3 It is the target data processing flow chart; Figure 4 Schematic diagram of falling edge operator and counter operator; Figure 5 It is a schematic diagram of addition, subtraction, multiplication and division operators; Figure 6 It is the logic flow chart for the internal processing of the operator; Figure 7 Schematic diagram of the up-through operator; Figure 8 Schematic diagram of the Greater operator; Fig. 9 It is a schematic diagram of the structure of the memory unit of the three-layer pointer model; Fig.10 Schematic diagram of another three-layer pointer model memory unit structure. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] The present invention uses a rule engine to develop a non-coding configuration tool, through which the message data reported by the IoT device can be split, parsed and stored in the warehouse according to the specified configuration rules, thereby avoiding the back-end R&D personnel from completing the message splitting, parsing and storage operations through hard coding, and improving the convenience of data collection and storage. The present invention uses a general rule engine (such as Aviatorscript) to design a set of function parameter specifications and logic processing standard processes to establish a series of expandable and memorable general operator libraries, supporting business personnel to drag, pull and drag in the interface in a non-coding manner, so that business personnel can independently complete more complex data analysis of business data, give full play to the business personnel's business knowledge advantages, effectively avoid communication with back-end developers, and achieve cost reduction and efficiency improvement.

[0020] See also Figure 1-10 The first aspect of the present invention provides: a data processing method based on a memory rule engine, comprising the following steps: Establish a dynamically scalable memory-erasable operator library; On the analysis task creation page, select the operator required for the task; Set data sources for each operator, transform all operators into independent operator objects, and generate task formula text based on the connection relationship of all operator objects; Save the task formula text into the database; When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0021] In this embodiment, if Figure 2 As shown, on the page provided by the system (S01 module), through the various operators provided by the system, select the required operators and assign the input data source to the operators, so as to create data analysis tasks and corresponding calculation formulas, and save the generated data analysis formulas to the database (the task table sample stored in the final S03 module is shown in Table 1); Table 1 exist Figure 2 After the data analysis formula exists in the S03 module shown in the figure, Figure 3 The task execution engine shown in S12 will automatically load the analysis task from S03. After the data preprocessing module shown in S11 processes the data into a standard format, it will automatically flow into the S12 module for analysis and calculation to calculate the result value to be calculated by the corresponding formula, and then Figure 3 The result signal processing module S13 shown in the figure automatically performs subsequent tasks after obtaining the result. The existing rule engines are all used for simple data operations, such as Figure 5 As shown in the figure, there are four simple addition, subtraction, multiplication and division operators. When using these operators to generate formulas, there is no need to consider the impact of historical values ​​on the latest calculation results. Instead, the same operator is treated as an independent new call each time it is called, and the need to use the last input value or result value to calculate the latest result is not considered. Figure 4 The two operators shown in the S21 and S22 modules need to use certain historical values ​​to calculate the result of the latest call (the logic of the falling edge operator shown in S21 is: if the last input timing signal is 1 and the current input timing signal is 0, then the output result value is 1, otherwise the output result value is 0. Obviously, the calculation result of the latest call of the operator needs to be combined with the last input value to get an accurate result; the logic of the counter operator shown in S22 is: count how many times the signal 1 appears in the continuous timing signal. Obviously, the calculation result of the latest call of the operator needs to be calculated based on the value of the last calculation result).

[0022] In order to enable the rule engine to cater to more complex business scenarios (adapting to time series data, non-time series data, and mixing memoryless and memoryless data), the present invention designs a set of technical specifications for data analysis systems and methods with memory-erasable rule engine operators based on general rule engines (such as Aviatorscript), so that Figure 2 On the S01 analysis task creation page shown in the figure, you can complete the formula generation of the data analysis task by dragging and dropping. Figure 3 After the data preprocessing module S11 processes the data into data that meets the technical specifications, the data is successfully executed in the task execution engine S12 to obtain the analysis results. Figure 2 The final analysis task shown in Table 1 is created on the S01 analysis task creation page. First, a set of scalable, memory-erasable operator libraries needs to be established for users to use directly on the interface provided by the S01 module.

[0023] The concept of turning operators into operator objects is a key point. It requires ensuring that the same operator is a different operator object in the same execution formula, and that the same operator is a different operator object in different execution formulas. Only in this way can Figure 3The S12 task execution engine shown in the figure precisely controls the historical memory cache value of each operator object entity, thereby ensuring the correctness of the calculation results of each execution formula. In order to ensure the above two points, when establishing an extensible memory-erasable operator library, the specifications that each independent operator needs to follow are shown in Table 2: Table 2 In each attribute field of Table 2, funcName is the name of the operator and paramNum is the number of external parameters received by the operator. These two parameters are used as the basic information for defining the operator. The memory attribute is the internal attribute value of the operator object, which is used to save the memory cache value of each operator object. The three attribute values ​​of funcName, paramNum and memory are not accepted when calling. The clear attribute is an environment parameter, not a value directly involved in the operator operation. It is a control variable used to control whether to reset the memory of the operator object. Figure 3 The S12 module shown in the figure automatically determines and assigns values ​​during the running environment [e.g., setting 1 to clear the memory cache value at 12:00 every night, or clearing the memory cache value of the specified operator object when the operation value of the operator object reaches a certain condition or threshold after multiple operations]; objName and operator operation parameters [paramObject1, paramObject2, ...] need to be clearly specified when calling, as they need to be in Figure 2 When the S02 module generates the task formula text, it needs external parameters. objName specifies the specific operator object involved in the operation, and the operator operation parameters [paramObject1, paramObject2, ...] specify the specific parameters involved in the operator operation. Figure 2 In the process of dragging operators and setting input data sources to generate corresponding formulas on the page provided by the S01 module shown in the figure, all operator objects (even Figure 5 The simplest addition, subtraction, multiplication and division operators shown in the figure need to create each operator object according to the following specifications: operator object => operator symbol (operator object name, parameter 1, parameter 2, parameter 3, ...).

[0024] In some embodiments, when setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required for the operator and nested calculations of the same operator.

[0025] In this embodiment, in order to improve the versatility of operator development and avoid overly complex and diverse operator operation parameter objects, the present invention sets the operator operation parameters according to the principle of the minimum number of operator operation parameters required by the operator and the nested calculation of the same operator. For example, in order to meet the scenario where multiple data need to be added together (for example, the sum of 4 parameter objects a+b+c+d needs to be calculated) or multiple data need to be multiplied together (for example, the product of 6 parameter objects a*b*c*d*e*f needs to be calculated), such as Figure 5 The number of parameters of the addition operator Add shown in S31 and the multiplication operator Mul shown in S32 are both 2. All operations with multiple parameters are processed as nested modes of the same operator. Taking the addition operation of multiple parameter objects as an example (such as calculating the sum of 4 parameter objects: a+b+c+d), there is no need to develop a new 4-parameter addition operator to meet user needs. Figure 1 In the S01 module shown in the figure, the four parameters set are processed as the same addition operator (Add) nested three times in a row, that is, the generated Figure 2 The expression corresponding to the S02 module shown is Add(name1,Add(name2,Add(name3,a,b),c),d).

[0026] The same addition operator Add is nested three times. According to the above operator object creation specifications, the creation of several operator objects and the related specific semantics are as follows: Figure 5 The expression text of the addition operator module of S31 when called once is as follows: Add(name1,a,b). This expression creates an addition operator object named name1, which is created by a memoryless operator whose funcName is Add. During the calculation process, parameters a and b are replaced by two parameter objects passed in by the data preprocessing module S11 in the task execution engine module S12. Since a memoryless operator is used to create the addition operator object named name1, each time the operator object is called, only the sum of the values ​​of parameters a and b passed in to the operator object at the current calling time is calculated, that is, a+b is used as the output result value of the operator object. Figure 7 The expression text of the crossover operator module of S51 when called once is CrossOver(name1,a,b). This expression creates an crossover operator object named name1. During the calculation process, parameters a and b are replaced by two parameter objects passed in by the data preprocessing module S11 in the task execution engine module S12. This operator object is created by the erasable operator CrossOver with memory. The two operation parameters passed in to the operator object when calling the operator object for the nth time are a and b. nWith b n , when the operator object is called for the n+1th time, the two operation parameters passed into the operator object are a n+1 With b n+1 , if a n <=b n And a n+1 >b n+1 The calculation result of the n+1th call is 1, otherwise the calculation result is 0; Figure 4 The expression text Down(name1,a) of the falling edge operator module of S21 when it is called once, this expression creates a falling edge operator object named name1. During the calculation process, the parameter a will be replaced by a parameter object passed in by the data preprocessing module S11 in the task execution engine module S12. This operator object is created by the erasable operator Down with memory. The operation parameter passed in to the operator object when the operator object is called for the nth time is a n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 , if a n+1 n The calculation result of the n+1th call is 1, otherwise the calculation result is 0; Figure 4 The expression text Counter(name1,a) of the counter operator module S22 when it is called once, this expression creates a counter operator object named name1. During the calculation process, the parameter a will be replaced by a parameter object passed in by the data preprocessing module S11 in the task execution engine module S12. This operator object is created by the erasable operator Counter with memory. The calculation result obtained after passing in the operator operation parameters when calling the operator object for the nth time is R. n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 , if a n+1 If it is greater than or equal to 1, the calculation result R of the n+1th call is output. n+1 =1+R n , otherwise the calculated result R n+1 =R n ; Operator objects created according to the above operator object creation specifications can be combined and nested to obtain more complex expressions. The following are several expressions and semantic examples composed of operator objects: Figure 5 ​The expression text of the nested call of the addition operator module of S31 is: Add(name2,Add(name1,a,b),c). This expression creates two addition operator objects, whose names are name1 and name2 respectively. The calculation of this expression is Figure 3 The S12 module shown will go through two calculation steps: the first step is to calculate the calculation result of the addition operator object named name1. The addition operator Add is a memoryless operator. During the calculation process, the parameters a and b will be replaced by two actual parameter objects for calculation, and the calculation result value of this step will be a+b; then in the second step, the calculation result value will be passed as an operator operation parameter object together with another operator operation parameter object c that actually participates in the calculation to the addition operator object named name2, and the calculation result value of the final expression Add(name2,Add(name1,a,b),c) will be a+b+c; Will Figure 4 The expression text of the two operator modules shown in the figure for multiple combined calls is Counter(name3,Down(name2,Down(name1,a))). This expression creates a total of three operator objects, namely the down edge operator object named name1, the down edge operator object named name2, and the counter operator object named name3. The calculation of this expression is Figure 3 The S12 module shown in the figure will go through two calculation steps: the first step is to calculate the calculation result of the falling edge operator object named name1, and the operation parameter a passed into the operator object when the operator object is called for the nth time n When the operator object is called for the n+1th time, the operation parameter passed into the operator object is a n+1 When the operator object is called for the n+2th time, the operation parameter passed into the operator object is a n+2 , with the symbol Dr representing the output result sequence of the falling edge operator object name1 (Dr 1 ,Dr 2 ,...,Dr n ,Dr n+1 ,Dr n+2 ), the calculation result of the operator object after being called for the n+1th time is recorded as Dr n+1 (If a n+1 n， Dr n+1 =1, otherwise Dr n+1 =0), the calculation result after the n+2th call is recorded as Dr n+2 (If a n+12 n+1， Dr n+2 ​​=1, otherwise Dr n+2 =0); In the second step, the value of the falling edge operator object name2 will be calculated. At this time, the calculation result value of the operator object named name1 will be used as the operator operation parameter of the falling edge operator name2, and the symbol DDr will represent the output result sequence of the falling edge operator object name2 (DDr 1 ,DDr 2 ,...,DDr n ,DDr n+1 ,DDr n+2 ), the calculation result after the operator is called for the n+1th time is recorded as DDr n+1 (If Dr. n+1 <Dr n， Then DDr n+1 =1, otherwise DDr n+1 =0), the calculation result after the n+2th call is recorded as DDr n+2 (If Dr. n+2 <Dr n+1， Then DDr n+2 =1, otherwise DDr n+2 =0); In the third step, the calculation result sequence of the falling edge operator object name2 will be used as the operator operation parameter sequence of the counter operator object Counter, and the symbol Cr will represent the calculation result sequence of each call of the counter object (Cr 1 ,Cr 2 ,...,Cr n ,Cr n+1 ,Cr n+2 ), the operator parameter object passed in when the operator object is called for the n+1th time is DDr n+1 , if DDr n+1 If it is greater than or equal to 1, the calculation result Cr n+1 =1+Cr n , otherwise Cr n+1 =Cr n The operator parameter object passed in when the operator is called for the n+2th time is DDr n+2 , if DDr n+2 If it is greater than or equal to 1, the calculation result Cr n+2 =1+Cr n+1 , otherwise Cr n+2 =Cr n+1 .

[0027] In some embodiments, the internal processing logic of the operators in the dynamically expandable memory-erasable operator library includes the following steps: The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned a value according to the operator operation parameters and the operator operation logic; If the environment parameter clear passed into the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset, otherwise the value in the memory unit corresponding to the operator object is retained for continued use in subsequent calculation processes; then the memory value is taken out from the memory unit corresponding to the operator object according to the objectName of the operator object, recorded as memory[objName]; Perform calculations according to the operator logic based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result; After completing the update of the memory cell value according to the operator calculation logic, the final calculation result object result is returned; The memory unit values ​​are stored in a three-layer tree storage structure, the first layer of the three-layer tree storage structure stores the names of each operator object, the second layer of the three-layer tree storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree storage structure stores the actual memory unit values.

[0028] In this embodiment, in addition to following the above-mentioned development specifications when developing operators, in order to better be compatible with scenarios where both memory and non-memory, time series data and non-time series data can be used, this technical solution also designs the processing logic inside the operator accordingly, such as Figure 6 As shown, the specific steps are as follows: Step 1: First, record the calculation result object as result, and perform corresponding initialization assignment operations on the result object result according to the passed operator operation parameters [paramObject1, paramObject2, ...] (if the operator operation parameters do not have a timestamp field value, fill the field with the default value -1) and the operator operation logic; Step 2: According to Figure 3 The environment parameter clear passed into the S12 module during execution, if clear is 1, the value of the memory unit corresponding to the operator object is reset first, and then step 3 is entered after reset; Step 3: According to the objectName of the operator object, take out the memory value from the memory unit corresponding to the operator object, recorded as memory[objName]; Step 4: Perform calculations based on the memory value memory[objName] retrieved in step 3 and the operator operation parameters [paramObject1, paramObject2, ...] passed in when the operator object is called according to the operator's operation logic, and update the operation result value to the result object; Step 5: After completing the update of the memory cell value according to the logic of the operator, the final calculation result result is returned.

[0029] Due to the existence of nested calls, in order to ensure that the calculation result of an operator object can be passed in as the operator calculation parameter of other operator objects in both memory and non-memory, time series data and non-time series data scenarios, when initializing the result to be returned in the above step 1, this technical solution needs to keep its structure the same as the operator calculation parameters [paramObject1, paramObject2, ...] in the aforementioned operator specification table, and also needs to include 2 fields: timestamp and value.

[0030] For business scenarios with non-time series data, because Figure 6 In step 1, the non-time series data involved in the calculation has been filled with the timestamp field value -1 (such as Figure 8 The non-time series operator operation parameter 10 of the Greater operator will be converted into time series data in step 1 and filled with timestamp -1). When the timestamp field of the result object result is assigned in step 1, result.timestamp is set to the maximum timestamp value of all operator operation parameter objects.

[0031] For the most recently called memoryless operator that does not need to use memory data for calculation, you can skip steps 2 and 3 and directly enter step 4 to use the operator calculation parameters [paramObject1, paramObject2, ...] to calculate according to the operator's own calculation logic, and update the calculation result value to the result object. Then enter step 5 and skip the update of the operator object memory unit, and directly return the final calculation result result. In order to be applicable to the scenario where a single operator object needs to adapt to multiple memory parameters, the present invention adopts the following method: Fig. 9 With Fig.10 The three-layer pointer model shown completes the cache of memory parameters. The first layer stores the names of each operator object of the operator, the second layer stores the parameter names required to be stored in the operator, and the last layer stores the memory cache values ​​corresponding to the parameter names.

[0032] In some embodiments, for a memoryless operator, after the calculation result object result is initialized and assigned a value, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and after the obtained operation result value is updated to the calculation result object result, the final calculation result object result is directly returned.

[0033] In some embodiments, when the expression of the down edge operator Down is called as Down(name1,a), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the down edge operator Down includes the following steps: Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result; According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit; If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated; Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

[0034] In this embodiment, it is assumed that Figure 4 The expression used for the S21 down edge operator (Down) module with erasable memory is Down(name1,a), and the environment parameter clear is Figure 3 The S12 task is assigned a value of 0 when executing the engine, and its operation steps are as follows: Step 11: Initialize the value field of the calculation result result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of result, that is, at this time result.timestamp = a.timestamp, result.value = 0; Step 12: Since Figure 3 The S12 execution engine module shown sets the environment parameter clear value to 0, so there is no need to reset the value of the memory unit, and directly enters step 13; Step 13: According to the operator object name1, select the memory unit (the memory unit structure of the falling edge operator Down only needs to store the parameter value of the last call, so each operator object only needs one memory unit. The memory unit structure is as follows Fig. 9Take out the last parameter value from the memory (as shown in the figure) (when called for the first time, the default last parameter value is set to 0, and it will be updated in step 15 after being called multiple times), and record the value as memory[name1]; Step 14: If memory[name1] retrieved in step 13 is greater than the value of parameter a [i.e. memory[name1]=1, a.value=0], then update result.value=1; otherwise, do not update the value of the calculation result object result (result.value is still 0); Step 15: Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the parameter object a during this call (place the value of a this time into Fig. 9 The calculation result object result is returned.

[0035] In some embodiments, when the expression of the CrossOver operator is CrossOver(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the CrossOver operator includes the following steps: Initialize the value field of the calculation result object result of the CrossOver operator to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit; Use the first parameter value, the second parameter value and the operator operation parameters a and b to perform calculations. If the first parameter value retrieved from the memory unit is less than or equal to the second parameter value and the value of the operator operation parameter a passed in this time is greater than the value of b, then update the value of the calculation result object result to 1, otherwise do not update the value of the calculation result object result; Update the memory value of the operator object in the memory unit of the CrossOver operator to the values ​​of the two operator operation parameters a and b of this call, and return the calculation result object result.

[0036] In this embodiment, it is assumed that Figure 7 The expression used to call the CrossOver module with erasable memory S51 and S52 is CrossOver(name1,a,b), and the environment parameter clear is Figure 3 The S12 task is assigned a value of 0 when executing the engine, and its operation steps are as follows: Step 21: Initialize the value field of the calculation result result of the operator CrossOver to 0, and assign a value to the result timestamp field according to the timestamp fields of the operator operation parameters a and b. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; Step 22: Since Figure 3 The execution engine module S12 shown sets the environment parameter clear value to 0, so there is no need to reset the value of the operator object memory unit, and directly enters step 23; Step 23: According to the operator object name1, the memory unit structure of the CrossOver operator needs to store the two parameter values ​​from the last call, so each operator object needs 2 memory units. The memory unit structure is as follows: Fig.10 25) and record the value as memory[name1] (which can be further divided into two values: memory[name1].last1 and memory[name1].last2); Step 24: Use memory[name1] retrieved in step 23 and the operator operation parameters a and b involved in this calculation to perform calculation. If the first parameter value retrieved from the memory unit (memory[name1].last1) is less than or equal to the second parameter value (memory[name1].last2) and the value of the operator operation parameter a passed in this time is greater than the value of b, then update result.value = 1, otherwise the value of the calculation result object result is not updated (result.value is still 0); Step 25: Update the memory value of the operator object in the CrossOver memory unit to the values ​​of the two operator operation parameters during this call (place the value of a this time into Fig.10 In the memory unit pointed to by name1->last1, the value of b is placed as follows Fig.10 In the memory unit pointed to by name1->last2 as shown), and return the result object of this calculation.

[0037] In some embodiments, when the expression of the memoryless multiplication operator Mul is called as Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the memoryless multiplication operator Mul includes the following steps: Initialize the value field of the calculation result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

[0038] In this embodiment, Figure 5 The expression when the memoryless multiplication operator (Mul) module is called is Mul(name1,a,b), and the environment parameter clear is Figure 3 The S12 task is assigned a value of 0 when executing the engine, and its operation steps are as follows: Step 31: Initialize the value field of the calculation result result of the operator Mul to 0, and assign a value to the result timestamp field according to the timestamp fields of the operator operation parameters a and b. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; Step 32, 33: Since this operator is a memoryless operator, skip steps 32 and 33 and go directly to step 34; Step 34: Since this operator is a memoryless operator, the value fields of the operator operation parameters a and b are directly used for calculation, and result.value = a.value * b.value is updated; Step 35: Since this operator is a memoryless operator, skip the memory unit update and directly return the calculation result object result.

[0039] In some embodiments, when the dynamically expandable erasable operator library with memory does not have the operator required by the task, the required operator is added to the expandable erasable operator library with memory according to the operator creation specification.

[0040] The second aspect of the present invention provides: a data processing system based on a memory rule engine, used to implement any of the above-mentioned data processing methods based on a memory rule engine, comprising: Operator library establishment module, used to establish a dynamically expandable and erasable operator library with memory; The operator selection module is used to select the operators required for the task on the analysis task creation page; The task formula text generation module is used to set the data source for each operator, transform all operators into independent operator objects, and generate the task formula text according to the connection relationship of all operator objects; Database module, used to save task formula text into database; The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

[0041] In this embodiment, according to the operator definition specification of the present invention, the processing flow is combined Figure 2 and Figure 3 The technical modules shown in the figure can develop a set of scalable user-defined data analysis systems suitable for real-time data and non-real-time data, memory and non-memory mixed scenarios. For example, after the developer has completed the development of the required operators, the user can directly configure on the page that if the number of jumps from 0 to 1 of a certain signal a is greater than N times (N=10), an alarm prompt will be issued [the user Figure 2 On the page shown in the S01 module, complete the following steps by dragging, dropping, and pulling. Figure 8 The topology shown and the corresponding data source settings will create the corresponding Figure 2 The task formula text shown in S02 is Greater(name3,Counter(name2,Down(name1,a)),10), for Figure 3 The S12 task execution engine is executed. When the expression execution result is 1 at a certain moment, Figure 3 In the result signal processing module S13 shown in FIG. 1 , a corresponding alarm is given; in another example, in stock trading, when a signal a crosses a signal b and a signal c crosses a signal d at the same time, a prompt signal is given [the user can Figure 2 On the page shown in the S01 module, complete the following steps by dragging, dropping, and pulling. Figure 7 After the topology shown in the figure is set up with the corresponding data source, the corresponding Figure 2 The task formula text And(name3,CrossOver(name1,a,b),CrossOver(name2,c,d)) shown in S02 is provided Figure 3 The S12 task execution engine is executed. When the expression execution result is 1 at a certain moment, Figure 3 The corresponding prompt processing is performed in the result signal processing module S13 shown].

[0042] Figure 2 and Figure 3 There are two independent processes for the overall processing of the system: data analysis task creation process and data analysis task execution process.

[0043] Figure 2The software page and corresponding system processing flow (data analysis task creation flow) provided to business personnel: business personnel Figure 2 In the S01 module shown in the figure, based on actual business needs, the operator to be used is selected by dragging and dropping from the available operator library established in accordance with the method described in the present invention (for example, if the sum of two data needs to be calculated, drag and drop from the optional operator library Figure 5 The Add operator shown in the figure), and set the data source required for the selected operator to be input (for example, if the two data required for the business to calculate and are A and B respectively, then double-click the Add operator to pop up the signal selection window, set the operator calculation parameters for the Add operator to A and B respectively, and the system automatically adds a globally unique name to each selected operator to turn the operator into an operator object), and then the S02 module generates the corresponding task formula text according to the specifications described in the technical solution (according to the input and output connection relationship of each operator object on the page, the final execution formula to be calculated is generated, such as: Figure 7 The input and output connection relationship of each operator is shown in the figure. The final execution formula is And(name3,CrossOver(name1,a,b),CrossOver(name2,c,d)). It is finally generated together with other information related to the task set by the user on the S01 page. Figure 2 The S03 module shown in the figure is saved together with the data analysis task table (the final data analysis task table is shown in Table 1, where the task execution formula column shown in Table 1 is the execution formula created by the S02 module in accordance with the specifications of this technical document) for use Figure 3 The following data analysis task execution process is shown.

[0044] Figure 3 To perform the processing flow for the actual data analysis task, users Figure 2 After the data analysis task creation process creates the task, it will run automatically without the need for further intervention and settings. When the analysis task is actually executed, the data analysis task execution process will first load, correspond and process the data sources required for each actual execution task after the data analysis task is created in the S11 module. For example, if the sum of signals A and B needs to be calculated, the data A and B required for the calculation need to be found from the actual data source, and necessary processing needs to be performed so that the processed data A and B can meet the requirements of the subsequent execution engine [such as: converting from hexadecimal messages to decimal data] or other business needs [such as: only calculating the sum of A and B within a certain period of time]. The S12 module accesses the actual processed data as the calculation parameters in the task execution formula, and the corresponding operator objects in the formula will be processed according to Figure 6The processing flow shown calculates the value of each operator object one by one until the final formula value; the above final formula value will be given to the result signal processing module S13 for subsequent result processing, such as: saving the analysis results to a database, drawing analysis charts, etc.

[0045] In order to enable the entire system to be compatible with processing both sequential and non-sequential business, mixed business scenarios with memory and without memory, this solution is based on Figure 2 and Figure 3 When creating and executing analysis tasks in the process, Figure 6 The overall operator object operation processing logic and the corresponding operator establishment specifications ensure the flexibility of system use and the correctness of the analysis task operation results.

[0046] The third aspect of the present invention provides: a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned data processing methods based on a memory rule engine is implemented.

[0047] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A data processing method based on a memory rule engine, characterized in that: The following steps are involved: Establish a dynamically scalable memory-erasable operator library; On the analysis task creation page, select the operator required for the task; Set data sources for each operator, transform all operators into independent operator objects, and generate task formula text based on the connection relationship of all operator objects; Save the task formula text into the database; When processing target data, the target data is first input into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

2. The data processing method based on a memory rule engine according to claim 1, characterized in that: The internal processing logic of the operators in the dynamically expandable memory-erasable operator library includes the following steps: The calculation result object of the operator is recorded as result, and the calculation result object result is initialized and assigned a value according to the operator operation parameters and the operator operation logic; If the environment parameter clear passed into the task execution engine during execution is 1, the value in the memory unit corresponding to the operator object is reset, otherwise the value in the memory unit corresponding to the operator object is retained for continued use in subsequent calculation processes; then the memory value is taken out from the memory unit corresponding to the operator object according to the objectName of the operator object, recorded as memory[objName]; Perform calculations according to the operator logic based on the memory value memory[objName] and the operator operation parameters passed in when the operator object is called, and update the obtained operation result value to the calculation result object result; After completing the update of the memory cell value according to the operator calculation logic, the final calculation result object result is returned; The memory unit values ​​are stored in a three-layer tree storage structure, the first layer of the three-layer tree storage structure stores the names of each operator object, the second layer of the three-layer tree storage structure stores the names of memory parameters that require operator logic access, and the third layer of the three-layer tree storage structure stores the actual memory unit values.

3. The data processing method based on a memory rule engine according to claim 2, characterized in that: For memoryless operators, after initializing and assigning values ​​to the calculation result object result, the operator operation parameters are directly used to perform calculations according to the operator operation logic, and after the obtained operation result value is updated to the calculation result object result, the final calculation result object result is directly returned.

4. The data processing method based on a memory rule engine according to claim 2, characterized in that: When setting operator operation parameters, they are set according to the principle of the minimum number of operator operation parameters required by the operator and nested calculations of the same operator.

5. The data processing method based on a memory rule engine according to claim 2, characterized in that: When the expression of the Down edge operator Down is called as Down(name1,a), and the environment parameter clear passed in by the task execution engine is assigned a value of 0 during execution, the operation logic of the Down edge operator Down includes the following steps: Initialize the value field of the calculation result object result of the falling edge operator Down to 0, and assign the timestamp field of the operator operation parameter a to the timestamp field of the calculation result object result; According to the operator object name of the falling edge operator Down, the parameter value of the last call is retrieved from the memory unit; If the parameter value of the last call is greater than the value of the operator operation parameter a, the value of the calculation result object result is updated to 1, otherwise the value of the calculation result object result is not updated; Update the memory value of the operator object in the memory unit of the falling edge operator Down to the value of the operator operation parameter a, and return the calculation result object result.

6. The data processing method based on a memory rule engine according to claim 2, characterized in that: When the expression of the CrossOver operator is CrossOver(name1,a,b), and the environment parameter clear passed in by the task execution engine is assigned a value of 0 during execution, the operation logic of the CrossOver operator includes the following steps: Initialize the value field of the calculation result object result of the CrossOver operator to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; According to the operator object name of the CrossOver operator, the first parameter value and the second parameter value of the last call are retrieved from the memory unit; Use the first parameter value, the second parameter value and the operator operation parameters a and b to perform calculations. If the first parameter value retrieved from the memory unit is less than or equal to the second parameter value and the value of the operator operation parameter a passed in this time is greater than the value of b, then update the value of the calculation result object result to 1, otherwise do not update the value of the calculation result object result; Update the memory value of the operator object in the memory unit of the CrossOver operator to the values ​​of the two operator operation parameters a and b of this call, and return the calculation result object result.

7. The data processing method based on a memory rule engine according to claim 2, characterized in that: When the expression of the memoryless multiplication operator Mul is called as Mul(name1, a, b), and the environment parameter clear passed in by the task execution engine during execution is assigned a value of 0, the operation logic of the memoryless multiplication operator Mul includes the following steps: Initialize the value field of the calculation result object result of the memoryless multiplication operator Mul to 0, and assign the timestamp fields of the operator operation parameters a and b to the timestamp field of the calculation result object result. At this time, result.timestamp = Max(a.timestamp, b.timestamp), result.value = 0; Use the value fields of operator parameters a and b to perform calculations, update result.value = a.value * b.value, and then directly return the value of the calculation result object result at this time.

8. The data processing method based on a memory rule engine according to claim 1, characterized in that: When the dynamically expandable erasable operator library with memory does not have the operator required by the task, the required operator is added to the expandable erasable operator library with memory according to the operator creation specification.

9. A data processing system based on a memory rule engine, characterized in that: The method for implementing the data processing method based on a memory rule engine according to any one of claims 1 to 8 comprises: Operator library establishment module, used to establish a dynamically expandable and erasable operator library with memory; The operator selection module is used to select the operators required for the task on the analysis task creation page; The task formula text generation module is used to set the data source for each operator, transform all operators into independent operator objects, and generate the task formula text according to the connection relationship of all operator objects; Database module, used to save task formula text into database; The target task execution module inputs the target data into the data preprocessing module for preprocessing to obtain the first data; then the first data is input into the task execution engine, and the task execution engine loads the task formula text from the database to analyze and operate the first data to obtain the calculation result value; finally, the calculation result value is input into the result signal processing module to execute the target task.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the data processing method based on the memory rule engine as described in any one of claims 1-8 is implemented.

Images